Reviewerly/exHarmony huggingface dataset:
Improving Image Captioning with Feature Filtering and Injection

Image captioning represents a challenging multimodal task, requiring the generation of corresponding textual descriptions for complex input images. Existing methods usually leverage object detectors to extract visual features of images, and thus utilize text generators for learning. However, the features extracted by these methods lack focus and tend to ignore the relationship between objects and background information. To solve the aforementioned problems, we exploit both region features and grid features of the image to fully leverage the information encapsulated within the images. Specifically, we first propose an Object Filter Module (OFM) to extract the primary visual objects. Furthermore, we introduce a Global Injection Cross Attention (GICA) to inject the global context of the image into the filtered primary objects. The experimental results substantiate the efficacy of our model. Our model’s effectiveness and immense potential have been demonstrated through extensive experimentation on the widely-used benchmark COCO dataset. It outperforms previous methods on the image captioning task, achieving a CIDEr score of 136.1.
Anchor Link Prediction Based on Trusted Anchor Re-identification

Cross-social network anchor link prediction plays a pivotal role in downstream tasks, such as comprehensively portraying user characteristics, user friend recommendations, and online public opinion analysis, which aims to find accounts that belong to the same natural person on different social networks. It is a common method to use manually marked anchors or anchors inferred through autonomous learning as supervisory information to guide the prediction of subsequent anchor links. However, the credibility of the anchor is not discussed. In this paper, to address this problem, we propose a new framework that can simultaneously complete the identification of trusted anchors and the prediction of anchor links across social networks under a unified framework. The proposed method can effectively identify non-trusted anchor links and improve the accuracy of the anchor link prediction model through the reconstruction of trusted anchors. Extensive experiments have been conducted on two large-scale real-life social networks. The experimental results demonstrate that the proposed method outperforms the state-of-the-art models with a big margin.
Few-Shot NeRF-Based View Synthesis for Viewpoint-Biased Camera Pose Estimation

Recently, several works have paid attention to view synthesis by neural radiance fields (NeRF) to improve camera pose estimation. Among them, LENS and Direct-PoseNet synthesize novel views from pre-trained NeRF and then train the pose regression convolutional network using real observations and the augmented synthetic views for better localization. Therefore, the performance depends on the three-dimensional (3D) consistency and the image quality of novel views. Especially, localization tends to fail if a diverse and high-quality training set is unavailable. To solve this issue, we tackle the problem of learning camera pose regressor from the viewpoint-biased and limited training set. We propose augmenting the regressor’s training set using a few-shot NeRF instead of an original NeRF, which is employed in the previous frameworks. We can render high-quality novel views with a consistent 3D structure for stable training of the regressor. The experiments show that few-shot NeRF is an effective data augmenter for camera pose estimation under the viewpoint-biased limited training set.
CLASPPNet: A Cross-Layer Multi-class Lane Semantic Segmentation Model Fused with Lane Detection Module

Multi-class lane semantic segmentation is a crucial technology in the traffic violation detection system. However, the existing models for multi-classification lane semantic segmentation suffer from low segmentation accuracy for special lanes (e.g., ramp, emergency lane) and lane lines. To address this problem, we propose a cross-layer multi-class lane semantic segmentation model called CLASPPNet (Cross-Layer Atrous Spatial Pyramid Pooling Network) fused with lane detection module. We first design a Cross-Layer Atrous Spatial Pyramid Pooling (CLASPP) structure to integrate the deep and shallow features in the image and enhance the integrity of the lane segmentation. Additionally, we integrate the lane detection module during training in the cross-layer structure, which can improve the model's ability of extracting lane line features. We evaluate CLASPPNet on the expressway dataset based on aerial view, and the experimental results show that our model significantly improves the segmentation performance of special lanes and lane lines. Additionally, it achieves the highest mIoU (mean Intersection over Union) of 86.4% while having 28.9M parameters.
Diffusion-Adapter: Text Guided Image Manipulation with Frozen Diffusion Models

Research on vision-language models has seen rapid development, enabling natural language-based processing for image generation and manipulation. Existing text-driven image manipulation is typically implemented by GAN inversion or fine-tuning diffusion models. The former is limited by the inversion capability of GANs, which fail to reconstruct pictures with novel poses and perspectives. The latter methods require expensive optimization for each input, and fine-tuning is still a complex process. To mitigate these problems, we propose a novel approach, dubbed Diffusion-Adapter, which performs text-driven image manipulation using frozen pre-trained diffusion models. In this work, we design an Adapter architecture to modify the target attributes without fine-tuning the pre-trained models. Our approach can be applied to diffusion models in any domain and only take a few examples to train the Adapter that could successfully edit images from unknown data. Compared with previous work, Diffusion-Adapter preserves a maximal amount of details from the original image without unintended changes to the input content. Extensive experiments demonstrate the advantages of our approach over competing baselines, and we make a novel attempt at text-driven image manipulation.
Cascaded Network-Based Single-View Bird 3D Reconstruction

Existing single-view bird 3D reconstruction methods mostly cannot well recover the local geometry such as feet and wing tips, and the resulting 3D models often appear to have poor appearance when viewed from a new perspective. We thus propose a new method that requires only images and their silhouettes to accurately predict the shape of birds, as well as to obtain reasonable appearance in new perspectives. The key to the method lies in the introduction of a cascaded structure in the shape reconstruction network. This allows for the gradual generation of the 3D shape of birds from coarse to fine, enabling better capturing of local geometric features. Meanwhile, we recover the texture, lighting and camera pose with attention-enhanced encoders. To further improve the plausibility of the reconstructed 3D bird in novel views, we introduce the Multi-view Cycle Consistency loss to train the proposed method. We compare our method with state-of-the-art methods and demonstrate its superiority both qualitatively and quantitatively.
Joint Edge-Guided and Spectral Transformation Network for Self-supervised X-Ray Image Restoration

X-rays are widely utilized in the security inspection field due to their ability to penetrate objects and visualize intricate details and structural features. However, X-ray images often suffer from degradation issues, such as heavy noise and artifacts, which can adversely affect the accuracy of subsequent high-level tasks. Therefore, X-ray image restoration plays a critical role in the applications of X-ray images. Existing supervised restoration methods depend on numerous noisy-clean image pairs for training, which restricts their application to X-ray images. Although there have been a few attempts to train models with single noisy images, they ignored the unique prior knowledge of X-ray images. This result in poor performance with artifacts and inadequate denoising. To tackle these challenges, we propose a novel self-supervised restoration method called the Joint Edge-guided and Spectral Transformation Network (ESTNet), which integrates edge guidance and spectral transformation techniques to restore color X-ray images. Specifically, ESTNet leverages an adaptive edge guidance module to emphasize edge details. In addition, to achieve a balance between noise suppression and detail preservation in image restoration, we propose spatial spectral blocks that enable the network to capture both global and local contextual information. Extensive experiments on real-world images confirm the superiority of ESTNet over state-of-the-art methods in terms of quantitative metrics and visual quality.
A Document-Level Relation Extraction Framework with Dynamic Pruning

Relation extraction (RE) has been a fundamental task in natural language processing (NLP) as it identifies semantic relations among entity pairs in texts. Because sentence-level RE can only capture intra-connections within a sentence rather than inter-connections between or among sentences, researchers shift their attentions to document-level RE to obtain richer and complex relations which may involve logic inference. Prior works on document-level RE suffer from inflexible pruning rules and lack of sentence-level features, which lead to the missing of valuable information. In this paper, we propose a document-level relation extraction framework with both dynamic pruning mechanism and sentence-level attention. Specifically, a weight-based flexible pruning mechanism is applied on the document-level dependency tree to remove non-relational edges dynamically and obtain the weight dependency tree (WDT). Moreover, a graph convolution network (GCN) then is employed to learn syntactic representations of the WDT. Furthermore, the sentence-level attention and gating selection module are applied to capture the intrinsic interactions between sentence-level and document-level features. We evaluate our framework on three benchmark datasets: DocRED, CDR, and GDA. Experiment results demonstrate that our approach outperforms the baselines and achieves the state-of-the-art performance.
MedNet: A Dual-Copy Mechanism for Medical Report Generation from Images

Generating medical reports from images is a complex task in the healthcare domain. Existing approaches often rely on predefined templates to retrieve sentences and do not take into account the hierarchical structure of medical reports. Additionally, they overlook the selective copying of input sequences to output sequences. To address these challenges, we propose MedNet, a generation-based model that employs a dual-copy mechanism to automatically generate medical reports from images. Our methodology involves first extracting features from images using an image encoder. Next, we use a dual-copy mechanism as the sequence encoder for retrieved reports to combine word generation in the decoder with bi-copying subsequences from the input sequence and placing them appropriately. Finally, a language decoder generates coherent and informative reports. We evaluate our MedNet on two public datasets, Open-I and MIMIC-CXR, and demonstrate that it outperforms current state-of-the-art methods. Our approach not only improves the quality of the generated reports but also allows for flexible generation, making it well-suited for a variety of healthcare applications. The proposed dual-copy mechanism in our approach enables the utilization of both integral tokens and sub-tokens, enhancing the accuracy and relevance of generated reports. Our work represents a significant step forward in the field of automated medical report generation from images.
Adaptive Interaction-Based Multi-view 3D Object Reconstruction

This paper introduces an end-to-end deep learning framework for multi-view 3D object reconstruction. The algorithm constructs an adaptive multi-view combination module in the 2D encoding through calculating the feature correlation of pixel points of each view, allowing each view contains the feature information of other views. It addresses the issue of inconsistent object reconstruction resulting from input images being presented in different orders. Additionally, a voxel refinement loss is employed to produce a comprehensive 3D voxel and establishes an adaptive voxel discrimination module for 3D voxel calibration. This reduces the production of superfluous voxels in the 3D voxel and enhances the completeness of the reconstruction. Extensive validation using the ShapeNet synthetic dataset and the Pix3D real-world dataset demonstrates that the proposed algorithm outperforms existing methods.
CLN: Complementary Learning Network for 3D Face Reconstruction and Alignment

In the complex and changeable unconstrained state, the 3D Morphable Model (3DMM) parameters regression methods lack the ability to express local details and they are difficult to enrich geometric details. In this paper, we propose a complementary learning network (CLN), which aims to improve the ability to extract global discriminative features and capture local details through complementary learning between global and local information. Specifically, we first elaborately design a cross-domain self-shuffling data augmentation method to simulate the case of the inconspicuous or obscured face information under unconstrained conditions, thus increasing the sensitivity of the network to local information. Different from other methods, our augmented data contains global data that can guide the network in reasoning about the inconspicuous areas of the face in special lighting and partially obscured areas of the face in large poses. To better achieve the complementary between global information and local details, we design a complementary learning network in which one stream extracts global discriminative features from the original input image. Meanwhile, another stream extracts local detailed features from the shuffled image. Then, we adopt the coordinate attention transformer module to enforce our network’s ability to capture the correlation between local information. Finally, we linearly fuse the global information that constrains the geometric shape of the face and the local information of the rich geometric details to improve the regression accuracy of the network. Extensive experiments on AFLW, AFLW2000-3D, and other datasets demonstrate that our method is superior to previous methods.
Multi-frame Tilt-angle Face Recognition Using Fusion Re-ranking

Tilt-angle face recognition is a common problem in public video surveillance. Based on the complementarity among multi-frame tilt-angle faces from far to near, it is a possible way to improve recognition performance by fusing them. However, the feature fusion in the existing multi-frame frontal face recognition methods is unsuitable for tilt-angle faces with large changes in resolution and angle. To solve this issue, a multi-frame tilt-angle face recognition approach based on fusion re-ranking is proposed, which can obtain a more accurate final similarity list by weighted fusion of initial similarity lists given by different tilt-angle faces. In order to get the person-specific and angle-specific fusion weights, an angle-guided adaptive weight prediction network is designed. Moreover, we propose a weighted ranking loss to train the network, which can make the gallery face with the same identity as the probe face rank higher in the fused list. Experimental results on the tilt-angle face datasets demonstrate the effectiveness of our method.
Input Layer Binarization with Bit-Plane Encoding

Binary Neural Networks (BNNs) use 1-bit weights and activations to efficiently execute deep convolutional neural networks on edge devices. Nevertheless, the binarization of the first layer is conventionally excluded, as it leads to a large accuracy loss. The few works addressing the first layer binarization, typically increase the number of input channels to enhance data representation; such data expansion raises the amount of operations needed and it is feasible only on systems with enough computational resources. In this work, we present a new method to binarize the first layer using directly the 8-bit representation of input data; we exploit the standard bit-planes encoding to extract features bit-wise (using depth-wise convolutions); after a re-weighting stage, features are fused again. The resulting model is fully binarized and our first layer binarization approach is model independent. The concept is evaluated on three classification datasets (CIFAR10, SVHN and CIFAR100) for different model architectures (VGG and ResNet) and, the proposed technique outperforms state of the art methods both in accuracy and BMACs reduction.
E-Patcher: A Patch-Based Efficient Network for Fast Whole Slide Images Segmentation

UNeXt is a leading medical image segmentation method that employs convolutional and multi-layer perceptron (MLP) structure for its segmentation network. It outperforms other image recognition algorithms, such as MedT and TransUNet, in terms of faster computation speed and has shown great potential for clinical applications. However, its reliance on limited pixel neighborhood information for pixel-level segmentation of large pathological images may lead to inaccurate segmentation of the entire image and overlook important features, resulting in suboptimal segmentation results. To this end, we designed a slight and universal plug-and-play block based patch-level for fully considering local features and global features simultaneously, named “Digging” and “Filling” ViT (DF-ViT) block. Specifically, a “Digging” operation is introduced to randomly select sub-blocks from each sub-patch. Multi-Head Attention (MHA) is then applied to integrate global information into these sub-blocks. The resulting sub-blocks with global semantic features are reassembled into the original feature map, and feature fusion is performed to combine the local and global features. This approach achieves global representation while maintaining a low computational complexity of the model at 0.1424G. Compared to UNeXt, it improves mIoU by 1.07%. Moreover, it reduces the parameter count by 58.50% and the computational workload by 68.09%. Extensive experiments on PAIP 2019 WSI dataset demonstrate that the DF-ViT block significantly enhances computation efficiency while maintaining a high level of accuracy.
Image Captioning for Nantong Blue Calico Through Stacked Local-Global Channel Attention Network

Nantong blue calico, a Chinese folk hand-made printing and dyeing craft, has become one of intangible cultural heritages (ICHs) in China. To inherite and promote the ICH of Nantong blue calico, this study applies the image captioning technology to explaining blue-calico images. For this purpose, a novel image captioning method, called the stacked local-global channel attention network (SLGCAN), is proposed. This new network focuses on extracting important features from blue-calico images so that it can generate more accurate captions for blue-calico images. SLGCAN contains three parts, residual network (ResNet), stacked local-global channel attention module (SLGCAM), and Transformer. First, the pre-trained ResNet-101 model is used to extract rough features from blue-calico images and then, SLGCAM is to obtain the fine-grained information from rough image features. Eventually, SLGCAN adopts Transformer to encode and decode the fine-grained information of blue-calico images to predict the word information for generating accurate image captions. Experiments are conducted on a collected blue-calico image dataset. In experiments, we compare our SLGCAN with baseline models and show that that the proposed model is feasible and effective.
Exploiting Multi-modal Fusion for Robust Face Representation Learning with Missing Modality

Current RGB-D-T face recognition methods are able to alleviate the sensitivity to facial variations, posture, occlusions, and illumination by incorporating complementary information, while they rely heavily on the availability of complete modalities. Given the likelihood of missing modalities in real-world scenarios and the fact that current multi-modal recognition models perform poorly when faced with incomplete data, robust multi-modal models for face recognition that can handle missing modalities are highly desirable. To this end, we propose a multi-modal fusion framework for robustly learning face representations in the presence of missing modalities, using a combination of RGB, depth, and thermal modalities. Our approach effectively blends these modalities together while also alleviating the semantic gap among them. Specifically, we put forward a novel modality-missing loss function to learn the modality-specific features that are robust to missing-modality data conditions. To project various modality features to the same semantic space, we exploit a joint modality-invariant representation with a central moment discrepancy (CMD) based distance constraint training strategy. We conduct extensive experiments on several benchmark datasets, such as VAP RGBD-T and Lock3DFace, and the results demonstrate the effectiveness and robustness of the proposed approach under uncertain missing-modality conditions compared with all baseline algorithms.
Imbalanced Few-Shot Learning Based on Meta-transfer Learning

Few-shot learning is a challenging task that aims to learn to adapt to new tasks with only a few labeled samples. Meta-learning is a promising approach to address this challenge, but the learned meta-knowledge on training sets may not always be useful due to class imbalance, task imbalance, and distribution imbalance. In this paper, we propose a novel few-shot learning method based on meta-transfer learning, which is called Meta-Transfer Task-Adaptive Meta-Learning (MT-TAML). Meta-transfer learning is used to transfer the weight parameters of a pre-trained deep neural network, which makes up for the deficiency of using shallow networks as the feature extractor. To address the imbalance problem in realistic few-shot learning scenarios, we introduce a learnable parameter balance meta-knowledge for each task. Additionally, we propose a novel task training strategy that selects the difficult class in each task and re-samples from it to form the difficult task, thereby improving the model’s accuracy. Our experimental results show that MT-TAML outperforms existing few-shot learning methods by 2–4%. Furthermore, our ablation experiments confirm the effectiveness of the combination of meta-transfer learning and learnable equilibrium parameters.
A Data Augmentation Based ViT for Fine-Grained Visual Classification

Fine-grained visual classification (FGVC) is a fundamental and longstanding problem aiming to recognize objects belonging to different subclasses accurately. Unfortunately, since categories are often confused, this task is genuinely challenging. Most previous methods solve this problem in two main ways: adding more annotations or constructing more complex structures. These approaches, however, require expensive labels or sophisticated designs. To alleviate these constraints, in this work, we propose an easy but efficient method called DA-ViT, just using data augmentations to supervise the model. Specifically, we adopt a vision transformer as the backbone. Then, we introduce highly interpretable visual heatmaps to guide the targeted data augmentations, and three methods (local area enlargement, flipping, and cutout) are created based on the high-response areas. Furthermore, the margins among confusing classes can be increased by simply using label smoothing. Extensive experiments conducted on three popular fine-grained benchmarks demonstrate that we achieve SOTA performance. Meanwhile, during the inference, our method requires less computational burden.