Reviewerly/exHarmony huggingface dataset:
Joint Super-Resolution and Classification Based on Bidirectional Mapping and Multiple Constraints

Since discriminant features are insufficient in the low-resolution (LR) images, it is challenging to accurately classify them. To address this issue, this paper proposes a joint super-resolution (SR) and classification network (JSRCN) based on bidirectional mapping and multiple constraints. In JSRCN, there is a SR sub-network containing a forward mapping path and several backward mapping paths. In the forward mapping path, high resolution (HR) features are progressively reconstructed. On the other hand, the backward mapping paths are used to alleviate the hardship of directly learning a nonlinear mapping from the LR space to the HR space. To effectively restore discriminant features for the image classification sub-network, multiple perceptual loss and multi-scale feature loss are presented to enhance the representation ability for the multi-scale features in images with different resolutions. Experiments show that compared with other competing methods, the proposed method achieves the highest accuracy.
Decoupled Mutual Distillation for Incremental Object Detection

Visual object detection aims to localize and classify objects in the images. Popular detectors perform well when all the object categories are pre-defined. However, they show poor performance when the new categories join in the training incrementally and the replay of old training samples is not allowed. This issue is widely known as catastrophic forgetting. In this paper, the detection head of classic Faster R-CNN is decoupled into the classification head and localization head. Based on the modified network, a dynamic mutual distillation framework is designed. The distillation is applied only to the classification head. On the one hand, this design allows the knowledge to be transferred from both the old model and the assistant model to the new. On the other hand, the degree of knowledge-transfer from either the old or assistant model is dynamically determined by their expertise on a certain category. Compared to the conventional teacher-student distillation, our design makes a better balance between the inheritance of knowledge from the old model and adapting to the new categories.
A Compact Transformer for Adaptive Style Transfer

Due to the limitation of spatial receptive field, it is challenging for CNN-based style transfer methods to capture rich and long-range semantic concepts in artworks. Though the transformer provides a fresh solution by considering long-range dependencies, it suffers from the heavy burdens of parameter scale and computation cost especially in vision tasks. In this paper, we design a compact transformer architecture AdaFormer to address the problem. The model scale shrinks about 20% compared to state-of-the-art transformer for style transfer. Furthermore, we explore the adaptive style transfer by letting the content to select the detailed style element automatically and adaptively, which encourages the output to be both appealing and reasonable. We evaluate AdaFormer comprehensively in the experiments and the results have shown the effectiveness and superiority of our approach compared to existing artistic methods. Diverse plausible stylized images are obtained with better content preservation, more convincing style sfumato and lower computation complexity.
LA-layer: General local attention layer for full attention networks

Attention layers have contributed to state-of-the-art results on vision tasks. Still, they leave room for improvement because position information is used in a fixed manner, and the computation cost is typically high. To mitigate both issues, we propose a convolution-style local attention layer (LA-layer) as a replacement for traditional attention layers. LA-layers not only encode the position information of pixels in a convolutional manner, but also produce position offsets following a novel constrained rule so that keys will deform and result in larger receptive fields. Query and keys are processed by a novel aggregation function that outputs attention weights for the values. In our experiments with different types of ResNets, we replace convolutional layers with LA-layers and address image recognition, object detection and instance segmentation tasks. We consistently demonstrate performance gains, despite having fewer FLOPs and training parameters. Our code is available at: https://github.com/hotfinda/LA-layer.
DocMAE: Document Image Rectification via Self-supervised Representation Learning

Tremendous efforts have been made on document image rectification, but how to learn effective representation of such distorted images is still under-explored. In this paper, we present DocMAE, a novel self-supervised framework for document image rectification. Our motivation is to encode the structural cues in document images by leveraging masked autoencoder to benefit the rectification, i.e., the document boundaries, and text lines. Specifically, we first mask random patches of the background-excluded document images and then reconstruct the missing pixels. With such a self-supervised learning approach, the network is encouraged to learn the intrinsic structure of deformed documents by restoring document boundaries and missing text lines. Transfer performance in the downstream rectification task validates the effectiveness of our method. Extensive experiments are conducted to demonstrate the effectiveness of our method.
Irecut+MM: Data Generalization and Metric Improvement for Few-shot Learning

Few-shot learning is intended to address situations where there are few training samples per class. The main challenge is how to adequately extend the data or identify tighter associations from the sparse data. In this study, the final classification and data from few-shot learning are taken into account. First, we provide an efficient regularization to internal network generalization to accomplish data augmentation. Second, to enhance the Nearest Class Mean classifier (NCM) and soft K-means methods for classification, we raise a new regularized estimator based on the concept of Mahalanobis distance. We conduct extensive experiments based on CIFAR-FS and FC100. The ablation experiments show that the proposed data augmentation method is 2% higher than the baseline on CIFAR-FS(1-shot), and the classification algorithm is equally effective. The comparison experiments reveal results that much exceed the vast majority of state-of-the-art performance measures.
Rethinking graph anomaly detection: A self-supervised Group Discrimination paradigm with Structure-Aware

Structural anomalies are the core problem in graph anomaly detection. However, the current mainstream self-supervised graph anomaly detection models do not directly model structural anomalies and their expensive time consumption limits the efficiency of graph anomaly detection. For this reason, we rethink graph anomaly detection and propose a self-supervised Group Discrimination paradigm with Structure-Aware (GDSA). Our model can be explicitly aware of the graph topology changes by multi-view structure disturbance. Moreover, GDSA transforms graph anomaly detection into discriminating the scalar summaries of positive and negative group nodes. The results of extensive experiments on four benchmark datasets show that GDSA outperforms current state-of-the-art methods, with the most significant AUC performance improvement of 28.7%. Notably, in scalability testing on a large-scale dataset, the training time and testing time of GDSA are 1181.0× and 5064.7× faster than the baseline, respectively, with 61.9% savings in memory usage.
Get a Head Start: Targeted Labeling at Source with Limited Annotation Overhead for Semi-Supervised Learning

Semi-supervised learning (SSL), which leverages limited labeled data and a large amount of unlabeled data for model training, has been widely studied to mitigate the requirement for expensive and time-consuming annotations. Recently proposed methods have achieved promising yet unstable results, which presume that initial samples are randomly selected and labeled. For improving the fluctuated performance while saving annotation overhead, effective prior labeling for SSL on the source cluttered unlabeled dataset is challenging but significant. In this paper, we propose a novel criterion and a distribution balance strategy to automatically achieve targeted labeling without access to the test set and any labels. Comprehensive experiments are conducted on commonly-used datasets to demonstrate the effectiveness of our method. Furthermore, targeted labeling is orthogonal to existing framework-centric SSL methods and can achieve state-of-the-art performance.
No Reference Image Quality Assessment Via Quality Difference Learning

For human beings, there is a natural preference for judging the relative quality rather than directly predicting the quality score of an image. Based on this view, we propose an image quality difference learning network (IQDLNet) for evaluating image quality in a no-reference manner. Specifically, the proposed IQDLNet consists of a quality difference-aware network (QDAN) and a quality assessment network (QAN). The QDAN aims to predict the score difference between two randomly matched images and the QAN aims to predict the quality score of these two images. To further enhance the mutual understanding of image semantics, a semantic interaction module (SIM) is proposed with a dual regressor set up to carry out competitive learning in combination with the quality difference-aware feature. Experimental results on five IQA datasets demonstrate the superior performance of the proposed method over eight state-of-the-arts.
Colo-SCRL: Self-Supervised Contrastive Representation Learning for Colonoscopic Video Retrieval

Colonoscopic video retrieval, which is a critical part of polyp treatment, has great clinical significance for the prevention and treatment of colorectal cancer. However, retrieval models trained on action recognition datasets usually produce unsatisfactory retrieval results on colonoscopic datasets due to the large domain gap between them. To seek a solution to this problem, we construct a large-scale colonoscopic dataset named Colo-Pair for medical practice. Based on this dataset, a simple yet effective training method called Colo-SCRL is proposed for more robust representation learning. It aims to refine general knowledge from colonoscopies through masked autoencoder-based reconstruction and momentum contrast to improve retrieval performance. To the best of our knowledge, this is the first attempt to employ the contrastive learning paradigm for medical video retrieval. Empirical results show that our method significantly outperforms current state-of-the-art methods in the colonoscopic video retrieval task.
Self-Attention Prediction Correction with Channel Suppression for Weakly-Supervised Semantic Segmentation

Single-stage weakly-supervised semantic segmentation (WSSS) with image-level labels has become a new research hotspot in the community for its lower cost and higher training efficiency. However, the pseudo label of WSSS generally suffers from somewhat noise, which limits the segmentation performance. In this paper, to explore the integral foreground activation, we propose the Channel Suppression (CS) module for preventing only activating the most discriminative regions, thereby improving the initial pseudo labels. To rectify the in-correct prediction, we explore the Self-Attention Prediction Correction (SAPC) module, which adaptively generates the category-wise prediction rectification weights. After extensive experiments, the proposed efficient single-stage framework achieves excellent performance with 67.6% mIoU and 39.9% mIoU on PASCAL VOC 2012 and MS COCO 2014 datasets, significantly exceeding several recent single-stage methods.
Fine-grained Primitive Representation Learning for Compositional Zero-shot Classification

Compositional zero-shot learning (CZSL) aims to recognize attribute-object compositions that are never seen in the training set. Existing methods solve this problem mainly by learning a single primitive representation for each attribute or object, ignoring the natural intra-attribute or intra-object diversity, i.e., the same attribute (or object) presents dramatically different visual appearance in different compositions. In this paper, we treat the same attribute (or object) under different compositions as fine-grained classes and propose a novel fine-grained primitive representation learning framework to learn more discriminative primitive representations by multiple compact feature sub-spaces. We employ attribute- or object-specific fine-grained primitive prototypes and embed them into a cross-modal space to improve the discriminative ability of primitive representations. Moreover, we leverage semantic-guided sample synthesis to estimate primitive representations of unseen compositions. Extensive experiments on three public benchmarks indicate that our approach can achieve state-of-the-art performance.
Feature Mixing and Disentangling for Occluded Person Re-Identification

Occluded person re-identification (Re-ID) has recently attracted lots of attention for its applicability in practical scenarios. However, previous pose-based methods always neglect the non-target pedestrian (NTP) problem. In contrast, we propose a feature mixing and disentangling method to train a robust network for occluded person Re-ID without extra data. Based on ViT, we design our network as follows: 1) A multi-target patch mixing (MPM) module is proposed to generate complex multi-target images with refined labels in the training stage. 2) We propose an identity-based patch realignment (IPR) module in the decoder layer to disentangle local features from the multi-target sample. In contrast to pose-guided methods, our approach overcomes the difficulties of NTP. More importantly, our approach does not bring additional computational costs in the training and testing phases. Experimental results show that our method effectively on occluded person Re-ID. For example, our method performs 3.3%/3.2% better than the baseline on Occluded-Duke in terms of mAP/rank-1 and outperforms the previous state-of-the-art.
Dynamically Adaptive Instance Normalization and Attention-Aware Incremental Meta-Learning for Generalizable Person Re-identification

Domain generalization person re-identification (DG-ReID) aims to train a generalizable model over several source domains that can perform well on unseen target domains, which makes the DG-ReID task challenging since the model is not allowed to access any target data during training. The classic meta-learning method, which simulates the train-test process of DG-ReID task, is a popular and effective way for DG-ReID. Nevertheless, the method still suffers from the unstable meta-optimization problem. We thus propose a novel Dynamically Adaptive Instance Normalization and Attention-Aware Incremental Meta-Learning (DAIML) optimization method to effectively address this issue. In addition, most existing DG-ReID studies generally utilize Instance Normalization (IN) to learn domain-irrelevant features for eliminating domain shift, which may lead to the removal of effective domain-specific discriminative information that is usually useful to enhance the discriminability of a particular source. Therefore, we propose a dynamically adaptive IN (DAIN) that can balance well the learning of domain-invariant representations and domain-specific discriminative features. Furthermore, we apply channel attention and spatial attention to the proposed DAIN for further improving the domain-irrelevant discriminative information. Extensive experiments demonstrate the effectiveness of our proposed method on several DG-ReID datasets.
Auxiliary Fine-grained Alignment Constraints for Vision-and-Language Navigation

Vision-and-Language Navigation (VLN) requires a visual agent to navigate in photo-realistic environments following instructions. Fine-grained cross-modal alignment is one critical challenge in VLN because the agent needs to focus on a particular sub-part within the complete instruction for the next movement. However, previous work failed to implement explicit supervision for matching the sub-trajectory to the corresponding sub-instruction. In this paper, we propose Auxiliary Fine-grained Alignment Constraints (AFAC) to facilitate decision-making learning during navigation. AFAC consists of two constraints, i.e., Attention Alignment Constraint (AAC) and Representation Alignment Constraint (RAC), which produce additional supervising signals from the perspective of attention and representation respectively. We test our method on the Landmark-RxR benchmark and achieve state-of-the-art results both in seen and unseen environments.
Learning a Multilevel Cooperative View Reconstruction Network for Light Field Angular Super-Resolution

Recently, many methods have been proposed to improve the angular resolution of sparsely-sampled Light Field (LF). However, the synthesized dense LF inevitably exhibits blurry edges and artifacts. This paper intents to model the global relations of LF views and quality degradation model by learning a multilevel cooperative view reconstruction network to further enhance LF angular Super-Resolution (SR) performance. The proposed LF angular SR network consists of three sub-networks including the Cooperative Angular Transformer Network (CATNet), the Deblurring Network (DBNet), and the Texture Repair Network (TRNet). The CATNet simultaneously captures global features of all LF views and local features within each view, which benefits in characterizing the inherent LF structure. The DBNet models a quality degradation model by estimating blur kernels to reduce the blurry edges and artifacts. The TRNet focuses on restoring fine-scale texture details. Experimental results over various LF datasets including large baseline LF images demonstrate the significant superiority of our method when compared with state-of-the-art ones.
Towards General and Fast Video Derain via Knowledge Distillation

As a common natural weather condition, rain can obscure video frames and thus affect the performance of the visual system, so video derain receives a lot of attention. In natural environments, rain has a wide variety of streak types, which increases the difficulty of the rain removal task. In this paper, we propose a Rain Review-based General video derain Network via knowledge distillation (named RRGNet) that handles different rain streak types with one pre-training weight. Specifically, we design a frame grouping-based encoder-decoder network that makes full use of the temporal information of the video. Further, we use the old task model to guide the current model in learning new rain streak types while avoiding forgetting. To consolidate the network’s ability to derain, we design a rain review module to play back data from old tasks for the current model. The experimental results show that our developed general method achieves the best results in terms of running speed and derain effect.
Code Verification Hashing for Image Retrieval

Existing image retrieval deep hashing methods employ full connection as the hash encoding layer and output each bit of hash code in parallel. They regards the hash code as the information of the image, ignoring the relevance between each bit and the redundancy of the whole hash code, resulting in limited performance. Hence, based on the concept of code verification in digital communication (called code verification), the code verification hashing (CVH) is proposed. Different from the exiting methods, CVH constructs hash encoding layer to output the hash code serially, and each output of partial hash code relies on the output of the previous one, so as to make full use of the relevance and redundancy of hash code to generate a more discriminative one. Moreover, CVH also progresses the hash center loss to constraint by all hash centers and makes further improvement. As experiment results, CVH demonstrates excellent retrieval ability.
Improving Vision Transformers with Nested Multi-head Attentions

Vision transformers have significantly advanced the field of computer vision in recent years. The cornerstone of these transformers is the multi-head attention mechanism, which models interactions between visual elements within a feature map. However, the vanilla multi-head attention paradigm independently learns parameters for each head, which ignores crucial interactions across different attention heads and may result in redundancy and under-utilization of the model’s capacity. To enhance model expressiveness, we propose a novel nested attention mechanism, Ne-Att, that explicitly models cross-head interactions via a hierarchical variational distribution. We conducted extensive experiments on image classification, and the results demonstrate the superiority of Ne-Att.
Accurate and Complete Captions for Question-controlled Text-aware Image Captioning

Question-controlled Text-aware Image Captioning (Qc-TextCap), is the task of generating a distinctive scene text aware caption according to several personalized questions when given an image. However, due to the diversity of visual scene, it is hard for current Optical Character Recognition (OCR) systems to extract complete scene text sentences from the images. Besides, existing works are limited in their use of question features and visual features. In this paper, we propose a Multimodal Transformer plus Scene text clustering and Cross modal attention (MTSC) to tackle the above challenges. We devise scene text clustering to group relevant scene text pieces which are detected as separate results by current OCR systems. To better utilize the information in questions and images, we design cross modal attention to enrich the features of both modalities. We extensively evaluate our model on the two Qc-TextCap datasets and superior results are achieved when comparing to state-of-the-art approaches.
A Progressive Gated Attention Model for Fine-Grained Visual Classification

Fine-grained image classification has been a hot research topic in computer vision, pattern recognition, and other fields in recent years. Most existing fine-grained methods use a single attention mechanism or multiple sub-networks to zoom in and find distinguishable local feature regions. These models seldom explore the intrinsic connections between cross-layer features with similar semantic features. This tends to show erratic performance in images with complex backgrounds. To this end, we propose a method focusing on cross-layer feature-context relations. In our method, propose a learning method based on feature channel weights and pyramid patterns, which can increase the diversity of global feature information. Second, we employ cross-layer attention, which can find key target regions. Finally, we propose a cross-gated attention mechanism that can find rich discriminative features from key regions of images. Experiments show that the proposed model performs well on three datasets: CUB-200-2011, Stanford Cars, and FGVC Aircraft.
Knowledge Prompt Makes Composed Pre-Trained Models Zero-Shot News Captioner

News image captioning aims to generate descriptions containing concrete named entities for news images by leveraging relevant news articles. However, existing approaches suffer from two shortcomings: 1) lack of commonsense knowledge required to understand named entities, and 2) limited multimodal context modeling capabilities. In this paper, we propose to migrate the ability of large-scale pre-trained models for news image captioning. To acquire factual knowledge for describing named entities, we induce a pre-trained language model for commonsense knowledge reasoning using context-aware knowledge prompts. To compose a new multimodal context modeling capability, we coordinate pre-trained models by a unified language representation and constrain joint multimodal context reasoning with cross-modal consistency objective. Experimental results on GoodNews and NYTimes datasets show that our proposed method exhibits considerable captioning capabilities even without training on news data.