Reviewerly/exHarmony huggingface dataset:
Improved Conditional Generative Adversarial Networks for SAR-to-Optical Image Translation

Synthetic aperture radar (SAR) has the potential to operate effectively in all weather conditions, making it a desirable tool in various fields. However, the inability of untrained individuals to visually identify ground cover in SAR images poses challenges in practical applications such as environmental monitoring, disaster assessment, and land management. To address this issue, generative adversarial networks (GANs) have been used to transform SAR images into optical images. This technique is commonly referred to as SAR to optical image translation. Despite its common use, the traditional methods often generate optical images with color distortion and blurred contours. Therefore, a novel approach utilizing conditional generative adversarial networks (CGANs) is introduced as an enhanced method for SAR-to-optical image translation. A style-based calibration module is incorporated, which learns the style features of the input SAR images and matches them with the style of real optical images to achieve color calibration, thereby minimizing the differences between the generated output and real optical images. Furthermore, a multi-scale strategy is incorporated in the discriminator. Each branch of the multi-scale discriminator is dedicated to capturing texture and edge features at different scales, thereby enhancing the texture and edge information of the image at both local and global levels. Experimental results demonstrate that the proposed approach surpasses existing image translation techniques in terms of both visual effects and evaluation metrics.
Broaden Your Positives: A General Rectification Approach for Novel Class Discovery

Novel category discovery (NCD), which is a challenging and emerging task, aims to cluster unlabelled instances with knowledge information transferred from labelled ones. A majority of recent state-of-the-art methods leverage contrastive learning to model labelled and unlabelled data simultaneously. Nevertheless, they suffer from inaccurate and insufficient positive samples, which are detrimental to NCD and even its generalized class discovery (GCD) setting. To solve this problem, we propose positive-augmented contrastive learning (PACL), which can mine more positive samples and additional pseudo-positive samples, while augmenting the loss cost corresponding to these positive pairs. Consequently, PACL alleviates the imbalance between positive and negative pairs in contrastive learning, and facilitates the knowledge transfer for novel class discovery. In addition, we develop a general feature rectification approach based on PACL to rectify the representation learning achieved by existing NCD or GCD models. Extensive experiments on three datasets exhibit the necessity and effectiveness of our approach for both NCD and GCD tasks, without loss of generality.
SemanticCrop: Boosting Contrastive Learning via Semantic-Cropped Views

Siamese-structure-based contrastive learning has shown excellent performance in learning visual representations due to its ability to minimize the distance between positive pairs and increase the distance between negative pairs. Existing works mostly employ RandomCrop or ContrastiveCrop to obtain positive pairs of an image. However, RandomCrop causes the cropped views to contain many useless backgrounds, while ContrastiveCrop produces positive pairs that are too similar. In this paper, we propose a novel SemanticCrop to yield cropped views containing as much semantic information as possible. Specifically, SemanticCrop first computes a heatmap of an image. Then, an empirical threshold is tuned to box out a semantic region whose heatmap values are over this threshold. Finally, we design a center-suppressed probabilistic sampling to avoid excessive similarity between positive pairs, making the cropped view contain more parts of an object. As a plug-and-play module, the MoCo, SimCLR, SimSiam, and BYOL models equipped with our SemanticCrop module achieve an accuracy improvement from 0.5% to 2.34% on the CIFAR10, CIFAR100, IN-200, and IN-1K datasets. The code is available at https://github.com/GZHU-DVL/SemanticCrop .
A Transformer-Based Adaptive Semantic Aggregation Method for UAV Visual Geo-Localization

This paper addresses the task of Unmanned Aerial Vehicles (UAV) visual geo-localization, which aims to match images of the same geographic target taken by different platforms, i.e., UAVs and satellites. In general, the key to achieving accurate UAV-satellite image matching lies in extracting visual features that are robust against viewpoint changes, scale variations, and rotations. Current works have shown that part matching is crucial for UAV visual geo-localization since part-level representations can capture image details and help to understand the semantic information of scenes. However, the importance of preserving semantic characteristics in part-level representations is not well discussed. In this paper, we introduce a transformer-based adaptive semantic aggregation method that regards parts as the most representative semantics in an image. Correlations of image patches to different parts are learned in terms of the transformer’s feature map. Then our method decomposes part-level features into an adaptive sum of all patch features. By doing this, the learned parts are encouraged to focus on patches with typical semantics. Extensive experiments on the University-1652 dataset have shown the superiority of our method over the current works.
Federated Learning Based on Diffusion Model to Cope with Non-IID Data

Federated learning is a distributed machine learning paradigm that allows model training without centralizing sensitive data in a single place. However, non independent and identical distribution (non-IID) data can lead to degraded learning performance in federated learning. Data augmentation schemes have been proposed to address this issue, but they often require sharing clients’ original data, which poses privacy risks. To address these challenges, we propose FedDDA, a data augmentation-based federated learning architecture that uses diffusion models to generate data conforming to the global class distribution and alleviate the non-IID data problem. In FedDDA, a diffusion model is trained through federated learning and then used for data augmentation, thus mitigating the degree of non-IID data without disclosing clients’ original data. Our experiments on non-IID settings with various configurations show that FedDDA significantly outperforms FedAvg, with up to 43.04% improvement on the Cifar10 dataset and up to 20.05% improvement on the Fashion-MNIST dataset. Additionally, we find that relatively low-quality generated samples that conform to the global class distribution still improve federated learning performance considerably.
Circle Representation Network for Specific Target Detection in Remote Sensing Images

Compared with natural images, remote sensing images have complex backgrounds as well as a variety of targets. The circular and square-like targets are very common in remote sensing images. For such specific targets, it is easy to bring background information when using the traditional bounding box. To address this issue, we propose a Circle Representation Network (CRNet) to detect the circular or square-like targets. We design a special network head to regression radius and it has smaller regression degrees of freedom. Then the bounding circle is proposed to represent the specific targets. Compared to the bounding box, the bounding circle has natural rotational invariance. The CRNet can accurately locate the object while carrying less background information. In order to reasonably evaluate the detection performance, we further propose the circle-IOU to calculate the mAP. The experiments evaluated on NWPU VHR-10 and RSOD datasets show that the proposed method has excellent performance when detecting circular and square-like objects, in which the detection accuracy of storage tanks is improved from 92.1% to 94.4%. Therefore, the CRNet is a simple and efficient detection method for the circular and square-like targets.
Global Consistency Enhancement Network for Weakly-Supervised Semantic Segmentation

Generation methods for reliable class activation maps (CAMs) are essential for weakly-supervised semantic segmentation. These methods usually face the challenge of incomplete and inaccurate CAMs due to intra-class inconsistency of final features and inappropriate use of deep-level ones. To alleviate these issues, we propose the Global Consistency Enhancement Network (GCENet) that consists of Middle-level feature Auxiliary Module (MAM), Intra-class Consistency Enhancement Module (ICEM), and Critical Region Suppression Module (CRSM). Specifically, MAM uses middle-level features which carry clearer edges information and details to enhance output features. Then, for the problem of incomplete class activation maps caused by the high variance of local context of the image, ICEM is proposed to enhance the representation of features. It takes into account the intra-class global consistency and the local particularity. Furthermore, CRSM is proposed to solve the problem of excessive CAMs caused by the over-activation of features. It activates the low-discriminative regions appropriately, thus improving the quality of class activation maps. Through our comprehensive experiments, our method outperforms all other competitors and well demonstrates its effectiveness on the PASCAL VOC2012 dataset.
A Survey: The Sensor-Based Method for Sign Language Recognition

Sign language is a crucial communication carrier among deaf people to express and exchange their thoughts and emotions. However, ordinary individuals cannot acquire proficiency in sign language in the short term, which leads to deaf people facing huge barriers with the sound community. Regarding this conundrum, it is valuable to investigate Sign Language Recognition (SLR) equipped with sensors which collect data for the following computer vision processing. This study has reviewed the sensor-based SLR methods, which can transform heterogeneous signals from various underlying sensors into high-level motion representations. Specifically, we have summarized current developments in sensor-based SLR techniques from the perspective of modalities. Addtionally, we have also distilled the sensor-based SLR paradigm and compared the state-of-the-art works, including computer vision. Following that, we have concluded the research opportunities and future work expectations.
A Real-Time Network for Fast Breast Lesion Detection in Ultrasound Videos

Breast cancer stands as the foremost cause of cancer-related deaths among women worldwide. The prompt and accurate detection of breast lesions through ultrasound videos plays a crucial role in early diagnosis. However, existing ultrasound video lesion detectors often rely on multiple adjacent frames or non-local temporal fusion strategies to enhance performance, consequently compromising their detection speed. This study presents a simple yet effective network called the Space Time Feature Aggregation Network (STA-Net). Its main purpose is to efficiently identify lesions in ultrasound videos. By leveraging a temporally shift-based space-time aggregation module, STA-Net achieves impressive real-time processing speeds of 54 frames per second on a single GeForce RTX 3090 GPU. Furthermore, it maintains a remarkable accuracy level of 38.7 mean average precision (mAP). Through extensive experimentation on the BUV dataset, our network surpasses existing state-of-the-art methods both quantitatively and qualitatively. These promising results solidify the effectiveness and superiority of our proposed STA-Net in ultrasound video lesion detection.
Deep Multi-task Image Clustering with Attention-Guided Patch Filtering and Correlation Mining

Deep Multi-task image clustering endeavors to leverage deep learning techniques for the simultaneous processing of multiple clustering tasks. Current multi-task deep image clustering approaches typically rely on conventional deep convolutional neural networks and transfer learning technologies. However, suboptimal clustering results are produced in the execution of each task including irrelevant redundant information. This paper proposes a novel end-to-end deep multi-task clustering framework named Deep Multi-Task Image Clustering with Attention-guided Patch Filtering and Correlation Mining (APFMTC) that eliminates redundant information between different tasks while extracting relevant information to achieve improved cluster division. Specifically, APFMTC partitions image samples into several patches, treating each patch as a word thus each image is regarded as an article, and the process of determining the cluster to which an image belongs is likened to categorizing articles. During the clustering process, several parts of each image sample generally carry more significance. Therefore, a weights estimation module is designed to evaluate the importance of different visual words extracted by the key patch filter for different categories. Ultimately, in each task, the final cluster division is determined by assigning weights to the words contained within the image samples. To evaluate the effectiveness of the proposed method, it is tested on multi-task datasets created from four datasets: NUS-Wide, Pascal VOC, Caltech-256, and Cifar-100. The experimental results substantiate the efficacy of the proposed method.
Multi-head Attention Induced Dynamic Hypergraph Convolutional Networks

Hypergraph neural networks (HGNNs) have recently attracted much attention from researchers due to the powerful modeling ability. Existing HGNNs usually derive the data representation by capturing the high-order adjacent relations in a hypergraph. However, incomplete exploration and exploitation of hypergraph structure result in the deficiency of high-order relations among the samples. To this end, we propose a novel hypergraph convolutional networks (M-HGCN) to capture the latent structured properties in a hypergraph. Specifically, two novelty designs are proposed to enhance the expressive capability of HGNNs. (1) The CNN-like spatial graph convolution and self-adaptive hypergraph incidence matrix are employed to capture both the local and global structural properties in a hypergraph. (2) The dual-attention scheme is applied to hyperedges, which can model the interactions across multiple hyperedges to form hypergraph-aware features. The experimental results on the benchmark citation network datasets demonstrate the superior performance of the proposed method over the existing strong baselines.
A Physically Feasible Counter-Attack Method for Remote Sensing Imaging Point Clouds

This research introduces an innovative approach to address the vulnerability of deep learning-based point cloud target recognition systems to adversarial sample attacks. Instead of directly tampering with the spatial location information of the point cloud data, the approach focuses on modifying specific attribute information. The modified point cloud data is then associated with a simulated physical environment for comprehensive attack testing. The experimental findings demonstrate the aggressive nature of the generated counter-samples achieved through signal amplitude modulation, effectively deceiving deep learning-based target recognition systems. The experimental results highlight the effectiveness of the adversarial object samples generated by modifying the signal amplitude of the point cloud data, showcasing their strong misguiding capabilities towards the deep learning-based target recognition algorithm. This approach ensures good stealthiness and practicality, making it a viable attack method applicable in physical scenarios.
PVT-Crowd: Bridging Multi-scale Features from Pyramid Vision Transformer for Weakly-Supervised Crowd Counting

Weakly-supervised crowd counting does not require location-level annotations, but only relies on count-level annotations to achieve the task of crowd counting for images, which is becoming a new research hotspot in the field of crowd counting. Currently, weakly-supervised crowd counting networks based on deep learning mostly use Transformers to extract features and establish global contexts, ignoring feature information at different scales, resulting in insufficient feature utilization. In this paper, we propose a well-designed end-to-end crowd counting network named PVT-Crowd bridging multi-scale features from the Pyramid Visual Transformer Encoder for weakly-supervised crowd counting. Specifically, Adjacent-Scale Bridging Modules (ASBM) enable the interaction of high-scale semantic and low-scale detailed information from both channel and spatial dimensions. The Global-Scale Bridging Module (GSBM) performs a secondary fusion of multi-scale feature information. Extensive experiments show that our PVT-Crowd outperforms most weakly-supervised crowd counting networks and obtains competitive performance compared to fully-supervised ones. In particular, cross-dataset experiments confirm that our PVT-Crowd had a remarkable generality.
Improving Transferability of Adversarial Attacks with Gaussian Gradient Enhance Momentum

Deep neural networks (DNNs) can be susceptible to subtle perturbations that may mislead the model. While adversarial attacks are successful in the white-box setting, they are less effective in the black-box setting. To address this issue, we propose an attack method that simulates a smoothed loss function by sampling from a Gaussian distribution. We calculated the Gaussian gradient to enhance the momentum based on the smoothing loss function to improve the transferability of the attack. Moreover, We further improve transferability by changing the sampling range to make the Gaussian gradient prospective. Our method has been extensively tested through experiments, and the results show that it achieves higher transferability compared to state-of-the-art (SOTA) methods.
EVCPP:Example-Driven Virtual Camera Pose Prediction for Cloud Performing Arts Scenes

Studying intelligent virtual shooting in cloud performing arts scenes is of great significance to the sustainable development of the cloud performing arts industry. Difficulties in summarizing the language of shots and the need to consider the attributes of cameras and actor movements are key concerns in stage filming. We propose EVCPP: Example-driven Virtual Camera Pose Prediction for cloud performing arts that uses existing shooting videos to guide the shooting of virtual scenes. By using the camera behavior information from reference videos as external guidance weights, along with the camera intrinsic parameters and actor state information as covariates, we combine them with the historical pose of the camera in the virtual scene to predict the future camera pose. Meanwhile, we propose a combined loss function for our task. Our method has achieved promising results in virtual 3D cloud performing arts scenes.
Cross-Dataset Distillation with Multi-tokens for Image Quality Assessment

No Reference Image Quality Assessment (NR-IQA) aims to accurately evaluate image distortion by simulating human assessment. However, this task is challenging due to the diversity of distortion types and the scarcity of labeled data. To address these issues, we propose a novel attention distillation-based method for NR-IQA. Our approach effectively integrates knowledge from different datasets to enhance the representation of image quality and improve the accuracy of predictions. Specifically, we introduce a distillation token in the Transformer encoder, enabling the student model to learn from the teacher across different datasets. By leveraging knowledge from diverse sources, our model captures essential features related to image distortion and enhances the generalization ability of the model. Furthermore, to refine perceptual information from various perspectives, we introduce multiple class tokens that simulate multiple reviewers. This not only improves the interpretability of the model but also reduces prediction uncertainty. Additionally, we introduce a mechanism called Attention Scoring, which combines the attention-scoring matrix from the encoder with the MLP header behind the decoder to refine the final quality score. Through extensive evaluations of six standard NR-IQA datasets, our method achieves performance comparable to the state-of-the-art NR-IQA approaches. Notably, it achieves SRCC values of 0.932 (compared to 0.892 in TID2013) and 0.964 (compared to 0.946 in CSIQ).
Probability-Based Nuclei Detection and Critical-Region Guided Instance Segmentation

Nuclear instance segmentation in histopathological images is a key procedure in pathological diagnosis. In this regard, a typical class of solutions are deep learning-like nuclear instance segmentation methods, especially those based on the detection of nuclear critical regions. The existing instance segmentation methods based on nuclear critical regions are still insufficient in detecting and segmenting adhesion nuclei. In this study, we proposed a Critical-Region Guided Instance Segmentation (CGIS) method to precisely segment adhesion nuclei. Specifically, CGIS embed the critical region of an instance into the original image to guide the model to segment only the target instance, and provide an accurate segmentation mask for each nucleus. To improve the accuracy of critical region detection in CGIS, we proposed a boundary-insensitive feature, Central Probability Field (CPF). The CPF is calculated based on global morphological characteristics of nuclei, and can reduce the negative effects from the unreliable nuclear boundary details on the nuclear critical regions detection. We evaluated the effectiveness of the proposed CGIS and CPF feature on the Lizard and PanNuke datasets. It is shown that the proposed CGIS method can accurately segment adhesion nuclei, and the CPF feature can efficiently detect the critical regions of nuclei.
Prototypical Transformer for Weakly Supervised Action Segmentation

Weakly supervised action segmentation aims to recognize the sequential actions in a video, with only action orderings as supervision for model training. Existing methods either predict the action labels to construct discriminative losses or segment the video based on action prototypes. In this paper, we propose a novel Prototypical Transformer (ProtoTR) to alleviate the defects of existing methods. The motivation behind ProtoTR is to further enhance the prototype-based method with more discriminative power for superior segmentation results. Specifically, the Prediction Decoder of ProtoTR translates the visual input into action ordering while its Video Encoder segments the video with action prototypes. As a unified model, both the encoder and decoder are jointly optimized on the same set of action prototypes. The effectiveness of the proposed method is demonstrated by its state-of-the-art performance on different benchmark datasets.