Reviewerly/exHarmony huggingface dataset:
Adaptive Semi-Supervised Mixup with Implicit Label Learning and Sample Ratio Balancing

Despite the impressive performance of deep neural networks, they are prone to over-fitting at labeled points rooting from the scarcity of annotated data. Applying mixup regularization in training provides an effective mechanism to improve generalization performance. On the other hand, semi-supervised learning(SSL) leverages an abundant amount of unlabeled data along with a small amount of labeled data in the training process. In this paper, we have introduced mixup regularization to SSL, along with an exploration-utilization training scheme to enhance the performance. Besides, due to the large volume imbalance between labeled/unlabeled data and the unwanted noise resulting from unlabeled samples, we also implement a balancing ratio between the labeled/unlabeled loss terms. Specifically, we devise a novel Sharp Entropy loss for model optimization with large-scale unlabeled samples and employ an uncertainty estimation technique to weigh unlabeled loss function. Extensive experiments show the state-of-the-art performance of SEMixup and uncertainty balancing ratio superior to baselines on image classification.
MCTE: Marrying Convolution and Transformer Efficiently for End-to-End Medical Image Segmentation

The purpose of the medical image segmentation task is to delineate different organs or lesion regions in the image, which is an important aid for intelligent clinical medical diagnosis. Recent approaches suffer from the inability to obtain reliable attention, are computationally intensive, and do not exploit the relationships between different samples. We marry convolution and Transformer effectively to establish MCTE for medical image segmentation. The proposed MCTE is an end-to-end network based on U-Net with a parallel learning of three types of attention, namely local attention learning with channel and spatial dimensional convolution, global attention learning with smaller computational effort of swin transformer, and external attention learning with two shared memory storing all medical image information. Extensive experimental results on the ACDC and Synapse dataset, which are widely used for the evaluation of medical image segmentation methods, demonstrate that our proposed method exceeds the compared baseline.
Multi-Dimensional Pruned Sparse Convolution for Efficient 3D Object Detection

In recent years, significant progress has been made in 3D object detection. The focus of research has primarily been on improving the detection accuracy of models, however, neglecting their efficiency during actual deployment. Aiming at this issue, in this paper, we propose a multi-dimensional pruning method from the perspectives of data and model. Specifically, given the input data represented by the voxel grid, we first measure the voxel importance and propose an importance-based sampling module to sparsify voxels while preserving informative ones. The model pruning is wrapped in the framework of weighted voxel distillation, where the student model is obtained by pruning the channels of teacher model and only the informative voxels in the teacher model are involved and transferred to the students. In addition, the proposed method can be seamlessly integrated into current voxel-based 3D detectors without any additional costs. Experimental results on the KITTI and ONCE datasets show that our method can achieve a reduction of over 80% in GFLOPs while maintaining superior performance.
Semi-Supervised Few-Shot Segmentation with Noisy Support Images

Motivated by the semi-supervised learning that uses the unlabeled data and pseudo annotations to improve the image classification, this paper proposes a new semi-supervised few-shot segmentation (FSS) framework of which the training process uses not only the annotated images, but also the unlabeled images, e.g. images from other available datasets, to enhance the training of the FSS model. Furthermore, in the test phase, more support images and pseudo-annotations can also be generated by the proposed framework to enrich the support set of novel classes and therefore benefit the inference. However, unlabeled images are not a free lunch. The noisy intra-class samples and inter-class samples existed in the unlabeled images as well as the interferences of the bad quality of pseudo annotations make it difficult to utilize the correct images and pseudo annotations for a certain class. To this end, we further propose a ranking algorithm consisting of an inter-class confidence term and an intra-class confidence term to efficiently utilize the pseudo annotations of the class with high quality. Extensive experiments on COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> dataset demonstrate that the proposed semi-supervised FSS framework is superior to many state-of-the-art methods.
Skeleton-based Human Action Recognition in Low-Resolution Infrared Images

This paper proposes a recognition method of human action based on human skeleton extraction in low resolution infrared images, which are collected by two thermopile infrared array deployed on the side. Each frame of image is processed sequentially through quantification, interpolation and background removal operations, and the original human skeleton data is then extracted using a thinning algorithm. A standard human skeleton model is established and fitted to acquire the standard human skeleton data. Finally, the continuous human action is divided into several separate actions, and an LSTM network is used for human action recognition. The experimental results have demonstrated that the proposed method can achieve an accurate recognition of human actions.
Fisheye Multiple Object Tracking by Learning Distortions Without Dewarping

We develop a new Multiple Object Tracking (MOT) scheme for fisheye cameras that can directly perform vehicle detection, re-identification, and tracking under fisheye distortions without explicit dewarping. Fisheye cameras provide omnidirectional coverage that is wider than traditional cameras, reducing fewer need of cameras to monitor road intersections. However, the problem of distorted views introduces new challenges for fisheye MOT. In this paper, we propose a Fish-Eye Multiple Object Tracking (FEMOT) approach with two novelties. We develop the Distorted Fisheye Image Augmentation (DFIA) method to improve object detection and re-identification on fisheye cameras, where fisheye model training can be performed on existing datasets of traditional cameras via fisheye data synthesis and augmentation. We also develop the Hybrid Data Association (HDA) method to perform tracking directly on fisheye views, without the need of de-warping. The developed FEMOT framework provides practical design and advancement that enables large-scale use of fisheye cameras in smart city and surveillance applications.
Combining Self-Supervised and Supervised Learning with Noisy Labels

Since convolutional neural networks (CNNs) can easily overfit noisy labels, which are ubiquitous in visual classification tasks, it has been a great challenge to train CNNs against them robustly. Various methods have been proposed for this challenge. However, none of them pay attention to the difference between representation and classifier learning of CNNs. Thus, inspired by the observation that classifier is more robust to noisy labels while representation is much more fragile, and by the recent advances of self-supervised representation learning (SSRL) technologies, we design a new method, i.e., CS <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">3</sup> NL, to obtain representation by SSRL without labels and train the classifier directly with noisy labels. Extensive experiments are performed on both synthetic and real benchmark datasets. Results demonstrate that the proposed method can beat the state-of-the-art ones by a large margin, especially under a high noisy level.
Film Grain Removal Using Metadata

Film grain has been shown as an effective way to improve the look of video in terms of aesthetic feeling and sharpness. For backward compatibility, a strategy is to inject the film grain before video compression at encoder to ensure that any decoder can acquire the film grain injected images after video decompression. Moreover, for designed decoder, the film grain can be removed, and further video processing can be applied, such as adding other type of film grain or adaptive film grain setting according to the viewing environment. Therefore, in this work, we propose a metadata-aided film grain removal system such that the film grain can be removed efficiently using metadata in video bitstream. We propose three filtering methods (Gaussian, optimal FIR, and guided filter) in the proposed system. Experimental results show that the proposed system can remove film grain effectively and yield high PSNR.
Continual Learning for Out-of-Distribution Pedestrian Detection

A continual learning solution is proposed to address the out-of-distribution generalization problem for pedestrian detection. While recent pedestrian detection models have achieved impressive performance on various datasets, they remain sensitive to shifts in the distribution of the inference data. Our method adopts and modifies Elastic Weight Consolidation to a backbone object detection network, in order to penalize the changes in the model weights based on their importance towards the initially learned task. We show that when trained with one dataset and fine-tuned on another, our solution learns the new distribution and maintains its performance on the previous one, avoiding catastrophic forgetting. We use two popular datasets, CrowdHuman and CityPersons for our cross-dataset experiments, and show considerable improvements over standard fine-tuning, with a 9% and 18% miss rate percent reduction improvement in the CrowdHuman and CityPersons datasets, respectively.
Odd: One-Class Anomaly Detection Via The Diffusion Model

Anomaly detection identifies instances that deviate the distribution of the normal class. Recently, the diffusion models have shown great promise. Our research revealed that by training the diffusion model solely on normal data, it is able to transform both normal and anomalous samples into normal images. Employing this discovery, we propose ODD (One-Class Anomaly Detection via the Diffusion model), which consists of: a diffusion model to convert both normal and anomalous data into normal data, and a similarity network enhanced with outlier exposure to measure the semantic distance between the input and output of the diffusion model. If the score is low, the input is considered as an anomaly instance. The ODD is evaluated on a variety of datasets. Both qualitative and quantitative results demonstrate that our method outperforms existing state-of-the-art techniques.
ScratchHOI: Training Human-Object Interaction Detectors from Scratch

Transformer-based approaches have exhibited outstanding performances in the field of human-object interaction (HOI) detection. However, these approaches rely on underlying object detectors that have undergone large-scale pre-trainings on the ImageNet and MS-COCO dataset. This limits the potential of unique architectural designs and induces a learning bias, causing ineffective HOI representation learning. In this paper, we propose ScratchHOI, a transformer-based method for human-object interaction detection that can be trained from scratch, eliminating the need for pre-trained object detectors. ScratchHOI employs dynamic and static affinity-based feature aggregation for processing local and long-range visual information. Additional techniques are also employed to improve detection performance, such as dynamic and interactive anchor refinement for objects and interactions. Experiments on the HICO-Det dataset show that ScratchHOI achieves competitive performance against other state-of-the-art approaches over a variety of different evaluation measures.
Realization of Digraph Filters Via Augmented GFT

This study proposes a filtering method for directed graph (digraph) signals. In order to realize digraph filtering, a novel graph Fourier transform (GFT), – Augmented GFT (AuGFT) –, is proposed by defining an Hermitian adjacency matrix. Although there has been the same method to give the adjacency matrix of digraphs, this study defines a novel digraph Laplacian. The existing digraph Laplacian does not give the graph signal variation considering the edge directions, while the novel one does. This paper introduces three important ideas. The first is the definition of a novel degree matrix to give the novel digraph Laplacian. The second is to decompose the symmetric and skew-symmetric components of the novel digraph Laplacian independently into their spectral components. The third is, based on the decomposition, to augment the conventional GFT for digraphs as an invertible real-valued dictionary. The new GFT is shown to provide a practical form of real-valued digraph filtering. The significance of the proposed method is verified through simulations of signal filtering on digraphs.
Comparison of artificial neural network adaptive control techniques for a nonlinear system with delay

This research paper compares two neural-network-based adaptive controllers, namely the Hybrid Deep Learning Neural Network Controller (HDLNNC) and the Adaptive Model Predictive Control with Nonlinear Prediction and Linearization along the Predicted Trajectory (AMPC-NPLPT), for controlling a nonlinear object with delay. Specifically, the study investigates the effect of delay on the accuracy of the two controllers. The experimental results demonstrate that the AMPC-NPLPT approach outperforms HDLNNC regarding control accuracy for the given nonlinear object control problem.
Abnormal-Aware Loss and Full Distillation for Unsupervised Anomaly Detection Based on Knowledge Distillation

Knowledge Distillation based Unsupervised Anomaly Detection (KD-UAD), which aims to detect anomalies according to the differences between features of student network and teacher network on the sample, has been widely concerned by researchers. However, the strong learning ability of student networks may lead to small differences on abnormal samples, making it impossible to distinguish between anomalies and non-anomalies. To alleviate the above problem, we propose an improved KD-UAD approach to enhance the network’s ability to perceive anomalies. Firstly, we propose abnormal-aware loss (AAL), which allows student network to gain anomaly repair capabilities. AAL enhances the differences between the features extracted by the student and teacher network on abnormal samples. Secondly, we design a full distillation loss (FDL) to enhance distillation strength. FDL allows the student network to learn the feature distribution more comprehensively. Experimental results show that our method outperforms the current state-of-the-art methods on the MVTec AD and BTAD datasets.
Learnable Snake R-CNN for Instance-Level Biomedical Image Segmentation

Precisely knowing each instance’s position and extents is a critical first step in many biological applications. State-of-the-art techniques rely either on deep learning models designed to predict segmentation masks on each Region of Interest (RoI) or on classic active contour methods. The former struggles to precisely delineating boundaries and tends to output masks at low resolutions when the cells/nuclei are very irregular while the latter often needs good initialization and manual setting of parameters, thus limiting their usefulness. To bridge this gap, we introduce Snake R-CNN, a new level of the learnable active contour model that predict boundary on each RoI in a sequent way. To do so, for each RoI, we reformulate the contour deformation task in terms of a hidden state evolution problem and update the evolution process using energy minimization. We learn snake parameterizations per instance in an end-to-end manner, and demonstrate its effectiveness for contour inferences of various cell/nucleus types where consistently higher performances were obtained for comparison against state-of-the-arts.
GNP Attack: Transferable Adversarial Examples Via Gradient Norm Penalty

Adversarial examples (AE) with good transferability enable practical black-box attacks on diverse target models, where insider knowledge about the target models is not required. Previous methods often generate AE with no or very limited transferability; that is, they easily overfit to the particular architecture and feature representation of the source, white-box model and the generated AE barely work for target, black-box models. In this paper, we propose a novel approach to enhance AE transferability using Gradient Norm Penalty (GNP). It drives the loss function optimization procedure to converge to a flat region of local optima in the loss landscape. By attacking 11 state-of-the-art (SOTA) deep learning models and 6 advanced defense methods, we empirically show that GNP is very effective in generating AE with high transferability. We also demonstrate that it is very flexible in that it can be easily integrated with other gradient based methods for stronger transfer-based attacks.
Detecting Stable Diffusion Generated Images Using Frequency Artifacts: A Case Study on Disney-Style Art

The use of Stable Diffusion models to generate realistic images has become a popular topic in recent years. However, this technology has also raised concerns about the potential harm it may cause to the copyright holders, particularly in the realm of art where these synthesized images can closely resemble the original work. As these synthesized images are hard for humans to distinguish from authentic ones, it is of great importance to develop methods that may identify them. In this paper, we propose a deep learning-based approach to detect synthesized images using information in the frequency domain. Since there exists no well-established dataset of images synthesized by stable diffusion models, in order to train and evaluate our network we generated a representative dataset consisting of carefully selecting human-created authentic images and synthesized animation images generated by the Stable Diffusion models. We chose to use Disney-style animated content for our case study, given its significance in the realm of intellectual property protection. Experimental results demonstrated that our proposed model outperforms humans and other state-of-the-art methods, achieving an accuracy rate of 99.46%.
Unsupervised Domain Adaptive Person Re-Identification with Adaptive Structure Learning

Unsupervised domain adaptive person re-identification has garnered considerable attention due to its practical significance. Previous research has focused on utilizing the teacher-student framework within the clustering and fine-tuning paradigm to reduce the domain gap between different person re-identification datasets. Building upon these recent advancements, we propose a novel approach to further explore the structural information of the teacher and student networks from multiple perspectives, including adaptive sample updating, global feature arrangement, and local discrepancy learning. By integrating these three components, we introduce the Adaptive Structure Learning Framework (ASL) for unsupervised domain adaptive person re-identification. Compared to the baseline method, our approach achieves a significant improvement of 7.3% in mean Average Precision (mAP) on the Market2MSMT task. Furthermore, our experimental results across three benchmark datasets provide further evidence of the effectiveness of our proposed method.
CR-UNIT: Unsupervised Image-to-Image Translation with Content Reconstruction

The goal of unsupervised image-to-image translation (UNIT) is to translate images between two different domains without any paired data. Recent research has shown great progress in the UNIT subject, but it still faces challenges when translating between artificial and natural objects. To overcome this challenge, we propose a novel unsupervised image-to-image translation with content reconstruction (CR-UNIT), a two-stage and from-coarse-to-fine training framework. Specifically, CR-UNIT builds a corresponding relationship among the content features of different domains on a coarse granularity in the first stage. In the second stage, a new content reconstruction module is constructed to extract the fine-grained content and style features, obtaining more detailed semantic correspondence and better fusion of the content and style features. Furthermore, we design a content reconstruction loss to facilitate the training of our model. Extensive experimental results demonstrate the superiority of the proposed CR-UNIT over the existing methods. Especially for the translation task between the artificial and natural objects, our CR-UNIT achieves outstanding effect in terms of perceptive quality and objective metric. The code is available at https://github.com/GZHU-DVL/CR-UNIT.
Improving Robustness of Single Image Super-Resolution Models with Monte Carlo Method

Deep learning-based methods have achieved promising results in single image super-resolution (SISR). However, the performance of existing deep SISR methods is very sensitive to image degradation. In addition, these methods are deterministic and do not introduce any uncertainty to the generated images, so we have no way of knowing the reliability of these generated images. To address these two challenging issues, we propose a model-agnostic approach for existing deep SISR networks to improve their robustness under various degradations. Our proposed method follows a probabilistic framework and applies Monte Carlo dropout to existing deep SISR methods. Instead of performing point estimation, the proposed method predicts the posterior distribution of super-resolved images. Based on this, we can determine the uncertainty of the generated images. Experiment results show that the proposed method can effectively improve the robustness of existing deep SISR methods, leading to state-of-the-art performance when applied to images having different degradations. The code is available at https://github.com/YangTracy/MCD-SR.
Single-Image HDR Reconstruction Based on Two-Stage GAN Structure

Because human eyes can capture a wide range of luminance, the high dynamic range (HDR) reconstruction is to expand the low dynamic range (LDR) image to that we actually see. It is challenging to recover an HDR image from a single LDR image due to the missing information in under-/over-exposed regions. In this paper, we proposed a novel two-stage model to settle this problem. The first stage model takes the generative adversarial network (GAN) and the attention mechanism to generate the missing information in under-/over-exposed areas. The second stage is a multi-branch convolutional neural network (CNN) to fuse the multiple different exposure LDR images from the first stage to generate an HDR image. The quantitative comparisons demonstrate that our method achieves higher scores than other state-of-the-art methods. Moreover, our method gene-rates smooth and noise-free HDR images in the qualitative comparisons.
Generalizable Embeddings with Cross-Batch Metric Learning

Global average pooling (GAP) is a popular component in deep metric learning (DML) for aggregating features. Its effectiveness is often attributed to treating each feature vector as a distinct semantic entity and GAP as a combination of them. Albeit substantiated, such an explanation’s algorithmic implications to learn generalizable entities to represent unseen classes, a crucial DML goal, remain unclear. To address this, we formulate GAP as a convex combination of learnable prototypes. We then show that the prototype learning can be expressed as a recursive process fitting a linear predictor to a batch of samples. Building on that perspective, we consider two batches of disjoint classes at each iteration and regularize the learning by expressing the samples of a batch with the prototypes that are fitted to the other batch. We validate our approach on 4 popular DML benchmarks.