Reviewerly/exHarmony huggingface dataset:
Text-Guided Facial Image Manipulation for Wild Images via Manipulation Direction-Based Loss

This paper proposes a novel text-guided facial image manipulation approach to improve robustness against the diversity of input images. Conventional text-guided facial image manipulation methods have achieved highly accurate image manipulation by restricting the aspect ratio and composition of the input image. In practical face image manipulation, images provided by the user often include regions other than the face, such as the upper body, and thus conventional methods have limitations in their application. To solve these problems, we tackle a new task of text-guided facial image manipulation with no restrictions on the composition and aspect ratio of the input image. As a base structure, we employ the face swapping method, which is robust to the diversity of input images and allows to swap faces. To achieve text-guided image manipulation based on the face swapping method, we also employ the manipulation direction, which focuses on the direction of changes between the image and its corresponding text that occurs before and after image manipulation. Experimental results show the effectiveness of the proposed method on wild images that include regions other than faces and have different aspect ratios.
Fuzzy-Conditioned Diffusion and Diffusion Projection Attention Applied to Facial Image Correction

Image diffusion has recently shown remarkable performance in image synthesis and implicitly as an image prior. Such a prior has been used with conditioning to solve the inpainting problem, but only supporting binary user-based conditioning.We derive a fuzzy-conditioned diffusion, where implicit diffusion priors can be exploited with controllable strength. Our fuzzy conditioning can be applied pixel-wise, enabling the modification of different image components to varying degrees. Additionally, we propose an application to facial image correction, where we combine our fuzzy-conditioned diffusion with diffusion-derived attention maps. Our map estimates the degree of anomaly, and we obtain it by projecting on the diffusion space. We show how our approach also leads to interpretable and autonomous facial image correction.
Exploring the Connection Between Neuron Coverage and Adversarial Robustness in DNN Classifiers

The lack of robustness in neural network classifiers, especially when facing adversarial attacks, is a significant limitation for critical applications. While some researchers have suggested a connection between neuron coverage during training and vulnerability to adversarial perturbations, concrete experimental evidence supporting this claim is lacking. This paper empirically investigates the impact of maximizing neuron coverage during training and assess the effectiveness of adversarial attacks on under-covered neurons. Additionally, we explore the potential of leveraging coverage for designing more efficient attacks. Our experiments reveal no clear correlation between neuron coverage, adversarial robustness, or attack effectiveness.
A Novel Class Activation Map for Visual Explanations in Multi-Object Scenes

Class activation maps (CAMs) have emerged as a popular technique to improve model interpretability of deep learning-based models. While existing CAM methods are able to extract salient semantic regions to provide high-confidence pseudo-labels for downstream tasks such as semantic segmentation, they are less effective when dealing with multi-object scenes. In this paper, we design a multi-channel weight assignment scheme that learns from both positive and negative regions to yield an improved CAM model for images comprising multiple objects. We demonstrate the effectiveness of our proposed method on two new data sets, a cat-and-dog dataset and a PASCAL VOC 2012-based multi-object dataset, and show it to compare favourably with other state-of-the-art CAM methods, outperforming them in terms of both mIoU and inter-object activation ratio (IAR), a new evaluation measure proposed to evaluate CAM performance in multi-object scenes.
The Elliptic Energy Loss for Rotated Object Detection in Aerial Images

Rotated object detection is a promising yet challenging task in computer vision. Existing algorithms mainly train the rotated detector by the L <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">n</inf> -norm loss, which is inconsistent with the evaluation metric of Intersection over Union (IoU). However, the concise and efficient solution using the loss based on IoU between oriented boxes is hindered by its non-differentiability. In this paper, we propose Elliptic Energy Loss, a differentiable loss based on curve energy, to fit with the evaluation metric IoU. Specifically, given a pair of predicted and ground truth boxes, we first convert them to curve representations using the elliptic transformation. Then, the curve energy is calculated to measure the similarity between the predicted and ground truth curves. Finally, the curve energy is used as regression loss to optimize rotated detectors. We conduct experiments with different detectors on DOTA and HRSC2016 datasets, which demonstrate that the performance is significantly improved by our proposed loss. The code is available at https://github.com/zhangc-uestc/EEL.
Non-profiling Time-Frequency Analysis based Correlation Optimization with Deep Learning

Correlation Optimization with Deep Learning Analysis (CO-DLA) is a state-of-the-art non-profiling side-channel analysis based on Deep Learning. In this paper, we proposes a novel Wavelet Scattering Transform-based CO-DLA (WST-CO-DLA). WST-CO-DLA utilizes the non-training-required Wavelet Scattering Transform (WST) to extract time-frequency features for CO-DLA, thus offering three main contributions. Firstly, WST eliminates the need for training-required convolution blocks in CNN-based CO-DLA or reduces the number of linear layers in MLP-based CO-DLA, resulting in faster execution time as compared to existing CO-DLA methods. WST-CO-DLA performs 28% and 60% faster than the reported MLP-based CO-DLA and CNN-based CO-DLA, respectively, on the ASCAD datasets. Secondly, WST transforms the 1-D time domain Physical Leakage Information (PLI) measurements to a 2-D time-frequency domain, providing rich time-frequency features for WST-CO-DLA to learn from. Experiments show that WST-CO-DLA requires 50% and 75% fewer PLI measurements to reveal the secret key from the aligned and misaligned ASCAD datasets, respectively, compared to reported CO-DLA methods. Thirdly, we implement a Short Time Fourier Transform-based CO-DLA (STFT-CO-DLA) and benchmark WST-CO-DLA against STFT-CO-DLA. Experimental results show that WST-CO-DLA requires up to 95.83% fewer measurements than STFT-CO-DLA to extract the secret key from the ASCAD datasets.
Transformation Consistency for Remote Sensing Image Super-Resolution

Single Image Super-Resolution (SISR) based on deep learning methods has been widely studied for applications on remote sensing images. With limited remote sensing images, most of the existing SISR methods simply adopt the regular data augmentation approaches (such as flip) in natural images to improve model performance. Considering the fact that remote sensing images are all taken from a bird’s-eye view and objects appear in multiple directions, we first introduce rotation augmentation method in remote sensing images to promote diversity of samples dramatically, as rotation does not cause semantic problems like people standing upside down in natural images. However, image rotation at various angles implemented by interpolation will cause the inconsistent pixel distribution problem for the pixel level task. Thus, we propose Transformation Consistency Loss Function (TCLF) to narrow the gap between the augmented and original distribution, while expanding the feature space with rotation augmentation method. Extensive experiments are performed on UC-Merced Land-use dataset of 21 remote sensing scenes, and the results as well as ablation studies demonstrate our proposed method outperforms mainstream methods.
Variational Feature Disentanglement for Few-Shot Domain Adaptation

In this paper, we focus on the few-shot domain adaptation problem. With limited training data in target domain, a new approach is emerging to acquire the transferable knowledge from the source domain. Previous methods aligned the embedding space between domains by reducing the pair-wise distance. However, these methods are reporting the misalignment and poor generalization. To solve this problem, we propose a variational feature disentanglement framework. The embedding features are explicitly disentangled into domaininvariant and domain-specific components. The distributions of domain-invariant variance are estimated and aligned by the variational inference. For further disentanglement, the domain-invariant and domain-specific components are separated by the orthogonal constraints of subspaces. The experiments on Digits dataset and VisDA-C dataset demonstrate that the proposed method can outperform the state-of-the-art methods.
Task-Aware Graph Convolutional Network for Active Learning

Active learning to select the most valuable examples for annotation from unlabeled data pools in high-dimensional datasets is particularly challenging and has attracted considerable attention. However, the impact of annotated examples on global data was often not considered. In this paper, we propose a novel batch active learning strategy called TA-GCNAL based on the perspective of task model. To consider data distribution in the labeled and unlabeled pool, we first calculate the correlation between data through a graph convolution network to predict the example’s uncertainty. Then, a subset of examples with high uncertainty is generated to replace the unlabeled data pools. It transformed into a probability space based on the task model for core dataset selection and annotation. Finally, we compute the representativeness of the core dataset in probability space for the example batch selection, taking into account the correlation between labeled and unlabeled pools. Our proposed TA-GCNAL achieve superior performance than the state-of-the-art methods on various benchmark datasets. The improvement is particularly significant on unbalanced and large number of classes datasets.
Model Doctor for Diagnosing and Treating Segmentation Error

Despite the remarkable progress in semantic segmentation tasks with the advancement of deep neural networks, existing U-shaped hierarchical typical segmentation networks still suffer from local misclassification of categories and inaccurate target boundaries. In an effort to alleviate this issue, we propose a Model Doctor for semantic segmentation problems. The Model Doctor is designed to diagnose the aforementioned problems in existing pre-trained models and treat them without introducing additional data, with the goal of refining the parameters to achieve better performance. Extensive experiments on several benchmark datasets demonstrate the effectiveness of our method. Code is available at https://github.com/zhijiejia/DoctorForSeg.
Content-Adaptive Parallel Entropy Coding for End-to-End Image Compression

State-of-the-art entropy models, e.g. autoregressive context models, utilize spatial correlation among latent representations, leading to more accurate entropy estimation. However, this autoregressive design naturally results in serial decoding and the infeasibility of parallelization, which makes the decoding procedure slow and less practical. To address the issue, we propose a Content-Adaptive Parallel Entropy Model (CAPEM) that takes a two-pass context calculation with dynamically generated patterns. Our CAPEM relaxes the strict coding order while the dynamic context mechanism still promotes flexibility in capturing latent dependency. This design greatly improves the parallelism of the context model, leading to higher coding efficiency while maintaining the same rate-distortion performance. We test it on the widely used Kodak and CLIC image datasets. Experimental results show that the proposed model outperforms the recent works with less complexity.
Dynamic Range Transformer (DRT): Learning Enhanced Log-Perceptual Information with Swin-Fourier Convolution Network for HDR Imaging

The image obtained using an image sensor with limited dynamic range cannot perfectly represent the various lighting conditions of the real world. Various HDR methods have been studied for expanding the dynamic range in a single image. However, it is difficult to avoid ghosting artifacts caused by the movement of the subject over time and the corresponding texture loss. To solve these problems, we present a novel HDR image acquisition method via dynamic range transformer (DrT) that learns enhanced log-perceptual information using Swin-Fourier convolutional neural network as a backbone. When training the DrT with Swin-Fourier network, it estimates the attention map to obtain an HDR image by minimizing the enhanced log-perceptual (ELP) loss. The Swin-Fourier network considers both local and global contexts simultaneously, which reduces ghosting and texture loss. By learning ELP, it also minimizes color distortion and restores fine details of the dynamic range. Experimental results demonstrate that the HDR results obtained using DrT show reduced color distortion, significantly decreased ghosting artifacts, and texture loss compared to conventional methods. We provide implementation code of our proposed methods in https://github.com/HeunSeungLim/DrT
Trilateral dual-resolution real-time semantic segmentation network for road scenes

In the face of complex and changing real-world road scenarios, two-branch network structure can maintain good accuracy while satisfying real-time conditions. However, choice of fusing only at the end of two branches and direct fusion of low-level and high-level features leads to the waste of various information at each stage, thus limiting performance improvement of existing models. To solve this problem, a trilateral dual-resolution real-time semantic segmentation network(TDRNet) was proposed. The first branch is maintained at high resolution, while the second branch is downsampled several times, ensuring full fusion of spatial and semantic information at different stages by compact bilateral connections between the two branches. The third branch selectively learns semantic information from the second branch and fuses the other two branches in a weight-guided manner. TDRNet achieved 75.5%mIoU at 139.7 FPS on CamVid test set and 76.1% mIoU at 155.9 FPS on Cityscapes validation set. Experiments on several road datasets demonstrate that the method achieves a better balance between accuracy and real-time performance compared to other real-time semantic segmentation algorithms.
You Only Need 80k Parameters to Enhance Image: Learning Periodic Features for Image Enhancement

Benefiting from the promising performance of CNNs models for high-level vision tasks, these networks have been extensively adopted to image enhancement tasks. However, recent methods have complex architecture resulting in poor generalization and high computational cost. Their activation functions are originally designed for other vision tasks. In this work, we present a lightweight network to learn periodic features (LPF) using the proposed wave presentation. Specifically, to better capture implicit feature representations, we represent features as signals with three parts: Cosine Wave Map (CWM), Sine Wave Map (SWM) and Direct Current Map (DCM). Thus, we formulate the image enhancement task as a signal modulation problem. Inspired by the Fourier transform, we build the Fourier Enhancement Module (FEM) that allows for efficient and scalable spatial mixing of local and non-local contents and dynamically learns the interaction between waves to enhance the images. LPF with only 80k parameters achieves better quantitative and qualitative results compared with SOTA methods on four image enhancement datasets. The source code and pretrained model are available at https://github.com/DeniJsonC/LPF.
HDR-LMDA: A Local Area-Based Mixed Data Augmentation Method for Hdr Video Reconstruction

Mainstream image manipulation-based data augmentation methods (e.g., CutMix) undermine the integrity of extracted features, which leads to limited effect for pixel-level image processing tasks. In this paper, a local area-based mixed data augmentation method called HDR-LMDA for HDR video reconstruction is proposed. Within it, the local exposure augmentation (LEA) applies different exposures to original LDR inputs among different regions, while the local RGB permutation (LRP) shuffles the color channels of the random patch instead of the entire frame. By mixing both two operations, the model is forced to learn how to apply concise ill-exposure recovery and color processing within the same training process. Experiments demonstrate that HDR-LMDA achieves a better PSNR-T boost of 0.93dB, compared with conventional works under the same conditions.
Robust Bounding Box Regression for Small Object Detection

Deep learning advances have propelled common object detection development. However, small object detection in aerial images remains inaccurate. Intersection-over-Union (IoU) has limitations in handling the separation or inclusion cases of prediction and ground truth boxes, especially in small object detection, which significantly hampers the process of bounding box regression. To tackle the issue, we propose Balanced Corner-IoU (BC-IoU) loss, which incorporates both corner point distances and IoU metric, while simultaneously introducing the instance area as a component of loss terms. Moreover, Point Offset Module (POM) branch is developed to generate additional positive samples for small object regression by dynamically controlling anchor point generation. With the above designs, Scale Adaptive Network (SAN) provides a solution to bounding box regression of small objects. Experiments on the small object detection dataset show that BC-IoU loss outperforms other IoU loss variants and that SAN significantly improves performance over the Fully Convolutional One-Stage (FCOS) baseline.
PFC-UNIT: Unsupervised Image-to-Image Translation with Pre-Trained Fine-Grained Classification

Unsupervised image-to-image translation has gained great attention in data augmentation by allowing the translation of images from one domain to another while preserving their content and style. However, existing methods face major challenges when these two domains have substantial discrepancies in shape and appearance. To overcome these challenges, we introduce a novel framework that can boost the naturalness and diversity of unsupervised image-to-image translation with pre-trained fine-grained classification (PFC-UNIT). Specifically, PFC-UNIT trains a content encoder to obtain the coarse-level content feature in the first stage. In the second stage, a new pre-trained fine-grained classification (PFC) is designed to generate fine-level images with style consistency. Furthermore, during the latter part of the second stage, a dynamic skip connection is added to generate finer-level images with content consistency. Experimental results show that as a plug-and-play tool, our PFC dramatically enhances the image translation effect by maintaining vivid details and keeping content and style consistent. And the proposed PFC-UNIT outperforms leading state-of-the-art methods. The code is available at https://github.com/GZHU-DVL/PFC-UNIT.
Cost-Efficient Multi-Instance Multi-Label Active Learning Via Correlation of Features

Multi-instance multi-label active learning (MIMAL) usually uses example uncertainty and label correlation to select the most valuable example-label pairs, maximizing the learner’s performance. However, the existing MIMAL solutions do not consider the correlation of example features when selecting example-label pairs. Here, this paper proposes a novel MIMAL framework that can effectively exploit the relationship between examples and features to reduce annotation cost. We first perform feature screening on the examples. It effectively eliminates the interference of useless features on the example to the annotations. Next, we quantify the correlation between features and examples as the basis for selecting example-label pairs. Finally, we query for the most likely positive subexample-label pair among the selected example-label pairs. The extensive experiments on multi-label datasets from diverse domains show that the proposed MIMAL can better save query cost and achieve superior performance than state-of-the-art MIMAL methods.
Model-Agnostic Visual Explanations via Approximate Bilinear Models

This paper proposes InteractionLIME: a model-agnostic attribution technique to explain deep models predictions in terms of feature interactions. Specifically, we regress a bilinear form to approximate the output of two-input models, by sampling perturbations of both inputs simultaneously. Upon training, we retrieve a global explanation and a set of feature partitioning maps via the singular value decomposition of the learned interaction matrix of the bilinear model. We demonstrate InteractionLIME on vision and text-vision contrastive models, using visual examples and quantitative evaluation metrics. Our results show that the bilinear model successfully retrieves important interacting features from both inputs, while strongly reducing the occurrence of incomplete or asymmetric explanations produced by a linear model.
Endoscopic Feature Enhancement for Stomach 3D Reconstruction without Dyeing

In the past decades, endoscopic 3D reconstruction has become increasingly important for medical diagnosis, computer-aided surgeries, and minimally invasive procedures. Many previous studies have been able to reconstruct the complete stomach shape using the structure-from-motion approach; however, such methods require real or virtual indigo carmine (IC) dyeing to enhance endoscopic images before the 3D reconstruction takes place. In this study, we propose an alternative approach by enhancing the appearance of gastric tissue texture and reducing the effect of noise. We show that it is possible to perform 3D reconstruction using monocular endoscopy and without the need for chromoendoscopy. In addition, we show that our method can increase the number of feature matches and maintain the feature consistency, which is critical to the performance of stomach 3D reconstruction.
Retrieve the Visible Feature to Improve Thermal Pedestrian Detection Using Discrepancy Preserving Memory Network

We propose an approach for enhancing pedestrian detection in thermal infrared images using paired visible-thermal images in training. Recently, approaches that retrieve the corresponding visible features from thermal features using a key-value memory network have been proven effective for improving detection results. However, for memory networks storing thermal-visible features, random initialization and end-to-end training may not be ideal, as this can reduce the diversity of memory slots. Also, the retrieved visible features have different reliability as the overall similarities between key slots in the memory network and thermal features differ. These motivate us to propose a DIscrepancy Preserving (DIP) Memory that is updated manually to prevent convergence of key-value memory slots. We also evaluate the reliability of each retrieved visible feature and adjust the training protocol of the detection head. Experiment results on two visible-infrared pedestrian detection datasets demonstrate the superiority of our framework.