Reviewerly/exHarmony huggingface dataset:
Feature Structure Similarity Index for Hybrid Human and Machine Vision

More and more images/videos will be consumed by both human and machine in many fields. Optimization of image processing algorithm for hybrid human and machine becomes a challenging task. To address this problem, feature structure similarity index (FSSIM) is proposed in this paper as an objective metric for image quality assessment (IQA), by defining structure similarity in low-level feature domain. Features extracted by the first convolutional layer of pre-trained resnet50 network are treated as common feature domain for both human and machine vision. Moreover, multi-scale structure similarity with weighting matrix is used as distance measure in the feature domain. FSSIM is capable of fully decoupling image processing and its downstream machine tasks, enabling image processing algorithm optimization for hybrid human and machine vision. Experimental results show FSSIM-optimized image processing algorithms achieve significant performance improvement over existing metrics in context of machine vision tasks including object detection and semantic segmentation. Meanwhile reconstructed images of FSSIM-optimized algorithms are better friendly to human vision.
Wavelet-Based Frequency-Dividing Interactive CNN for Image Classification

The vanilla tensor in convolutional neural networks (CNNs) can be seen as a mixture of feature information at different frequencies, which is currently only used as a carrier of information. However, few people notice that vanilla tensor is spatial information redundant and information interaction of different frequency bands is beneficial for CNNs. In this paper, we design a novel Wavelet-based frequency-dividing interactive block (WFDI) to factorize a vanilla tensor into a pair of tensors with complementary information to reduce redundancy. Based on this, we embed it into the CNN (WFDI-CNN) for image classification. Specifically, the WFDI-CNN factorizes the vanilla tensor into a low-frequency tensor with lower spatial resolution and a high-frequency tensor with complementary information. Then, the information interaction and forward propagation between the high-frequency and low-frequency tensors not only save computational resources but also improve the network performance. Experimental results on CIFAR10 and CIFAR100 datasets all demonstrate the effectiveness of the proposed WFDI block.
Retinex-based Image Denoising / Contrast Enhancement Using Gradient Graph Laplacian Regularizer

Images captured in poorly lit conditions are often corrupted by acquisition noise. Leveraging recent advances in graph-based regularization, we propose a fast Retinex-based restoration scheme that denoises and contrast-enhances an image. Specifically, by Retinex theory we first assume that each image pixel is a multiplication of its reflectance and illumination components. We next assume that the reflectance and illumination components are piecewise constant (PWC) and continuous piecewise planar (PWP) signals, which can be recovered via graph Laplacian regularizer (GLR) and gradient graph Laplacian regularizer (GGLR) respectively. We formulate quadratic objectives regularized by GLR and GGLR, which are minimized alternately until convergence by solving linear systems—with improved condition numbers via proposed preconditioners—via conjugate gradient (CG) efficiently. Experimental results show that our algorithm achieves competitive visual image quality while reducing computation complexity noticeably.
CSSBA: A Clean Label Sample-Specific Backdoor Attack

In backdoor attacks, attackers usually implant the backdoor into the target model during the training phase by manipulating some training samples with backdoor triggers. The state-of-the-art backdoor attack is Sample-Specific Backdoor Attack. Although this attack can bypass most existing backdoor defenses, it can be detected by checking the mapping relationship between samples and their labels. In this paper, we propose a new backdoor attack called Clean Label Sample-Specific Backdoor Attack (CSSBA). We use advanced deep steganography to hide the trigger in source images from training dataset and obtain the backdoor images. Then we generate the poisoned images similar to the backdoor images in feature space without changing their original labels, and the poisoned images are used to train the target model and implant the backdoor. We demonstrate the effectiveness of our method with good invisibility on different datasets and models.
A Multi-Scale Cell Segmentation Method for Detecting Hematological Disorders

Cell segmentation, conducted on a microscopic image that contains blood cells, plays a crucial role for the detection of various hematological disorders. Existing methods often yield inferior performance in the presence of elongated and irregularly-shaped cells, as well as to those adjacent cells with partial overlapping among themselves. To address these issues, a novel multi-scale cell segmentation (MCS) method is proposed in this paper that involves three scales, denoted by coarse, medium, and fine, for demonstrating the effectiveness and efficacy of the proposed multiscale approach. It has been shown in our work that noise and insignificant cell structures can be effectively suppressed at the coarse scale. Consequently, those elongated and irregularly-shaped cells are more accurately identified. Furthermore, our developed coarse-to-fine scale tracing algorithm, performed to trace each identified cell, improves segmentation accuracy. To solve the problem of cell overlapping, an effective cell identification technique is developed for extracting cell contours for each scale. Extensive experimental results obtained from three benchmark datasets have shown that the proposed MCS outperforms the state-of-the-art methods by a large margin.
Atten-Adapter: A Unified Attention-Based Adapter for Efficient Tuning

Recently, more and more large pre-trained models have emerged. Several parameter-efficient tuning methods have been studied to transfer the prior knowledge of the pre-trained models to specific downstream tasks and achieve promising results. This paper proposes a simple yet effective method called Atten-Adapter. To the best of our knowledge, this is the first work that utilizes attention with learnable parameters as the internal structure of the adapter in the field of fine-tuning. The attention-based adapter can provide better information fusion ability and pay more attention to the global features compared to the MLP-based adapter. As a plug-and-play module, Atten-Adapter can be easily adapted to different types of vision models such as ConvNets and Transformer architectures in different tasks like classification and segmentation. Moreover, we demonstrate the generality of our proposed adapters by conducting experiments on language models. With small amounts of tunable parameters, our method achieves significant improvements compared to the previous state-of-the-art methods.
Optical Character Recognition for Medical Records Digitization with Deep Learning

The importance of document digitization has increased due to recent technological advancements, including in the medical field. Digitization of medical records plays a vital role in the healthcare sector as it helps expedite emergency treatment. Due to the scarcity of published studies and public German textual resources, a medical records database with German handwriting was collected and digitized. In this study, document digitization was accomplished by implementing deep learning, region of interest (ROI) detection, and optical character recognition (OCR) on a dataset containing medical forms filled with German and English characters. To find the best model for ROI detection, YOLOv5, and SSDResNet50 models were utilized and compared with YOLOv5 producing a better mean average precision (mAP) of 0.91. OCR was then carried out on the output from YOLOv5 with two different methods again for comparison. The Gated-CNN-BLSTM algorithm yielded a character error rate (CER) of 9%, while transformer-based OCR (TrOCR) achieved a CER of 6%. The proposed system could be implemented and further tested in local hospitals, with the OCR dictionary being expandable to include other Roman character-based languages.
Deep Cross-Modal Steganography Using Neural Representations

Steganography is the process of embedding secret data into another message or data, in such a way that it is not easily noticeable. With the advancement of deep learning, Deep Neural Networks (DNNs) have recently been utilized in steganography. However, existing deep steganography techniques are limited in scope, as they focus on specific data types and are not effective for cross-modal steganography. Therefore, We propose a deep cross-modal steganography framework using Implicit Neural Representations (INRs) to hide secret data of various formats in cover images. The proposed framework employs INRs to represent the secret data, which can handle data of various modalities and resolutions. Experiments on various secret datasets of diverse types demonstrate that the proposed approach is expandable and capable of accommodating different modalities.
Quantifiable Robustness Estimation for Object Detection with CNNs Using Intrinsic Dimensionality

Interpreting the results of Convolutional Neural Networks remains a challenging task. Quantitative evaluations apart from precision, recall, and their extensions are rare and usually do not cover the necessary aspects of specific applications. In this work, a methodology based on the intrinsic dimensionality of the image space and latent space in multiple layers is presented. This methodology has been used in other literature for classification but is leveraged to object detection where the interpretation of the results is more complex. The suitability of the intrinsic dimensionality is evaluated first for general augmentation techniques in multiple datasets and with multiple networks and later on a specific use case with multiple disturbances included. With the help of the intrinsic dimensionality, conclusions about the robustness can be drawn which are not apparent from the precision and the suitability of the methodology as an auxiliary quantifiable metric therefore shown.
Deep Unfolding Network with Physics-Based Priors for Underwater Image Enhancement

We propose an underwater image enhancement algorithm that leverages both model- and learning-based approaches by unfolding an iterative algorithm. We first formulate the underwater image enhancement task as a joint optimization problem, based on the image formation model with physical model and underwater-related priors. Then, we solve the optimization problem iteratively. Finally, we unfold the iterative algorithm so that, at each iteration, the optimization variables and regularizers for image priors are updated by closed-form solutions and learned deep networks, respectively. Experimental results demonstrate that the proposed algorithm outperforms state-of-the-art underwater image enhancement algorithms.
Learnt Deep Hyperparameter Selection in Adversarial Training for Compressed Video Enhancement with a Perceptual Critic

Image based Deep Feature Quality Metrics (DFQMs) have been shown to better correlate with subjective perceptual scores over traditional metrics. The fundamental focus of these DFQMs is to exploit internal representations from a large scale classification network as the metric feature space. Previously, no attention has been given to the problem of identifying which layers are most perceptually relevant. In this paper we present a new method for selecting perceptually relevant layers from such a network, based on a neuroscience interpretation of layer behaviour. The selected layers are treated as a hyperparameter to the critic network in a W-GAN. The critic uses the output from these layers in the preliminary stages to extract perceptual information. A video enhancement network is trained adversarially with this critic. Our results show that the introduction of these selected features into the critic yields up to 10% (FID) and 15% (KID) performance increase against other critic networks that do not exploit the idea of optimised feature selection.
Target-Discriminability-Induced Multi-Source-Free Domain Adaptation

Source-free domain adaptation (SFDA) aims at target adaptation without access to source data, but with only pre-trained source model. Some recent works proposed to automatically combine source models with learnable weights when there are multiple pre-trained source models. In this paper, we propose a simple yet effective framework for multi-source-free domain adaptation (MSFDA). Specifically, based on discriminability towards target samples, we determine transferability of source models before adaptation and generate pseudo-labels during training. To quantify target discriminability, we introduce net confidence which refers to the probability difference between the largest and the second largest probabilities. We empirically show, on several benchmark datasets, our proposed method is competitive to the state-of-the-art methods.
Image Inpainting with Information Loss Reduction and Texture-Structure Feature Fusion

Image inpainting has made great progress with the help of deep learning. However, existing methods show performance degradation when restoring corrupted images with complex scenes. In this paper, we propose a novel image inpainting method by reducing intermediate layer information loss and fusing texture-structure features. To be specific, we first compute a Local Binary Pattern (LBP) map of the corrupted image as the input of structure feature extraction, considering that LBP contains richer structure information than edges and contours. Then, we introduce a Wide Identical Residual Weighting (WIRW) module to utilize the intermediate layer features in the structure encoder. Furthermore, we introduce a Spatial-Transformer (ST) module consisting of Convolutional Neural Network (CNN) and Transformer branches to fuse the structure and texture features, where the CNN and Transformer branches are responsible for capturing the local and global information, respectively. Various experiments on public datasets including CelebA, Paris StreetView, and Places2 demonstrate the effectiveness of the proposed method. Especially, our ablation study separately verifies the contribution of each module to the whole framework. The code is publicly available at https://github.com/GZHU-DVL/LFang.
Hierarchical Conditional Semi-Paired Image-to-Image Translation for Multi-Task Image Defect Correction on Shopping Websites

On shopping websites, product images of low quality negatively affect customer experience. Although there are plenty of work in detecting images with different defects, few efforts have been dedicated to correct those defects at scale. A major challenge is that there are thousands of product types and each has specific defects, therefore building defect specific models is unscalable. In this paper, we propose a unified Image-to-Image (I2I) translation model to correct multiple defects across different product types. Our model leverages an attention mechanism which hierarchically incorporates high-level defect groups and specific defect types to guide the network to focus on defect-related image regions. Evaluated on eight public datasets, our model reduces the Frechet Inception Distance (FID) by 24.6% in average compared with MoNCE, the state-of-the-art I2I method. Another practical challenge on shopping websites is the lack of high quality paired images. We extend our model to be semi-paired by leveraging both paired and unpaired data. Tested on a shopping website dataset to correct three image defects, our model reduces (FID) by 63.2% in average compared with WS-I2I, the state-of-the art semi-paired I2I method.
Enabling the Encoder-Empowered GAN-based Video Generators for Long Video Generation

Despite the remarkable progress in the video generation field, generating videos of longer-term remains challenging due to the challenge of sustaining the temporal consistency and continuity in the resulting synthesized movement while ensuring realism. In this paper, we propose a recall mechanism for enabling an encoder-empowered short-term video generator to produce long-term videos. This mechanism connects smoothly short video clips by modeling their temporal connections. We propose the Recall Encoder-GAN3 (REncGAN3), which enables an Encoder-based Generative Adversarial Network (GAN) to connect short generated video clips into longer sequences of hundreds of frames. The recall mechanism, defined through a loss function, enables an appropriate plasticity-continuity balance in the resulting long video stream. The proposed long-term video generation method ensures the generation of several hundred frames displaying consistent movement, which is non-repetitive while the computational memory costs are similar to those of short video generation models.
Parallel Gradient Blend for Class Incremental Learning

Neural Networks’ performance on a sequence of incremental class tasks drops over time for Class Incremental Learning (IL). The gradient-based IL methods can simultaneously adapt to both new and previous tasks by promoting the update of model in the correct direction. However, existing methods simply consider the previous/new task gradients separately. In this paper, we propose Parallel Gradient Blend (PGB) paradigm. On the one hand, PGB uses the gradients generated by mixing previous and new samples in equal proportions with Batch-Normal layers to adjust a reasonable model update direction. By comparing gradient similarities, the model selects either the previous task gradient or mixed gradient to update. On the other hand, PGB uses the sample feature gradient distribution difference to construct a regularized gradient. Finally, we experimentally demonstrate that PGB outperforms state-of-the-art methods on class-IL benchmarks.
Complexity Scalable Learning-Based Image Decoding

Recently, learning-based image compression has attracted a lot of attention, leading to the development of a new JPEG AI standard based on neural networks. Typically, this type of coding solution has much lower encoding complexity compared to conventional coding standards such as HEVC and VVC (Intra mode) but has much higher decoding complexity. Therefore, to promote the wide adoption of learning-based image compression, especially to resource-constrained (such as mobile) devices, it is important to achieve lower decoding complexity even if at the cost of some coding efficiency. This paper proposes a complexity scalable decoder that can control the decoding complexity by proposing a novel procedure to learn the filters of the convolutional layers at the decoder by varying the number of channels at each layer, effectively having simple to more complex decoding networks. A regularization loss is employed with pruning after training to obtain a set of scalable layers, which may use more or fewer channels depending on the complexity budget. Experimental results show that complexity can be significantly reduced while still allowing a competitive rate-distortion performance.
Spatial-Temporal Transformer Network for Human Mocap Data Recovery

Human Motion Capture (MoCap) has emerged as the most popular method for human animation production. However, due to joint occlusion, marker shedding, and equipment imprecision, the raw motion data is often corrupted, leading to missing motion data. To address this issue, a missing motion data recovery method utilizing attention-based transformers is proposed in this paper. The proposed model consists of two levels of transformers and a regression head. The first level of transformers extract the spatial features within each frame, and the second level of transformer integrates the per-frame features across time to capture temporal dependencies. The integrated features are then sent to the regression head to derive the complete motion. Extensive experiments on the CMU database demonstrate that the proposed model consistently outperforms the other state-of-the-art methods in recovery accuracy.
Learn More: Sub-Significant Area Learning for Fine-Grained Visual Classification

Fine-grained visual classification is more challenging as a subtask of image classification due to the large intra-class and slight inter-class variations. Recent work has focused on localizing discriminative features using attentional mechanisms. However, attention tends to focus on the salient parts of feature maps, ignoring other regions that are not salient but are discriminative for fine-grained classification. In this regard, we propose an efficient method called Discriminative Region Learning Dual-Branch Attention Network (DAL-Net) to address this problem. We propose: (1)Avoid over-focusing on local features by pixel-level attention wipe on salient features. (2)The features are enhanced and suppressed by the channel space enhancement module to mine multiple discriminative regions. (3)To learn complementary semantic information, we fuse cross-regional discriminative features from another branch. Our method can be trained end-to-end without bounding boxes and annotations. We demonstrate that our method can obtain competitive results on the CUB200-2011, FGVC-Aircraft, Stanford Cars, and Stanford Dogs datasets through comprehensive experiments.
Feature Enhancement and Fusion for RGB-T Salient Object Detection

Cross-modal information fusion plays a vital role in the RGB-T salient object detection. Due to RGB and thermal images come from different domains, the modality difference will lead to the unsatisfactory effect of simple feature fusion. How to explore and integrate useful information is the key to the RGB-T saliency detection methods. In this paper, we introduce an Enhancement and Fusion Network. In detail, we propose a Self-modality Feature Enhancement Module that effectively integrate the feature representation of a single modality through global context information. And we propose a Cross-modality Feature Dynamic Fusion Module to realize the effective fusion of cross-modal features in the way of dynamic weighting. Experiments on public datasets show that the proposed method achieves satisfactory results compared with other state-of-the-art salient object detection approaches.
Scene Text Segmentation by Paired Data Synthesis

Scene text segmentation task has numerous practical applications. However, the number of images in the available datasets for scene text segmentation is too small to effectively train deep learning-based models, leading to limited performance. To solve this problem, we perform the segmentation in two aspects: paired data synthesis and methodology. The former is executed via the proposed Text Image-conditional GANs to generate realistic paired data. We exploit real-world images by self-supervised pre-training scheme via inpainting approach before training the proposed GANs to produce realistic synthetic data. The latter is carried out by the proposed scene text segmentation network to optimize learning the generated paired data, called Multi-task Cascade Transformer. It includes two auxiliary tasks and one main task for text segmentation. The functions of the two auxiliary tasks are to learn the text region to focus on, together with learning the structure of text through their fonts, and then they support the main task. We implement three publicly available datasets for scene text segmentation: ICDAR13 FST, Total Text, and TextSeg datasets to demonstrate the effectiveness of our method. Our experimental result outperforms existing methods.