Reviewerly/exHarmony huggingface dataset:
Independent and Collaborative Demosaicking Neural Networks

Existing demosaicking neural models generally reconstruct the red, green and blue channels by one unified network. This has the advantage of allowing the RGB channels to share latent features. However, it is unnoticed that the training samples are severely unbalanced among the three channels. As a consequence, a unified model trained to fit the red or blue sample distributions is prone to under-fit the green samples. In this paper, we demonstrate the existence of such conflicts, and analyze the reason behind this phenomenon. To solve this disadvantage, we decouple the traditional three-in-one demosaicking to three independent but collaborative sub-tasks, respecting the distribution of samples in each spectrum. We show that, by our decoupling strategy, a very simple architecture can achieve high accuracy with high efficiency. For substantiation, we construct a model that has merely 1.3M parameters in total, and compares its accuracy to state of the art methods of similar or larger model size, across a wide variety of benchmark datasets. The proposed independent and collaborative architecture gains performance improvement over comparable models in both accuracy and inference time.
End-to-End Variable-Rate Image Compression with Bi-Resolution Spatial-Channel Context Aggregation

Recently, neural network-based image compression techniques have demonstrated remarkable compression performance. The use of context-adaptive entropy models greatly enhances the rate-distortion (R-D) performance by effectively capturing spatial redundancy in latent representations. However, latent representations still contain some spatial correlations(e.g. same spatial structure), it needs to be eliminated by further processing. And many compression models are single-rate model, which is difficult to cover a big range of bitrate. In order to address this issue, we propose a novel variable-rate image compression algorithm that efficiently leverages bi-resolution spatial-channel information through learned mechanisms. In this paper, we first proposed a BRP network to divide our latent representations and side information into HR and LR components, eliminating the spatial redundancy in same location. Combining the spatial-channel context, we proposed a BSC context model, including a decreasing-granularity checkerboard pattern and channel grouping based on cosine slicing strategy. To cover a wide range of bitrate, we take a weight map as input to control bit allocation, achieving multiple compression rates. Our experimental results show that our method provides a better rate-distortion trade-off than BPG, JPEG and other recent image compression methods based on deep learning.
Lambda-Domain Rate Control for Neural Image Compression

Rate control based on rate-distortion modeling is a classic problem in lossy image compression. Despite extensive research in neural image compression, its rate control remains understudied. In this paper, we introduce a variable rate neural image compression scheme that supports precise rate control with one-pass encoding. Our approach utilizes the Lagrangian multiplier method to transform rate control into an unconstrained optimization problem, mapping the target bitrate to λ for rate-distortion trade-off adjustment. We propose an improved exponential R-λ model and estimate the bitrates with a hybrid convolution-transformer network for model fitting. The encoder is controlled by λ, and a multi-layer modulation mechanism ensures variable rate ability. In our experiments, the proposed method outperforms the intra-frame coding of Versatile Video Coding (VVC). Meanwhile, the average rate control error is less than 5.1%, while maintaining almost identical rate-distortion performance and acceptable complexity.
Few-Shot Learning for Word Recognition in Handwritten Seventeenth-Century Spanish American Notary Records

Historical records are invaluable sources of information that provide insights into multiple aspects of past events and societies. The analysis of historical records using deep learning poses critical challenges such as the lack of sufficient labeled data and at times the poor quality of scanned images. In this paper, we propose SpanishFSL, a few-shot learning (FSL) approach for word recognition in 17th-century handwritten Spanish American notary records. SpanishFSL draws inspiration from a zero-shot learning approach developed for image classification. It leverages an autoencoder to construct class-attribute signatures to effectively bridge the gap between seen and unseen classes. This enables SpanishFSL to generalize and accurately recognize words not present in the training set. Our labeled dataset was prepared by paleography experts using a subset of the notary records drafted by two notaries. Through experimental evaluation, we observed that SpanishFSL can outperform other FSL classifiers in terms of word recognition accuracy.
EmAGAN: Embedded Blocks Search and Mask Attention GAN for Makeup Transfer

Currently, the results of makeup transfer are generally satisfactory in most scenarios. However, the transfer results show that the transfer makeup details is not accurate, such as in blush and lip corners. To this end, we propose a variant model of generative adversarial networks (GAN) based on the embedded block search and the mask attention, called EmAGAN, in which we use a latent space module to supplement the detail information for makeup transfer. The latent space module is a randomly initialized Embedded Block. We quantify the transferred image matrix as a discrete probability value and fill the details of different regions by the closest image information found in the Embedded Block. Another, we propose a novel makeup alignment module, which divides the facial feature regions of the source and reference image into blocks by transformer and extracts the mask of makeup area by image segmentation approach. After that we can perform the makeup alignment by matching the similarity relationship learned from the mask of reference image through attention mechanism. Extensive experiments demonstrate that our proposed method can accurately transfer the makeup details. We use the Frechet Inception Distance (FID) and Peak Signal-to-noise Ratio (PSNR) to qualitatively evaluate the quality and similarity of the generated image. Compared with other related schemes, our method achieves better scores, which are 60.12 and 23.226 respectively.
SOFTCUTMIX: Data Augmentation and Algorithmic Enhancements for Cross-Modality Person Re-Identification

One of the primary challenges in achieving Infrared-Visible Person Re-Identification (IV Re-ID) is the significant differences in modalities between visible (VIS) and infrared (IR) images.In addressing this challenge, we propose a new data augmentation method-SOFTCUTMIX and introduce a new algorithm called SOFTCUTMIX Auxiliary Modality(SCAM). SOFTCUTMIX augmentation strategy aims to randomly crop and blend portions of two images with random weights, and meanwhile blend their non-cropped portions with other random weights. SCAM algorithm generates mixed modality images by blending visible light and infrared images and serves as an auxiliary modality to reduce the inherent modality differences. We also design a Channel Random Selection (CRS) to adjust the channels of the three-channel visible light image to reduce differences with the single-channel infrared image. Furthermore, we propose a Weighted Regularization Center Triplet Loss (WRCT) and combine it with the Weighted Regularization Triplet Loss (WRT). This approach reduces intra-class variations and increases inter-class separability, thereby enhancing the discriminative power of the learned features. Experimental results on the SYSU-MM01 and RegDB datasets demonstrate that our algorithm significantly outperforms the state-of-the-art method.
Contextual Associated Triplet Queries for Panoptic Scene Graph Generation

The Panoptic Scene Graph generation (PSG) task aims to extract the triplets composed of subject, object, and relation based on panoptic segmentation. For one-stage methods, PSGTR predicts the subject, object, and relation by one query. However, the integrated query is too implicit to simultaneously ascertain pairs of instances and relations. In PSGFormer, it learns instances and relation queries separately and establishes matches between subject-relation and object-relation pairs by employing the relation as an index. Nevertheless, this method could potentially impede the accurate determination of the optimal match. To address the aforementioned issues, we propose a new one-stage method, Contextual Associated Triplet Queries (CATQ), which employs three branches to decode subject, object, and relation features separately. Additionally, we leverage instance information to guide the relation decoding process. Furthermore, we introduce the triplet context fusion block to enable the extraction of more comprehensive instance pairs and triplet relations. Our proposed method achieves 34.8 Recall@20 and 20.9 mRecall@20 respectively and surpasses the state-of-the-art baseline method by 22.5% and 26.0% with half of the training session.
From Global to Local: An Adaptive Environmental Illumination Estimation for Non-uniform Scattering

The atmospheric scattering model is one of the most widely used model to describe the optical imaging processing of hazy images. However, the global atmospheric light used in the traditional atmospheric scattering model has limitations in describing images with varying local environmental illumination. In this paper, by extending the global atmospheric light to the local illumination, a non-uniform scattering model is proposed, which can better describe real scenes under non-uniform environmental illumination. Based on this model, an adaptive local illumination estimation for hazy image is proposed, which can adapt to the local differences of environment illumination. The experimental results demonstrate that the proposed algorithm can outperform the state-of-the-art algorithms in terms of not only the non-uniform scattering removal but also the adaptability.
Prior Knowledge Guided Network for Video Anomaly Detection

Video Anomaly Detection (VAD) involves detecting anomalous events in videos, presenting a significant and intricate task within intelligent video surveillance. Existing studies often concentrate solely on features acquired from limited normal data, disregarding the latent prior knowledge present in extensive natural image datasets. To address this constraint, we propose a Prior Knowledge Guided Network(PKG-Net) for the VAD task. First, an auto-encoder network is incorporated into a teacher-student architecture to learn two designated proxy tasks: future frame prediction and teacher network imitation, which can provide better generalization ability on unknown samples. Second, knowledge distillation on proper feature blocks is also proposed to increase the multi-scale detection ability of the model. In addition, prediction error and teacher-student feature inconsistency are combined to evaluate anomaly scores of inference samples more comprehensively. Experimental results on three public benchmarks validate the effectiveness and accuracy of our method, which surpasses recent state-of-the-arts.
Efficient Hand Gesture Recognition using Multi-Task Multi-Modal Learning and Self-Distillation

In this paper, we propose a lightweight model for hand gesture recognition using an RGB camera. The proposed model enables recognition of first-person hand gestures using a single camera and achieves near-real-time computational performance on both high-end and low-end computing devices. The proposed framework utilizes multi-task multi-modal learning and self-distillation to deal with the challenges in hand gesture recognition. We integrate additional modalities (depth) and a future prediction mechanism to enhance the model’s ability to learn spatio-temporal information. Furthermore, we employ self-distillation to compress the model, achieving a balance between accuracy and computational efficiency. We compared the proposed hand gesture recognition model with the state-of-the-art method, and our model outperforms the SOTA by 0.88% and 3.52% on the EgoGesture and NVGesture datasets, respectively. In terms of computational efficiency, our model takes only 161ms in average to recognize a gesture on a device with low-end GPUs (NVIDIA Jetson TX2), which is acceptable for interaction in XR applications.
Dual-domain Feature Learning and Cross Dimension Interaction Attention for Nighttime Image Dehazing

Nighttime image dehazing is critical for many computer applications. Directly transferring daytime dehazing models to nighttime scenes often introduces haze residual, detail loss and color distortion for the uneven distribution by artificial lights. Therefore, we propose a nighttime dehazing method by defining the Dual-domain Feature Learning Module (DFLM) and the Feature Optimization Module (FOM). Firstly, we construct the DFLM in both frequency and spatial domains to accurately predict the image degradation caused by haze and remove most haze in nighttime hazy images. Secondly, to address the challenges of uneven illumination distribution and color interference of light sources in nighttime, we construct the FOM based on the proposed Cross Dimension Interaction Attention (CDIA), which captures the feature dependencies by crossing different dimensions including the channel-channel, height-channel and width-channel. By precisely representing illumination and color features, the FOM alleviates color distortion in nighttime dehazing. Extensive experiments on several synthetic and real-world datasets demonstrate that our method outperforms most state-of-the-art methods. Code will be available.
NuclSeg: nuclei segmentation using semi-supervised stain deconvolution

Recently, deep learning-inferred stain deconvolution/separation-based nuclei segmentation works demonstrated significant results by translating low-cost and prevalent immunohistochemical (IHC) slides to more expensive-yet-informative multiplex immunofluorescence (mpIF) images. However, when the input stain style is changed to Hematoxylin and Eosin stain (H&E), which is one of the principal tissue stains used in histology, the stain deconvolution/separation based works can not achieve satisfactory performance because the features of the input stain image are greatly changed. To solve this problem, we integrate stain transfer (H&E->IHC) before stain deconvolution (IHC->mpIF) to revise the image style. Moreover, a new semi-supervised learning strategy collaborating supervised and unsupervised learning processes are employed to diversify training data content for stain deconvolution.Firstly, in the supervised learning process, generative adversarial network (GAN) based image-to-image mapping (stain deconvolution) and inverse mapping models are trained on the dataset of co-registered IHC staining and mpIF staining of the same slides to respectively convert IHC to mpIF (I2m), and mpIF to IHC (m2I). After that, in the unsupervised learning process, high-quality IHC images from m2I model are selected according to the Discriminator score on another unpaired dataset, and then, IHC images are paired with input mpIF as the unsupervised training data for further improving the I2m model. This semi-supervised scheme balances two supervised and unsupervised errors while optimizing to limit the effect of imperfect pseudo inputs but still enhance stain deconvolution. Furthermore, image enhancement is applied after the stain deconvolution model to obtain high-quality segmentation masks. We thoroughly evaluate our method on publicly available benchmark datasets. The results show the proposed model obtains significant improvement compared to the SOTAs.
Class-aware Convolution and Attentive Aggregation for Image Classification

Deep learning has been proven to be effective in image classification tasks. However, existing methods may face difficulties in distinguishing complex images due to the distraction caused by diverse image content. To overcome this challenge, we propose a class-aware convolution and attentive aggregation (CA-Net) framework that improves the effectiveness of representation learning and reduces the influence of irrelevant background. CA-Net includes three main modules: the discrete representation learning (DRL) module that uses a group learning method to learn discriminative representations, the class-aware score of discrete representation (CSDR) module that infers class-aware scores to generate weights for representation learners, and the class-aware representation fusion module(CRF) that aggregates class-aware representations using the class-aware scores as a guide. Our experimental results on three benchmarking datasets show that CA-Net improves the performance of state-of-the-art backbones and enhances feature extraction robustness.
Exploring Feature Fusion from A Contrastive Multi-Modality Learner for Liver Cancer Diagnosis

Self-supervised contrastive learning has achieved promising results in computer vision, and recently it also received attention in the medical domain. In practice, medical data is hard to collect and even harder to annotate, but leveraging multi-modality medical images to make up for small datasets has proved to be helpful. In this work, we focus on mining multi-modality Magnetic Resonance (MR) images to learn multi-modality contrastive representations. We first present multi-modality data augmentation (MDA) to adapt contrastive learning to multi-modality learning. Then, the proposed cross-modality group convolution (CGC) is used for multi-modality features in the downstream fine-tune task. Specifically, in the pre-training stage, considering different behaviors from each MRI modality with the same anatomic structure, yet without designing a handcrafted pretext task, we select two augmented MR images from a patient as a positive pair, and then directly maximize the similarity between positive pairs using Simple Siamese networks. To further exploit multi-modality representation, we combine 3D and 2D group convolution with a channel shuffle operation to efficiently incorporate different modalities of image features. We evaluate our proposed methods on liver MR images collected from a well-known hospital in Taiwan. Experiments show our framework has significantly improved from previous methods.
Adaptive Fusion for Visual Question Answering: Integrating Multi-Label Classification and Similarity Matching

Visual Question Answering (VQA) is an important multimodal task in which models are required to answer questions based on visual cues. However, most visual question-answering models suffer from the language prior problem, which is caused by data bias. Specifically, VQA models tend to output high-frequency answers to answer questions while ignoring the information contained in the images. Many approaches have emerged to solve the language prior problem. However, previous approaches could only improve the performance of easy classes, and there is no means to solve hard classes effectively. In this paper, we will utilize more semantic information to guide the model for learning and better handle hard questions. Specifically, in addition to the classification task, we map the image question pairs and the answers to the same dimensional space and construct a similarity metric between the two to get the answers’ similarity-matching output. Moreover, we learn a set of parameters to fuse the classification output with the answers’ similarity-matching output, and finally, we use the fused output for prediction. We use answer weighting for each output to mitigate the language priors in computing the loss function. Moreover, we use answer masks for the classification outputs. Experimental results demonstrate the effectiveness of our method, which achieves a state-of-the-art performance of 62.20% on VQA-CP v2.
Research on Multi-Person Pose Estimation Based on YOLO and Decoupled Multi-Level Feature Layers Fusion

Multi-person pose estimation is fundamental research in the fields of AIGC, multimedia understanding,virtual reality, human-computer interaction, etc. Existing algorithms have problems such as large computational complexity, low accuracy, and an inability to effectively predict occlusions, making them difficult to deploy on devices with limited computational resources. This paper introduces coordinate attention mechanisms within the YOLOv7tiny framework, improves the OKS loss function tailored for pose estimation tasks, and proposes the FDPose method that decouples bounding box detection and pose regression tasks by fusing different feature layers. Experimental results show that compared with the original algorithm, the standard version FDPose-n used in this paper reduces the number of parameters by 14.8% while improving AP by 9.7%. In comparison with some similarly sized coordinate regression based bottom-up algorithms, FDPose-n achieves higher AP than Associative Embedding, DeepPose, YOLOv5s-pose, YOLOv8n-pose, etc. We use the CrowdPose dataset as the training set and without any pre-trained weights.
Multi-head Siamese Prototype Learning against both Data and Label Corruption

The training of the Deep Neural Network (DNN) has been seriously challenged by insidious noise in the dataset, including noise in raw data and errors in annotations. Existing methods usually limit their efforts to the defense of one particular kind of noise, which would be powerless when facing the coexistence of various noise. To deal with it, we propose a novel Multi-head Siamese Prototype Learning (MSPL) method to promote discriminative features and representative prototypes by modeling invariance in samples and sieving out incorrectness in labels. More specifically, a multi-head Siamese network structure is constructed, where prototype learning with the multi-consistency constraint is performed to improve the resilience of the model to noise. Under this regime, adversarial contrastive learning is performed to train the model with the dynamically generated vicious adversarial examples, further enhancing the invariant predictive ability against data noise. At the same time, to deal with label noise, an effective multi-granularity sample selection strategy is designed to filter out noisy labels by measuring the error distribution in both global and local (i.e., class-specific) perspectives. Semi-supervised learning is accordingly conducted to train the model with the resulting labelled data (i.e., data with clean labels) and unlabelled data (i.e., data with noisy labels). Extensive experiments on benchmarks demonstrate the effectiveness of the proposed method in the extremely noisy learning environment.
Semantic-Aware Dynamic Feature Selection and Fusion for Object Detection in UAV Videos

Keypoint-based detectors perform well in surveillance videos but face challenges in detecting objects in UAV videos due to missed corners and mismatches. To address this, we propose a semantic-aware module with a feature fusion sub-module and a feature selection sub-module. The feature fusion module adaptively combines low-level and high-level features, enhancing corner recall. The feature selection module determines spatial location importance, improving discriminative capabilities and reducing background interference, resulting in better precision. Experiments on the UAVDT benchmark show our method achieves competitive results. Notably, our method improves corner recall by 4.0% and reduces the mismatch rate by 2.9% compared to the baseline. Code is available at https://github.com/jianpingZhonggit/SemanticAwareModule.