Reviewerly/exHarmony huggingface dataset:
Omission-Free Inpainting: A Three-Stage Approach to Ensure Object Generation

This paper proposes a novel inpainting framework, omission-free inpainting, which ensures generating the desired object in the masked region. Despite recent advancements in text-driven and class-conditional inpainting models, they often fail to restore the missing object. To address this issue, the proposed framework includes a separate object generation stage, resulting in omission-free inpainting. The framework consists of three stages: background generation, object generation and refinement. The background generation stage restores a harmonious background with the surrounding pixels, while the object generation stage creates the desired object using a blending mask that allows the object to be influenced by the background’s color and brightness. Finally, the refinement stage blends the object and background to produce a visually realistic image. We compare the results qualitatively with the state-of-the-art methods, and our method outperforms the existing methods in CLIP score.
HI-RES ACG: Towards High-Resolution Anime Characters Generation

In this paper, we present a portrait-to-anime framework for the application of portrait translation. Our work focuses on synthesizing anime figures that adhere to the style of a given reference anime. The existing methods often encounter challenges in transferring the depth texture information from the reference anime faces, resulting in notable artifacts and distortions in the generated facial shapes. To address these issues, we propose Hi-Res ACG, a dedicated series of model designs tailored specifically for anime portraits. This framework enables the synthesis of high-quality anime faces. Unlike previous GAN-based approaches that primarily rely on color and global semantics for image translation, our method incorporates a multi-head attention high-level semantic module to assist in the translation process. Additionally, we introduce a novel surface texture reconstruction encoder within the U-Net structure, which effectively enhances the texture quality and generates visually excellent anime portraits. Through extensive experimentation, we demonstrate that Hi-Res ACG achieves impressive performance in reconstructing images in the portrait-to-anime task. Our approach exhibits a general improvement in image translation tasks, as evidenced by comparisons with other state-of-the-art models on classical tasks such as portrait-to-anime, horse-to-zebra, and cat-to-dog.
Image-Coupled Volume Propagation for Stereo Matching

Several leading depth-from-stereo methods rely on intensive 4D cost volumes and computationally intensive 3D convolutions for feature matching. We propose to integrate two independent concepts into a new 4D cost volume to achieve a symbiotic relationship. A feature matching part is responsible for identifying matching pixel pairs along the baseline, while a concurrent image volume part is inspired by depth-from-mono CNNs. More technically, the processing of the 4D cost volume is divided into a 2D propagation and a 3D propagation part. Starting from the feature maps of the left image, the 2D propagation, instead of directly predicting depth, supports the 3D propagation part of the cost volume at different levels by adding visual features to the geometric context. Combining the two parts reduces the number of the 3D convolution layers without sacrificing accuracy. Experiments show that our end-to-end trained CNN ranks 2nd on the KITTI2012 and ETH3D benchmarks, while being significantly faster than the 1st ranked method. The source code is available at https://github.com/ohkwon718/icvp.
Detection Transformer with Diversified Object Queries

This paper addresses the issue of redundancy in object queries in the fully end-to-end transformer-based detector (DETR). We demonstrate that the redundancy arises from the positional dependence of object queries, the multiple interactions between object queries and feature maps through self- and cross-attention, and the unstable Hungarian matching algorithm. To maintain the expressiveness of the object queries, we propose a novel loss that reduces the pairwise correlation of learned object query features across different decoder layers. This novel plug-and-play approach, called DOQ-DETR, can be applied to various DETR variants. Our experiments on the large-scale COCO2017 dataset demonstrate that the proposed training scheme improves various state-of-the-art DETR-like models, including Deformable-, Conditional-, and DAB-DETRs.
Encrypted Mobile Traffic Classification with a Few-shot Incremental Learning Approach

Mobile traffic classification is an essential task for network security and management. Even though some progress has been made, the existing methods have limitations regarding plasticity and the requirement for large amounts of labeled data for training. In real-world wireless networks, new applications are constantly emerging. The lack of plasticity means the model must be retrained entirely whenever a larger dataset with new classes is obtained, which is time-consuming. Furthermore, obtaining large amounts of labeled data is often complicated and expensive. To overcome these limitations, we proposed a novel approach for classifying encrypted mobile traffic using the few-shot incremental learning with a Long Short-Term Memory (LSTM) model. We pre-train an LSTM model with a base dataset, then incrementally add classes and update the model with few-shot datasets. We leverage the exemplar selection and knowledge distillation to keep the stability and plasticity of the model. We validate our method by collecting Downlink Control Information (DCI) of twenty different mobile applications from commercial Long Term Evolution (LTE) networks. Our experimental results demonstrate the effectiveness of the proposed method.
End to End Generative Meta Curriculum Learning for Medical Data Augmentation

Current medical image synthetic augmentation techniques rely on the intensive use of generative adversarial networks (GANs). However, the nature of GAN architecture leads to heavy computational resources to produce synthetic images and the augmentation process requires multiple stages to complete. To address these challenges, we introduce a novel generative meta curriculum learning method that trains the task-specific model (student) end-to-end with only one additional teacher model. The teacher learns to generate curriculum to feed into the student model for data augmentation and guides the student to improve performance in a meta-learning style. In contrast to the generator and discriminator in GAN, which compete with each other, the teacher and student collaborate to improve the student's performance on the target tasks. Extensive experiments on the histopathology datasets show that leveraging our framework results in significant and consistent improvements in classification performance.
Investigating the Performance of ECA-YOLOv5n Model for Appearance Quality Detection in Air Conditioner External Units

In an era of rapid technological advancements, keeping up with the iterative updates of factory technology in Intelligent manufacturing presents a daunting challenge. Against this backdrop, this paper investigates the performance of the ECA-YOLOv5n model, a novel approach for appearance quality detection in air conditioner external units. Building upon an open-source dataset, ECA-YOLOv5n model integrates the ECA-Net attention mechanism into the YOLOv5-nano model. ECA-Net effectively captures channel-wise dependencies and studies discriminative features, allowing for seamless integration into the convolutional neural networks of the YOLOv5 architecture without the need for extensive modifications. Furthermore, the efficient channel attention mechanism of ECA-Net demands fewer parameters and computations in comparison to other attention mechanisms, conferring a significant advantage in terms of computational efficiency. Experimental results show that the newly proposed model achieves approximately 98% precision, 99% recall, and 99% mAP on the dataset. The ECA-YOLOv5n model reduces storage usage and minorly increases detection speed in comparison to other YOLO models.
CLIP-FG:Selecting Discriminative Image Patches by Contrastive Language-Image Pre-Training for Fine-Grained Image Classification

Fine-grained Visual Classification (FGVC), which aims to identify objects from subcategories, presents great challenges for classification due to large intra-class differences and subtle inter-class differences. To address these issues of FGVC, this paper proposes a patch selection model referenced from CLIP for Fine Grained Visual Classification, namely CLIP-FG. Specifically, unlike the previous CLIP, which focused only on the level of text and image, we calculate the similarity between labels and image patches. Top k image patches are selected and their indexes fed into the Vision Transformer to select discriminative areas to improve the performance of fine grained image classification. Quantitative evaluations show CLIP-FG’s competitive performance against mainstream methods.
Learning to Draw Through A Multi-Stage Environment Model Based Reinforcement Learning

Machine drawing has gradually become a hot research topic in computer vision and robotics domains recently. However, decomposing a given target image from raster space into an ordered sequence and reconstructing those strokes is a challenging task. In this work, we focus on the drawing task for the images in various styles where the distribution of stroke parameters differs. We propose a multi-stage environment model based reinforcement learning (RL) drawing framework with fine-grained perceptual reward to guide the agent under this framework to draw details and an overall outline of the target image accurately. The experiments show that the visual quality of our method slightly outperforms SOTA method in nature and doodle style, while it outperforms the SOTA approaches by a large margin with high efficiency in sketch style.
Learning Multi-Scale Features for Jpeg Image Artifacts Removal

A recently proposed quantization-table convolutional network (QCN) has proven as a state-of-the-art JPEG image artifacts removal method. To reduce computational complexity, the QCN learns image features from the down-sampled version of the input image. Consequently, the performance might be compromised as some salient features that can only be learned from the original input image with full resolution will be lost. To solve this problem, a novel multi-scale feature extraction block (MFEB) is proposed in this paper, which contains a coarse-scale branch and a fine-scale branch for learning salient image features from the down-sampled input image and the original full resolutions, respectively. To avoid introducing too much additional computational complexity due to the fine-scale branch, in our MFEB, this branch uses only one layer, while the coarse-scale branch exploits multiple layers. The feature sets obtained from the two branches are then fused. With the MFEB, a novel multi-scale artifacts removal network (MARN) is then developed to remove JPEG image artifacts. Extensive experiments have clearly shown that our MARN can deliver superior performance to that of a number of state-of-the-art methods.
Feature Integration via Back-Projection Ordering Multi-Modal Gaussian Process Latent Variable Model for Rating Prediction

In this paper, we present a method of feature integration via back-projection ordering multi-modal Gaussian process latent variable model (BPomGP) for rating prediction. In the proposed method, to extract features reflecting the users' interest, we use the known ratings assigned to the viewed contents and users' behavior information while viewing the contents which is related to the users' interest. BPomGP has two important approaches. Unlike the training phase, where the above two types of heterogeneous information are available, behavior information is not given in the test phase. Thus, the projection that transforms content features into integrated features is trained by using both content and behavior features. Furthermore, to reflect the unique characteristics of ratings, their ordering, we constrain the distance between integrated features in a latent space based on the difference in the degree of known ratings. Finally, experimental results show that the proposed method can calculate features that are useful for rating prediction.
Improving Spherical Image Resampling Through Viewport-Adaptivity

The conversion between different spherical image and video projection formats requires highly accurate resampling techniques in order to minimize the inevitable loss of information. Suitable resampling algorithms such as nearest neighbor, linear or cubic resampling are readily available. However, no generally applicable resampling technique exploits the special properties of spherical images so far. Thus, we propose a novel viewport-adaptive resampling (VAR) technique that takes the spherical characteristics of the underlying resampling problem into account. VAR can be applied to any mesh-to-mesh capable resampling algorithm and shows significant gains across all tested techniques. In combination with frequency-selective resampling, VAR outperforms conventional cubic resampling by more than 2 dB in terms of WS-PSNR. A visual inspection and the evaluation of further metrics such as PSNR and SSIM support the positive results.
Robust Nucleus Classification with Iterative Graph Representational Learning

Classifying nuclei communities in histology images is vital for early cancer treatment, but it remains challenging due to the similar structure of nuclei communities. To address this, we propose an iterative neural graph improvement and broadcasting approach. A fully connected graph is constructed with nuclei as nodes starting with a baseline classification. Node and edge features are updated and exchanged along a Hamiltonian path, removing weak connections. This process filters communities by disconnecting weakly connected nodes and iterates until stability is reached. Loose nodes from this refining stage are then assigned to their closest community clusters. Experimental results on two public datasets demonstrate the superiority of the proposed approach over state-of-the-art methods.
Spectral Grouping Driven Hyperspectral Super-Resolution

Convolutional neural networks have proven to be proficient when extracting low-level concepts in an image. With the wonderful performance of transformers in exploiting the long-range correlations in an image, many methods have been explored where one exploit benefits of both the architectures. Therefore, in order to strengthen our network we add an important feature to transformers wherein single image super-resolution (SISR) is exploited using band grouping leveraging a simple CNN architecture. This paper aims to train a set of simple residual modelling architectures and then integrate them into a transformer architecture to solve super-resolution problem in HSI. We take a step forward to analyse how to adapt swinIR to fully exploit the information derived from band grouping for efficient SISR.
A Novel Selective Encryption Scheme for H.266/VVC Video

Video encryption aims to protect video content in open networks. Although some existing algorithms ensure visual distortion, they are incapable to support fast-forward or rewind, and are vulnerable to chosen-plaintext attack (CPA). Based on the 2D-SCL chaotic map with dynamical parameters, a novel selective encryption algorithm (SEA) is proposed in this paper. To our knowledge, it is the first SEA with interactive characteristics designed for H.266/VVC video. The encryption and decryption process are independent for different network abstract layer (NAL) units, thus supporting decryption starting at any given frame in the video bitstream. Then the initial values and system parameters of the 2D-SCL map are closely related to the original video sequence, hence the proposed SEA is efficient to resist CPA. In addition, the confusion-diffusion mechanism is designed to further encrypt quantized transform coefficients (QTCs). Experimental results demonstrated outstanding performance of the proposed scheme in video security.
3D Brain Registration with Intensity Shift Robustness

Technological advances in medical imaging are enabling us to understand healthcare datasets in great detail. Machine Learning enabled methods, specifically, deep neural networks are continuously achieving benchmark performances in terms of accuracy and computational efficiency. However, the lack of agreed-upon standard procedures, variations in the devices by different vendors, and artifacts induced by the physical phenomenon in the sensors make the data inconsistent and noisy. These variations in the data are detrimental to the performance of learning-based methods. In this study, we analyze the behavior of traditional and deep learning-based image registration methods and explore strategies to handle the problem of intensity distributional shifts without compromising the performance. To achieve this, we propose an intensity-based loss function and demonstrate that the models trained with our proposed loss function are better at handling unseen data from different sites using machines from different vendors. In addition, our trained model is superior in preserving the boundaries of anatomical regions after registration.
NERD: Neural Field-Based Demosaicking

We introduce NeRD, a new demosaicking method for generating full-color images from Bayer patterns. Our approach leverages advancements in neural fields to perform demosaicking by representing an image as a coordinate-based neural network with sine activation functions. The inputs to the network are spatial coordinates and a low-resolution Bayer pattern, while the outputs are the corresponding RGB values. An encoder network, which is a blend of ResNet and U-net, enhances the implicit neural representation of the image to improve its quality and ensure spatial consistency through prior learning. Our experimental results demonstrate that NeRD outperforms traditional and state-of-the-art CNN-based methods and significantly closes the gap to transformer-based methods.
Single Image LDR to HDR Conversion Using Conditional Diffusion

Digital imaging aims to replicate realistic scenes, but Low Dynamic Range (LDR) cameras cannot represent the wide dynamic range of real scenes, resulting in under-/overexposed images. This paper presents a deep learning-based approach for recovering intricate details from shadows and highlights while reconstructing High Dynamic Range (HDR) images. We formulate the problem as an image-to-image (I2I) translation task and propose a conditional Denoising Diffusion Probabilistic Model (DDPM) based framework using classifier-free guidance. We incorporate a deep CNN-based autoencoder in our proposed framework to enhance the quality of the latent representation of the input LDR image used for conditioning. Moreover, we introduce a new loss function for LDR-HDR translation tasks, termed Exposure Loss. This loss helps direct gradients in the opposite direction of the saturation, further improving the results’ quality. By conducting comprehensive quantitative and qualitative experiments, we have effectively demonstrated the proficiency of our proposed method. The results indicate that a simple conditional diffusion-based method can replace the complex camera pipeline-based architectures.
Cross-Domain Few-Shot Classification Via Inter-Source Stylization

The goal of Cross-Domain Few-Shot Classification (CDFSC) is to accurately classify a target dataset with limited labelled data by exploiting the knowledge of a richly labelled auxiliary dataset, despite the differences between the domains of the two datasets. Some existing approaches require labelled samples from multiple domains for model training. However, these methods fail when the sample labels are scarce. To overcome this challenge, this paper proposes a solution that makes use of multiple source domains without the need for additional labeling costs. Specifically, one of the source domains is completely tagged, while the others are untagged. An Inter-Source Stylization Network (ISSNet) is then introduced to enhance stylisation across multiple source domains, enriching data distribution and model’s generalization capabilities. Experiments on 8 target datasets show that ISSNet leverages unlabelled data from multiple source data and significantly reduces the negative impact of domain gaps on classification performance compared to several baseline methods.
Improving Translation Invariance in Convolutional Neural Networks with Peripheral Prediction Padding

Zero padding is often used in convolutional neural networks to prevent the feature map size from decreasing with each layer. However, recent studies have shown that zero padding promotes encoding of absolute positional information, which may adversely affect the performance of some tasks. In this work, a novel padding method called Peripheral Prediction Padding (PP-Pad) method is proposed, which enables end-to-end training of padding values suitable for each task instead of zero padding. Moreover, novel metrics to quantitatively evaluate the translation invariance of the model are presented. By evaluating with these metrics, it was confirmed that the proposed method achieved higher accuracy and translation invariance than the previous methods in a semantic segmentation task.
Query-Based Video Summarization with Pseudo Label Supervision

Existing datasets for manually labelled query-based video summarization are costly and thus small, limiting the performance of supervised deep video summarization models. Self-supervision can address the data sparsity challenge by using a pretext task and defining a method to acquire extra data with pseudo labels to pre-train a supervised deep model. In this work, we introduce segment-level pseudo labels from input videos to properly model both the relationship between a pretext task and a target task, and the implicit relationship between the pseudo label and the human-defined label. The pseudo labels are generated based on existing human-defined frame-level labels. To create more accurate query-dependent video summaries, a semantics booster is proposed to generate context-aware query representations. Furthermore, we propose mutual attention to help capture the interactive information between visual and textual modalities. Three commonly-used video summarization benchmarks are used to thoroughly validate the proposed approach. Experimental results show that the proposed video summarization algorithm achieves state-of-the-art performance.
Learning Spatially-Adaptive Squeeze-Excitation Networks for Few Shot Image Synthesis

Learning light-weight yet expressive deep networks for image synthesis is a challenging problem. Inspired by a recent observation that it is the data-specificity that makes the multi-head self-attention (MHSA) in the Transformer model so powerful, this paper proposes to extend the widely adopted light-weight Squeeze-Excitation (SE) module to be spatially-adaptive to reinforce its data specificity, as a convolutional alternative of the MHSA, while retaining the efficiency of SE and the inductive bias of convolution. It proposes a spatially-adaptive squeeze-excitation (SASE) module for image synthesis task.SASE is tested in low-shot image generative learning task, and shows better performance than prior arts.