Reviewerly/exHarmony huggingface dataset:
Optimizing Transformer for Large-Hole Image Inpainting

In recent years, leveraging Convolutional Neural Network (CNN) to optimize Transformer (called hybrid model) has achieved great progress in image inpainting. However, the slow growth of the effective receptive field of CNN in processing large-hole regions significantly limits the overall performance. To alleviate this problem, this paper proposes a new Transformer-CNN-based hybrid framework (termed PUT+) by introducing the fast Fourier convolution (FFC) into the CNN-based refinement network. The proposed framework introduces an improved Patch-based Vector Quantized Variational Auto-Encoder (P-VQVAE+). The encoder transforms the masked region into non-overlapping patch-based unquantized feature vectors as the input of Un-Quantized Transformer (UQ-Transformer). The decoder restores the masked region from the predicted quantized features output by the UQ-Transformer while maintaining the unmasked region unchanged. Many experimental results show that the proposed method outperforms the state-of-the-art by a large margin, especially for image inpainting with large masked areas. The code is available at https://github.com/GZHU-DVL/PUTplus.
Multi-Semantic Alignment Co-Reasoning Network for Video Question Answering

Video question answering challenges models on understanding textual questions with varying complexity and searching for clues from visual content with different hierarchical semantics. In this paper, we propose a novel Multi-Semantic Alignment Co-Reasoning Network (MACN) to accomplish an interactive inference between the question and the video input. The design of our MACN comprises two modules of Question-Centric Interaction (QCI) and Contextual Semantic Reasoning (CSR). Specifically, QCI establishes a question-centric heterogeneous graph model to align visual content at different temporal scales with questions to enable the extraction of visual representations under better textual understanding. CSR exploits self-attention mechanisms to extract the contextual dependencies of visual semantics at different hierarchies to achieve co-reasoning of answer clues. Experiments on three benchmarks demonstrate that our proposed method is superior to previous state-of-the-art performance.
More Synergy, Less Redundancy: Exploiting Joint Mutual Information for Self-Supervised Learning

Self-supervised learning (SSL) is now a serious competitor for supervised learning, even though it does not require data annotation. Several baselines have attempted to make SSL models exploit information about data distribution, and less dependent on the augmentation effect. However, there is no clear consensus on whether maximizing or minimizing the mutual information between representations of augmentation views practically contribute to improvement or degradation in performance of SSL models. This paper is a fundamental work where, we investigate the role of mutual information in SSL, and reformulate the problem of SSL in the context of a new perspective on mutual information. To this end, we consider joint mutual information from the perspective of partial information decomposition (PID) as a key step in reliable multivariate information measurement. PID enables us to decompose joint mutual information into three important components, namely, unique information, redundant information and synergistic information. Our framework aims for minimizing the redundant information between views and the desired target representation while maximizing the synergistic information at the same time. Our experiments lead to a re-calibration of two redundancy reduction baselines, and a proposal for a new SSL training protocol. Experimental results on multiple datasets and two downstream tasks show the effectiveness of this framework.
CLIP4Stereo: Revisiting Domain Generalized Stereo Matching via Clip

Despite supervised deep stereo matching networks have achieved impressive performance given sufficient training data, the poor generalization ability caused by the domain shifts prevents them from being applied to unseen domains. Recent progress has shown that CLIP could be a promising alternative for zero-shot visual representation task under the natural language supervision. In this paper, we present a new framework for domain generalized stereo matching by leveraging the contrastive language-image pre-training (CLIP), which distills text-guided discriminative content information rather than task-irrelevant style information. Extensive experiments show that the model generalization ability can be improved significantly in the unseen domain when transferring from SceneFlow to Middlebury, ETH3D and KITTI.
OOD Attack: Generating Overconfident out-of-Distribution Examples to Fool Deep Neural Classifiers

Deep neural networks (DNNs) are dominating various computer vision solutions. However, DNN classifiers suffer from the out-of-distribution (OOD) overconfidence issue, i.e., making overconfident predictions on OOD samples. In this paper, we consider a new OOD attack task, i.e., generating OOD examples that fool DNN classifiers to trap into this issue. Specifically, we first generate seed examples by sampling from common OOD distributions, and then lift the prediction to be overconfident. Extensive experiments with different seeds and confidence-lifting solutions under white-and black-box settings validate the feasibility of OOD attack. Besides, we demonstrate its usefulness in evaluating OOD detection and alleviating the OOD overconfidence issue.
Exploring Effective Knowledge Distillation for Tiny Object Detection

Detecting tiny objects is a long-standing and critical problem in object detection, with broad real-world applications such as autonomous driving, surveillance, and medical diagnosis. Recent studies for tiny object detection often cause extra computational costs during inference due to introducing feature maps with increased resolution or additional network modules. This scarifies the inference speed for better detection accuracy and may heavily limit their availability to real-world applications. Therefore, this paper turns to knowledge distillation to improve the representation learning of a small model regarding both superior detection accuracy and fast inference speed. The masked scale-aware feature distillation and local attention distillation are proposed to address the critical issues in the distillation of tiny objects. Experimental results on two tiny benchmarks indicate that our method can bring noticeable performance gains to different detectors while keeping their original inference speeds. Our method also shows competitive performance compared to state-of-the-art methods for tiny object detection. Our code is available at https://github.com/haotianll/TinyKD.
Modeling and Interpreting 6-D Object Pose Estimation

This work aims to estimate the 6-Degrees of Freedom Pose of an object using simple convolutional neural networks. The problem is that most methods require previous knowledge of the 3D model of the object of interest, which could be unobtainable. We mitigate the problem by simplifying the object’s 3D model to a single and generic primitive solid to create a model that could estimate the pose of unknown objects. Besides that, we study the interpretability of the neural network by using visualization techniques to understand how the network is splitting the high-dimension feature space to reach a Pose estimation.
Fourier Series and Laplacian Noise-Based Quantization Error Compensation for End-to-End Learning-Based Image Compression

Quantization is a core operation in lossy image compression. In the end-to-end learning-based image compression framework, quantization is conducted by a rounding operation during test, while it is replaced by additive uniform noise during training, leading to a mismatched problem between train and test. To address this problem, we propose a quantization error compensation method for the end-to-end learning-based image compression framework. The method uses Fourier series to approximate the periodic changes of the quantization error, and adds Laplacian noise to the quantized latent during test. The proposed method can be flexibly combined with different end-to-end learning-based image compression methods. Experimental results show that higher coding efficiency can be achieved by adding the proposed method with the state-of-the-art methods.
Hierarchical Feature Fusion Transformer for No-Reference Image Quality Assessment

Recently, increasing interest has been drawn in Transformer-based models for No-reference Image Quality Assessment (NR-IQA), especially for the hybrid approach. The hybrid approach tend to apply Transformer to aggregate quality information from feature maps extracted by Convolutional Neural Networks (CNN). However, existing methods cannot fully utilize the information of hierarchical features extracted by the deep neural network, resulting in the limited performance of image quality evaluation. In this work, we propose a novel Hierarchical Feature Fusion Transformer for NR-IQA (HiFFTiq), which is able to effectively exploit complementary strengths of features extracted by different layers. Further, we propose a new Uniform Partition Pooling (UPP) which can reduce the resolution of input features via uniform partitions and can well retain the quality-related information compared to the traditional pooling method Sliding Window Pooling (SWP). The results of experiment demonstrate that HiFFTiq leads to improvements of performance over the state-of-the-art methods on three large scale NR-IQA datasets.
Robust Graph Neural Diffusion for Image Matching

Image matching identifies matching street landmark patches between the images captured by a vehicular camera and those stored in a database. Applications include autonomous driving perception and localization. However, in practical scenarios, challenging conditions such as changing weather, illumination, and dynamic objects result in perturbations of the captured images, leading to inaccurate matching. To achieve robust landmark patch matching, we present a method, named GRAND-Mat, which leverages a neural diffusion over graph embeddings to counteract perturbations. We first extract high-dimensional features of landmark patches using a ResNet. Then, we utilize graph neural diffusion models to aggregate the self and cross-graph information from these features. Furthermore, we apply feature similarity learning to acquire the final matching score. We evaluate the performance of our model on a street scene dataset, which demonstrates state-of-the-art matching performance under additive perturbations.
CKT: Cross-Image Knowledge Transfer for Texture Anomaly Detection

Most anomaly detection models are often sensitive to unavoidable disturbance or non-defective "visual defects", and such near abnormal samples are easily identified as anomalies, resulting in a high false detection rate. To this end, we propose a novel multi-scale Cross-image Knowledge Transfer anomaly detection model, namely CKT. Different from most existing intra-image distillation methods, our model transfers both the intra-image knowledge of the normal image and the inter-image knowledge of the normal image and the near-anomaly prototype, to assist the model to learn more robust normal patterns. Furthermore, we develop a cross-image attention module for explicitly enhancing the near-abnormal pattern learning during the distillation procedure, to alleviate the problem of high false detection rate induced by near-abnormal instances. Extensive experiments on texture datasets, such as KSDD2, MT, AITEX, and the textural subset of Mvtec-AD, show that the proposed CKT model can outperform most of the current unsupervised anomaly detection methods. Compared with the existing distillation based anomaly detection frameworks, our work can get significant gains with a margin of 2%.
The Oil and Water Separation Phenomenon Inspired Loss for Feature Learning

Learning low-dimensional features that are both diverse and discriminative from high-dimensional data is a fundamental and critical task. This paper proposes a novel loss/principle for feature learning inspired by the "oil and water separation" phenomenon. The features learned with the proposed loss are shown to be more robust to label corruptions in empirical evaluations. Moreover, the proposed method achieves state-of-the-art performance in certain downstream tasks.
Segmentation and Classification-Based Diagnosis of Tumors From Breast Ultrasound Images Using Multibranch Unet

Breast ultrasound is useful for the diagnosis of breast tumors which can be benign or malignant. However, accurate segmentation of breast tumors and the classification of breast ultrasound into benign, malignant, or normal (no tumor) categories is challenging because of different reasons including poor contrast of the tumor region and absence of clear margins. We propose a Multibranch UNet architecture that uses multitask learning for the automated segmentation of breast tumors and classification of breast ultrasound images. Our model exploits the principle of autoencoding to achieve the aforementioned goals by utilizing salient image features. Experiments on publicly available datasets shows the superiority of our model over several state-of-the-art approaches.
Consistent and Diverse Human Motion Prediction Using Conditional Variational Autoencoder with Context-Aware Latent Space

3D human motion prediction is the task of forecasting future motions from past observations. Previous work predicts diverse motions to cover uncertain future possibilities over a longer time. However, since most of them do not consider the surrounding context and use only the target person’s motion as the past observation, they generate overly diverse predictions rather than actual possibilities. Therefore, we propose a model that narrows down the possibilities by considering the context, especially interactions with nearby objects. Our proposed model predicts motions based on latent codes diversely sampled from a context-aware latent space. The experimental results show that our model improves accuracy and generates diverse predictions by focusing on possibilities that are consistent with object locations.
Learned Image Compression with Large Capacity and Low Redundancy of Latent Representation

Learned image compression has attracted a lot of attention in recent years. Currently, popular learned image compression methods usually exploit hyperprior and autoregressive models to facilitate probability estimation and reduce the redundancy of latent representation. These models ignore different image contents, and it is difficult to eliminate the spatial redundancy of image, resulting in the performance saturation. In this work, we propose a learned image compression method with large capacity and low redundancy of latent representation. We design two enhancement modules, i.e., the network capacity expansion module (NCEM) and the high-entropy content guided reconstruction module (HCGR), to construct network architectures with better rate-distortion performance than the existing hyperprior and autoregressive models. Experimental results show that our method can produce superior results compared to the state-of-the-art methods.
NTrans-Net: A Multi-Scale Neutrosophic-Uncertainty Guided Transformer Network for Indoor Depth Completion

Dense depth maps are important constituents in a variety of tasks and have wide ranging applications. However, depth maps captured by indoor depth sensors have an extensive range of missing depth values and are also sparse in nature. Predicting dense depth from sparse input has been widely studied and is solved either as a regression or classification problem. In this paper, we propose a novel representation, termed Unified Ordinal Vectors, to realise the combined advantages of regression and classification methods. To disinter the potential of this representation, we propose NTrans-Net, a novel multi-scale network that can extract hierarchical and complementary information with neutrosophic indeterminacy feature handling. We also propose a dual encoder-decoder transformer structure to handle these neutrosophic domain features with guided attention to better capture inter-modal dependencies for superior depth completion performance. NTrans-Net is designed to be flexible enough to adapt to the dynamic nature of spatial input contexts and be robust to sensor-dependent distributions. We conduct extensive experiments on NYUv2 and ToF18K datasets to demonstrate the superiority of the proposed method in multiple settings, especially in realistic indoor environments as captured by commodity depth sensors.
Face Photo-Sketch Synthesis Via Domain-Invariant Feature Embedding

Face photo-sketch synthesis involves transforming photos into sketches and vice versa. A well-transformed image should preserve its original identity characteristics and naturalness. However, identity preservation remains a challenge because of the large discrepancy between the photo and sketch domains. To this end, we propose a novel face photo-sketch synthesis framework that uses domain-invariant feature embedding (DIFE). The DIFE framework generates images assuming the domain-invariant feature of an image pair for the same person to be the identity information. A joint feature embedding module considers latent features from two different domains as input and transfers them into the domain-invariant latent space. Subsequently, a semantic-aware decoder completes the desired image guided by multiscale facial parsing masks. Experimental results demonstrate that the DIFE method outperforms state-of-the-art approaches visually and perceptually.
SiamCLIM: Text-Based Pedestrian Search Via Multi-Modal Siamese Contrastive Learning

Text-based pedestrian search (TBPS) aims at retrieving target persons from the image gallery through descriptive text queries. Despite remarkable progress in recent state-of-the-art approaches, previous works still struggle to efficiently extract discriminative features from multi-modal data. To address the problem of cross-modal fine-grained text-to-image, we proposed a novel Siamese Contrastive Language-Image Model (SiamCLIM). The model implements textual description and target-person interaction through deep bilateral projection, and siamese network structure to capture the relationship between text and image. Experiments show that our model significantly outperforms the state-of-the-art methods on cross-modal fine-grained matching tasks. We conduct the downstream task experiments on the benchmark dataset CUHK-PEDES and the experimental results demonstrate that our model is state-of-the-art and outperforms the current methods by 11.55%, 11.02%, and 7.76% in terms of top-1, top-5, and top-10 accuracy, respectively.
3D Human Motion Prediction via Activity-Driven Attention-MLP Association

Human motion prediction aims to forecast the future motion based on historical sequences of human poses. However, due to incomplete aggregation of the spatio-temporal information, the predicted poses based on existing methods often suffer from discontinuity and error accumulation. To tackle these problems, we design an activity-driven attention-MLP association method, which adopts discrete cosine transform(DCT) to encode temporal information and extracts spatial features by attention-MLP association layers. For better enhancing spatial feature representation capability of the global and local united graph attention, a channel excitation module is inserted into each layer. Meanwhile, the spatio-temporal MLP is associated with attention to further aggregate the features both in temporal and spatial dimension. Moreover, we introduce a novel activity-driven supervision of the optimization. Empirically, the proposed method outperforms other state-of-the-art algorithms on datasets human3.6m and CMU Mocap. Code is available at https://github.com/alanzhangv123/HMP_am.
Lossy LiDAR Point Cloud Compression via Cylindrical 3D Convolution Networks

Compared with object-level and human-level point clouds, LiDAR point clouds have larger data scales and are more sparse, posing a challenge for the existing learning-based lossy compression scheme. In this paper, we resolve this issue by transforming the point cloud into a cylindrical coordinate system. In this way, we can better retain points close to the sensor with a high density while extending the receptive field of convolution in areas of low point density. Following cylindrical quantization, an autoencoder is utilized to progressively downsample voxels. The coordinates and latent features are compressed by G-PCC and hyperprior-based entropy encoding respectively. The results demonstrate that our approach performs better than PCGCv2. The visualization results also show that our algorithm can better retain the shape of objects. Ablation studies further prove the efficiency of the cylindrical coordinates. The code is publicly available at https://github.com/AirManH/cylindrical_pcc.
Context-Aware Multi-Stream Networks for Dimensional Emotion Prediction in Images

Teaching machines to comprehend the nuances of emotion from photographs is a particularly challenging task. Emotion perception— naturally a subjective problem, is often simplified for computational purposes into categorical states or valence-arousal dimensional space, the latter being a lesser-explored problem in the literature. This paper proposes a multi-stream context-aware neural network model for dimensional emotion prediction in images. Models were trained using a set of object and scene data along with deep features for valence, arousal, and dominance estimation. Experimental evaluation on a large-scale image emotion dataset demonstrates the viability of our proposed approach. Our analysis postulates that the understanding of the depicted object in an image is vital for successful predictions whilst relying on scene information can lead to somewhat confounding effects.
Frequency-Aware Re-Parameterization for Over-Fitting Based Image Compression

Over-fitting-based image compression requires weights compactness for compression and fast convergence for practical use, posing challenges for deep convolutional neural networks (CNNs) based methods. This paper presents a simple re-parameterization method to train CNNs with reduced weights storage and accelerated convergence. The convolution kernels are re-parameterized as a weighted sum of discrete cosine transform (DCT) kernels enabling direct optimization in the frequency domain. Combined with L1 regularization, the proposed method surpasses vanilla convolutions by achieving a significantly improved rate-distortion with low computational cost. The proposed method is verified with extensive experiments of over-fitting-based image restoration on various datasets, achieving up to -46.12% BD-rate on top of HEIF with only 200 iterations.
GPCGC: A Green Point Cloud Geometry Coding Method

A low-complexity point cloud compression method called the Green Point Cloud Geometry Codec (GPCGC), is proposed to encode the 3D spatial coordinates of static point clouds efficiently. GPCGC consists of two modules. In the first module, point coordinates of input point clouds are hierarchically organized into an octree structure. Points at each leaf node are projected along one of three axes to yield image maps. In the second module, the occupancy map is clustered into 9 modes while the depth map is coded by a low-complexity high-efficiency image codec, called the green image codec (GIC). GIC is a multi-resolution codec based on vector quantization (VQ). Its complexity is significantly lower than HEVC-Intra. Furthermore, the rate-distortion optimization (RDO) technique is used to select the optimal coding parameters. GPCGC is a progressive codec, and it offers a coding performance competitive with MPEG's V-PCC and G-PCC standards at significantly lower complexity.