Reviewerly/exHarmony huggingface dataset:
Depth Estimation of Multi-Modal Scene Based on Multi-Scale Modulation

As multimodal information is complementary, effectively utilizing scene multimodal information has become an increasingly important research topic for many scholars. This paper proposes a novel multi-scale global learning strategy that utilizes both echo and visual modal data as inputs to estimate scene depth. The framework involves constructing a multi-scale feature extraction method using pyramid pooling modules to aggregate contextual information from different regions and improve global information acquisition ability. Furthermore, a recurrent multi-scale feature modulation module is introduced to generate more semantic and accurate spatial representations in each iteration update process. Additionally, a multi-scale fusion method is constructed for the fusion of echo and visual modalities. The proposed method's superior performance is demonstrated through sufficient experiments conducted on the Replica dataset.
PFTA-Net: Progressive Feature Alignment and Temporal Attention Fusion Networks for Video Inpainting

The goal of video inpainting is to fill in missing regions with reasonable and coherent content in a video sequence. Due to the motion of cameras and objects, the reference frame and the target frame are not aligned, and the useful information of the reference frame cannot be well utilized. Therefore, temporal alignment plays an important role in video inpainting. Some studies have attempted to divide the remote alignment into multiple sub-alignments and process them step by step, but error accumulation is inevitable. In this paper, we present a novel progressive feature alignment and temporal attention fusion network, namely PFTA-Net. Specifically, we design a progressive feature alignment module, which employs sub-alignments with a progressive refinement scheme, resulting in more accurate motion compensation. After alignment, we propose a temporal attention fusion module, which computes temporal attention weights for each aligned reference frame feature, resulting in modulated features for reconstructing the target frame. Our extensive evaluations, including both quantitative and qualitative assessments, demonstrate the better performance and efficacy of our video inpainting network.
CDNet: Cluster Decision for Deepfake Detection Generalization

The fast development of deepfake generation technology has caused serious security threats to human society. Many deep-fake detection methods have been proposed recently, but most of them can only show high detection performance for the deepfakes generated by the similar techniques with the training dataset. To improve the ability of detecting unseen types of deepfakes, some deepfake detection methods have constructed self-generated datasets to train their models. However, the artifacts on these self-generated datasets are usually caused by some specific face-blending algorithms and lack of generality. In this paper, we propose cluster decision network (CDNet) to improve the deepfake detection generalizability. We design a selective attention module that decides the attention areas by manually cropping the facial areas (e.g., eyes, nose, and lips), which greatly reduce the model size and ensure a small model size. Inspired by the contrastive learning, we also propose a cluster classifier to equally utilize the feature representation. Extensive experiments show that our method outperforms existing state-of-the-art methods in deepfake detection generalizability and has the minimum model size.
BCKD: Block-Correlation Knowledge Distillation

In this paper, we propose Block-Correlation Knowledge Distillation (BCKD), a novel and efficient knowledge distillation method that differs from the classical method, using the simple multilayer-perceptron (MLP) and the classifier of the pre-trained teacher to train the correlations between adjacent blocks of the model. Over the past few years, the performance of some methods has been restricted by the feature map size or the lack of samples in small-scale datasets. By our proposed BCKD, the above problem is satisfactorily solved and has a superior performance without introducing additional overhead. Our method is validated on CIFAR100 and CI-FAR10 datasets, and experimental results demonstrate the effectiveness and superiority of our method.
Image Coding Via Perceptually Inspired Graph Learning

Most codec designs rely on the mean squared error (MSE) as a fidelity metric in rate-distortion optimization, which allows to choose the optimal parameters in the transform domain but may fail to reflect perceptual quality. Alternative distortion metrics, such as the structural similarity index (SSIM), can be computed only pixel-wise, so they cannot be used directly for transform-domain bit allocation. Recently, the irregularity-aware graph Fourier transform (IAGFT) emerged as a means to include pixel-wise perceptual information in the transform design. This paper extends this idea by also learning a graph (and corresponding transform) for sets of blocks that share similar perceptual characteristics and are observed to differ statistically, leading to different learned graphs. We demonstrate the effectiveness of our method with both SSIM- and saliency-based criteria. We also propose a framework to derive separable transforms, including separable IAGFTs. An empirical evaluation based on the 5th CLIC dataset shows that our approach achieves improvements in terms of MS-SSIM with respect to existing methods.
Domain Generalization Method for Person Re-Id Using Metabin and Mixstyle

Recently, person re-identification with domain generalization has focused on reducing the gap between source domains and unseen target domains. Meta Batch Instance Normalization (MetaBIN) is one of the most effective methods for addressing the domain gap problem. However, the lack of style variation from limited source domains still makes diversifying virtual simulations difficult. To alleviate this, an improved generalizable person re-identification method is proposed; when this method is combined with MixStyle and meta-leaning, it is called MetaMix. Initially, we represent the MixStyle layers of MetaBIN to create a diverse style during the meta-learning process. Moreover, fine-tuning a BIN module is shown to be applicable for pre-trained re-identification (ReID) models by batch normalization. Extensive experimental results show the effectiveness of the proposed method in a large-scale cross-domain scenario.
KD-Fixmatch: Knowledge Distillation Siamese Neural Networks

Semi-supervised learning (SSL) has become a crucial approach in deep learning as a way to address the challenge of limited labeled data. The success of deep neural networks heavily relies on the availability of large-scale high-quality labeled data. However, the process of data labeling is time-consuming and unscalable, leading to shortages in labeled data. SSL aims to tackle this problem by leveraging additional unlabeled data in the training process. One of the popular SSL algorithms, FixMatch [1], trains identical weight-sharing teacher and student networks simultaneously using a siamese neural network (SNN). However, it is prone to performance degradation when the pseudo labels are heavily noisy in the early training stage. We present KD-FixMatch, a novel SSL algorithm that addresses the limitations of FixMatch by incorporating knowledge distillation. The algorithm utilizes a combination of sequential and simultaneous training of SNNs to enhance performance and reduce performance degradation. Firstly, an outer SNN is trained using labeled and unlabeled data. After that, the network of the well-trained outer SNN generates pseudo labels for the unlabeled data, from which a subset of unlabeled data with trusted pseudo labels is then carefully created through high-confidence sampling and deep embedding clustering. Finally, an inner SNN is trained with the labeled data, the unlabeled data, and the subset of unlabeled data with trusted pseudo labels. Experiments on four public data sets demonstrate that KD-FixMatch outperforms FixMatch in all cases. Our results indicate that KD-FixMatch has a better training starting point that leads to improved model performance compared to FixMatch.
Semantic Learning Network for Controllable Video Captioning

Video captioning is fundamental for visual understanding, which aims at describing the content of a video using natural language. Previous works make great efforts to visual representation learning by comparing the generated sentences and the ground truth in a supervised way. However, they neglect to explore linguistic semantics adequately due to the insufficient learning of visual words. In this paper, we propose the Semantic Learning Network (SLN) that explicitly learns specific semantics of all the visual words by aggregating static and dynamic features. Besides, to achieve controllable video captioning and alleviate the problem that one identical video is mapped to multiple annotations during training, we propose the predicate-based feature selection approach to convert the video to different textual features under the guidance of the predicate in the captions. Experimental results on the MSVD and MSR-VTT datasets show that our SLN outperforms recent state-of-the-art methods.
Capsule Transformer Network for Dynamic Hand Gesture Recognition Using Multimodal Data

In recent years, deep learning techniques have achieved remarkable success in video analysis and more especially in action and gesture recognition. Even though convolutional neural networks (CNNs) remain the most widely used models, they have difficulty in capturing the global contextual information involving spatial and temporal domains or intermodality due to the local feature learning mechanism. This paper introduces a Capsule Transformer Network, which composed of a frame capsule module for extracting hand features and a gesture transformer module for modeling the temporal features and recognizing the dynamic gesture. Spatial attention is ensured through the capsule module to enhance the spatial information of the hand image, while the transformer module guarantees temporal attention through gesture sequence. We propose to use multimodal data, including RGB, depth and IR data, which improves the accuracy of our approach as it better captures the 3D structure of the hand and can distinguish between similar hand gestures. Testing on two datasets, Briareo and SHREC17, the proposed approach outperforms or equals previous methods.
JPEG Pleno Light Field Encoder with Mesh based View Warping

We introduce mesh-based view warping to the JPEG Pleno light field coding framework and replace the standardized sample-based forward warping and splatting of reference texture with mesh-based backward warping, which allows for a more disciplined interpolation of the reference texture for predicting the target view. Instead of coding depth maps with JPEG 2000, which is the default option of the JPEG Pleno framework, we employ a recent extension referred to as JPEG 2000 Part 17. This extension utilises breakpoints to describe discontinuity boundary geometry for the purpose of modifying the predict and update lifting steps in the vicinity of detected discontinuities. We directly decode breakpoints and corresponding DWT coefficients onto a mesh and describe a scheme to construct a single, consolidated mesh for a large group of views, borrowing information from multiple coded depth maps. Results show that the cumulative impact of all these modifications enable improved rate-distortion performance in comparison with the default operation of the JPEG Pleno encoder.
Distilling Knowledge of Bidirectional Language Model for Scene Text Recognition

This paper proposes a knowledge distillation method for an external bidirectional language model trained by masked language modeling to achieve high accuracy in scene text recognition. In Asian languages such as Japanese, it is necessary to perform text recognition in units of multiple words or sentences rather than individual words because words are not separated by spaces, and so high-level linguistic knowledge is needed to recognize text correctly. To enhance linguistic knowledge, several methods that use an external language model have been proposed, but these methods fail to consider future context well in performing text recognition because they revise the text candidates yielded by autoregressive text recognition models, which consider mainly past context. To overcome this deficiency, our key idea is to enhance a text recognition model by utilizing knowledge of an external bidirectional language model trained by masked language modeling, which reflects not only past but also future context. So as to actively consider future context in text recognition, our proposed method introduces a distillation loss term that makes the output probability of the text recognition model closer to that of the bidirectional language model. Experiments on Japanese scene text recognition demonstrate the effectiveness of the proposed method.
Overlap Loss: Rethinking Weakly Supervised Instance Segmentation in Crowded Scenes

Weakly supervised instance segmentation (WSIS) has gained increasing popularity in recent years due to low labelling cost. However, its performance deteriorates dramatically in more challenging crowded scenario, which is caused by overlapping among similar objects. To ameliorate the negative effects of instance overlapping, we propose a new loss, i.e., OverlapLoss, which achieves instance disentanglement between masks according to the degree of overlapping among instances. Besides, a new dataset of CrowdHuman Instance Segmentation (CIS) is presented to bridge the gap in crowded scenes. Experiments on the CIS and COCO datasets validate that the proposed loss can improve the baseline in typical crowed scenes by at least 2% and in uncrowded scenes by more than 0.3% w.r.t. absolute AP. The code and dataset are available at: https://github.com/shanghangjiang/CIS.
NUCQ: Non-Uniform Conditional Quantization for Learned Image Compression

Learned Image Compression (LIC) has surpassed traditional image compression in terms of Rate-Distortion (R-D) performance with strong and powerful non-linear transformations. However, most existing LIC approaches discretize the latent representation into integers. Such method remains the statistical redundancy and conditional information of image content, which could be further leveraged to systematically reduce bit rate in the lossy compression regime. In this work, we formulate the problem of Non-Uniform Conditional Quantization (NUCQ) in LIC and accordingly propose to learn a content-adaptive NUCQ scheme conditioned on image textural prior information. In our method, a parameterized quantile model is learned from the hyperprior information, thus more content adaptive quantization is realized in the latent domain. Experiments demonstrate that our proposed NUCQ improves the R-D performance of LIC by 4.83% and 6.03% on the Bjøntegaard-delta bit rate (BD-rate) metric concerning the MSE and MS-SSIM metric, respectively.
Robust Feature Learning Against Noisy Labels

Supervised learning of deep neural networks heavily relies on large-scale datasets annotated by high-quality labels. In contrast, mislabeled samples can significantly degrade the generalization of models and result in memorizing samples, further learning erroneous associations of data contents to incorrect annotations. To this end, this paper proposes an efficient approach to tackle noisy labels by learning robust feature representation based on unsupervised augmentation restoration and cluster regularization. In addition, progressive self-bootstrapping is introduced to minimize the negative impact of supervision from noisy labels. Our proposed design is generic and flexible in applying to existing classification architectures with minimal overheads. Experimental results show that our proposed method can efficiently and effectively enhance model robustness under severely noisy labels.
Pushing the Limits of the Wiener Filter in Image Denoising

As modern image denoiser networks have grown in size, their reported performance in popular real noise benchmarks such as DND and SIDD have now long outperformed classic non-deep learning denoisers such as Wiener and Wavelet-based methods. In this paper, we propose to revisit the Wiener filter and re-assess its potential performance. We show that carefully considering the implementation of the Wiener filter can yield similar performance to popular networks such as DnCNN.
A two-Dimensional Difference Histogram Equalization with fuzzy Cumulative Distribution Correction for dark Images

Aiming at the problems of over-enhancement, detail loss, and poor subjective visual effect in dark images enhancement, this paper proposes a two-dimensional difference histogram equalization (2DDHE) algorithm using fuzzy C-means (FCM) to cluster and correct the cumulative distribution to enhance the dark image. First, a construction method of the two-dimensional (2D) difference histogram is proposed, which can enhance the dark area details. Then the cumulative distribution correction method based on FCM clustering and membership matrix modification is introduced to improve the over-enhancement and subjective visual effect. The experimental results show that the enhanced results by this method have a better subjective visual effect and outperform many existing algorithms in contrast and detail enhancement.
Comparative Study of Saliency- and Scanpath-Based Approaches for Patch Selection in Image Quality Assessment

Over the past few years, the development of deep learning-based methods has revolutionised the field of image quality assessment. These methods have shown remarkable success in estimating the quality of 2D images. However, most of these approaches are trained using small patches of the image, with the subjective score of the entire image serving as the target. assuming that all patches have an equal perceptual impact on the image. This assumption is not entirely consistent with our Human Visual System (HVS), which processes different regions of the image to form an overall perception. In this study, we focus on the use of saliency information to estimate image quality. In this context, we explore the use of saliency information to estimate image quality by selecting only the most perceptually relevant patches. Specifically, we evaluate the accuracy of saliency- or scanpath-based patch selection methods for predicting 2D image quality. Our goal is to determine which approach provides the most accurate estimation of image quality and whether the use of saliency information can improve the performance of deep learning-based methods for image quality assessment.
Task-Adaptive Feature Matching Loss for Image Deblurring

Image deblurring is a highly challenging and ill-posed image restoration problem. Contemporary deep learning-based approaches usually tackle this problem by exploiting the encoder-decoder-based models trained by the commonly used mean squared error loss with the feature matching loss as a regularization to obtain perceptual consistent restored results as the ground truths. We argue that since the general backbone models for computing feature matching loss are usually not trained on the image deblurring task, the loss lacks specific knowledge of blur and usually leads to suboptimal performance. To address this issue, we propose a task-adaptive feature matching loss for image deblurring where we synthesize blurred images in different blur extents and employ triplet loss to finetune the backbone model for learning specific blur priors. Then, we leverage the finetuned backbone to compute feature matching loss which can greatly enhance the existing image deblurring models for better perceptual results. With extensive experiments on the GoPro and RealBlur datasets, both qualitative and quantitative results show that the SOTA deblurring models trained with the proposed loss can effectively obtain better and sharper restored images in terms of various perceptual image quality metrics than the original models while maintaining comparable PSNR and SSIM performances.
Spatio-Temporal Perception-Distortion Trade-Off in Learned Video SR

Perception-distortion trade-off is well-understood for single-image super-resolution. However, its extension to video super-resolution (VSR) is not straightforward, since popular perceptual measures only evaluate naturalness of spatial textures and do not take naturalness of flow (temporal coherence) into account. To this effect, we propose a new measure of spatio-temporal perceptual video quality emphasizing naturalness of optical flow via the perceptual straightness hypothesis (PSH) for meaningful spatio-temporal perception-distortion trade-off. We also propose a new architecture for perceptual VSR (PSVR) to explicitly enforce naturalness of flow to achieve realistic spatio-temporal perception-distortion trade-off according to the proposed measures. Experimental results with PVSR support the hypothesis that a meaningful perception-distortion tradeoff for video should account for the naturalness of motion in addition to naturalness of texture.
Enhanced Temporal Motion Derivation Beyond VVC

Temporal motion information has been widely exploited in advanced video coding standards to facilitate the efficiency of inter coding. In prior-arts, temporal cues are derived from fixed locations in a single collocated frame without considering trajectory consistency. This paper proposes to enhance temporal motion derivation in two aspects. On one hand, two collocated frames are utilized to provide more abundant temporal motion information from both forward and backward directions. On the other hand, motion shifts to locate regular temporal motion vector prediction (TMVP) or sub-block based TMVP (SbTMVP) are adaptively determined from multiple locations according to template matching costs. Simulation results indicate the proposed methods can achieve 0.11% and 0.15% coding gains respectively for RA and LDB configurations on ECM-7.0, with negligible complexity increase. The proposed method has been adopted into ECM-8.0.
MTJND: Multi-Task Deep Learning Framework for Improved JND Prediction

The limitation of the Human Visual System (HVS) in perceiving small distortions allows us to lower the bitrate required to achieve a certain visual quality. Predicting and applying the Just Noticeable Distortion (JND), which is a threshold for maximum unperceived level of distortions, is among the popular ways to do so. Recently, machine learning based methods have been able to reduce bitrate even further by improving JND prediction accuracy. However, accurate modeling of JND is very challenging, as it is highly content dependent. Furthermore, existing datasets provide little information to learn the best parameters. To remedy this issue, we propose a multi-task deep learning framework that jointly learns various complementary visual information. We design three separate methods and training strategies that jointly learn: (1) three JND levels, (2) visual attention map and a JND level, and (3) three JND levels and the visual attention map. We show that accumulating information from multiple tasks leads to a more robust prediction of JND. Experimental results confirm the superiority of our framework compared to the state-of-the-art.
Neural Augmented Exposure Interpolation for HDR Imaging

Brightness order reversal usually appears when two large-exposure-ratio images of a high dynamic range scene are directly fused together by an existing multi-scale exposure fusion algorithm. To address the problem, a novel neural augmented framework is introduced to interpolate an image with the medium exposure by integrating physics-driven and data-driven approaches. The physics-driven method infers high-frequency information while the data-driven approach learns remaining information for the interpolated image. The interpolated image and two large-exposure-ratio images are fused together. Experimental results show that the proposed framework can indeed solve the brightness order reversal problem for the fusion of of two large-exposure-ratio images.