Reviewerly/exHarmony huggingface dataset:
Sample Weighted Multiple Kernel K-means via Min-Max optimization

A representative multiple kernel clustering (MKC) algorithm, termed simple multiple kernel k-means (SMKKM), is recently proposed to optimally mine useful information from a set of pre-specified kernels to improve clustering performance. Different from existing min-min learning framework, it puts a novel min-max optimization manner, which attracts considerable attention in related community. Despite achieving encouraged success, we observe that SMKKM only focuses on combination coefficients among kernels and ignores the relationship among the importance of different samples. As a result, it does not sufficiently consider different contributions of each sample to clustering, and thus cannot effectively obtain the "ideal" similarity structure, leading to unsatisfying performance. To address this issue, this paper proposes a novel sample weighted multiple kernel k-means via min-max optimization (SWMKKM), which sufficiently considers the sum of relationship between one sample and the others to represent the sample weights. Such a weighting criterion helps clustering algorithm pay more attention to samples with more positive effects on clustering and avoids unreliable overestimation for samples with poor quality. Based on SMKKM, we adopt a reduced gradient algorithm with proved convergence to solve the resultant optimization problem. Comprehensive experiments on multiple benchmark datasets demonstrate that our proposed SWMKKM dramatically improves the state-of-the-art MKC algorithms, verifying the effectiveness of our proposed sample weighting criterion.
Image Inpainting Detection via Enriched Attentive Pattern with Near Original Image Augmentation

As deep learning-based inpainting methods have achieved increasingly better results, its malicious use, e.g. removing objects to report fake news or to provide fake evidence, is becoming threatening. Previous works have provided rich discussions on network architectures, e.g. even performing Neural Architecture Search to obtain the optimal model architecture. However, there are rooms in other aspects. In our work, we provide comprehensive efforts from data and feature aspects. From the data aspect, as harder samples in the training data usually lead to stronger detection models, we propose near original image augmentation that pushes the inpainted images closer to the original ones (without distortion and inpainting) as the input images, which is proved to improve the detection accuracy. From the feature aspect, we propose to extract the attentive pattern. With the designed attentive pattern, the knowledge of different inpainting methods can be better exploited during the training phase. Finally, extensive experiments are conducted. In our evaluation, we consider the scenarios where the inpainting masks, which are used to generate the testing set, have a distribution gap from those masks used to produce the training set. Thus, the comparisons are conducted on a newly proposed dataset, where testing masks are inconsistent with the training ones. The experimental results show the superiority of the proposed method and the effectiveness of each component. All our codes and data will be online available.
Domain Reconstruction and Resampling for Robust Salient Object Detection

Salient Object Detection (SOD) aims at detecting the salient objects covering the whole natural scene. However, one of the main problems in SOD is data bias. Natural scenes vary greatly, while each image in the SOD dataset contains a specific scene. It means that each image is just a sampling point in a specific scene, which is not representative and causes serious sampling bias. Building larger datasets is one solution but costly to address the sampling bias. Our method regards the data distribution of natural scenes as a Gaussian Mixture Distribution, and each scene follows a sub-Gaussian distribution. Our main idea is to reconstruct the data distribution of each scene from the sampling images and then resample from the distribution domain. We represent a scene by a distribution instead of a fixed sampling image to reserve the sampling uncertainty in SOD. Specifically, we employ a Style Conditional Variational AutoEncoder (Style-CVAE) to reconstruct the data distribution from image styles and a Gaussian Randomize Attribute Filter (GRAF) to reconstruct data distribution from image attributes (such as lightness, saturation, hue, etc.). We resample the reconstructed data distribution according to the Gaussian probability density function and train the SOD model. Experimental results prove that our method outperforms 16 state-of-the-art methods on five benchmarks.
Asymmetric Adversarial-based Feature Disentanglement Learning for Cross-Database Micro-Expression Recognition

Recently, micro-expression recognition (MER) has gained tremendous progress. However, most methods are based on individual-database micro-expression recognition and are difficult to generalize into complicated scenarios. Therefore, cross-database micro-expression recognition (CDMER) has drawn growing attention due to its robustness and generalizability. In this paper, we propose a novel CDMER algorithm with asymmetric adversarial-based feature disentanglement learning, which implements the disentanglement of domain features and emotion features aiming to learn domain-invariant and discriminative representation. Furthermore, to facilitate the feature disentanglement learning, a Domain Information Filtering (DF) module is designed to filter out the domain component of the micro-expression features (emotion feature). Extensive experiments on the SMIC and CASME II databases have shown that our proposed method outperforms the state-of-the-art method and has superior performance against excessive domain discrepancies.
ADGNet: Attention Discrepancy Guided Deep Neural Network for Blind Image Quality Assessment

This work explores how to efficiently incorporate semantic knowledge for blind image quality assessment and proposes an end-to-end attention discrepancy guided deep neural network for perceptual quality assessment. Our method is established on a multi-task learning framework in which two sub-tasks including semantic recognition and image quality prediction are jointly optimized with a shared feature-extracting branch and independent spatial-attention branch. The discrepancy between semantic-aware attention and quality-aware attention is leveraged to refine the quality predictions. The proposed ADGNet is based on the observation that human visual systems exhibit different mechanisms when viewing images with different amounts of distortion. Such a manner would result in the variation of attention discrepancy between the quality branch and semantic branch, which are therefore employed to enhance the accuracy and generalization ability of our method. We systematically study the major components of our framework, and experimental results on both authentically and synthetically distorted image quality datasets demonstrate the superiority of our model as compared to the state-of-the-art approaches.
TransCNN-HAE: Transformer-CNN Hybrid AutoEncoder for Blind Image Inpainting

Blind image inpainting is extremely challenging due to the unknown and multi-property complexity of contamination in different contaminated images. Current mainstream work decomposes blind image inpainting into two stages: mask estimating from the contaminated image and image inpainting based on the estimated mask, and this two-stage solution involves two CNN-based encoder-decoder architectures for estimating and inpainting separately. In this work, we propose a novel one-stage Transformer-CNN Hybrid AutoEncoder (TransCNN-HAE) for blind image inpainting, which intuitively follows the inpainting-then-reconstructing pipeline by leveraging global long-range contextual modeling of Transformer to repair contaminated regions and local short-range contextual modeling of CNN to reconstruct the repaired image. Moreover, a Cross-layer Dissimilarity Prompt (CDP) is devised to accelerate the identifying and inpainting of contaminated regions. Ablation studies validate the efficacy of both TransCNN-HAE and CDP, and extensive experiments on various datasets with multi-property contaminations show that our method achieves state-of-the-art performance with much lower computational cost on blind image inpainting. Our code is available at https://github.com/zhenglab/TransCNN-HAE.
Micro Expression Generation with Thin-plate Spline Motion Model and Face Parsing

Micro-expression generation aims at transfering the expression from the driving videos to the source images, which can be viewed as a motion transfer task. Recently, several works have been proposed to tackle this problem and achieve great performance. However, due to the intrinsic complexity of the face motion and different attributes of face regions, the task still remains challenging. In this paper, we propose an end-to-end unsupervised motion transfer network to tackle this challenge. As the motion of the face is non-rigid, we adopt an effective and flexible thin-plate spline motion estimation method to estimate the optical flow of the face motion. What's more, we find that several faces with eyeglasses show weird deformation in motion transfering. Thus, we introduce face parsing method to pay specific attention to the eyeglasses regions to ensure the reasonability of the deformation. We conduct several experiments on the provided datasets of the ACM MM 2022 micro-expression grand challenge (MEGC2022) and compare our method with several other typical methods. In comparison, our method shows the best performance. We (Team: USTC-IAT-United) also compare our method with other competitors' in MEGC2022, and the expert evaluation results show that our method performs best, which verifies the effectiveness of our method. Our code is available at https://github.com/HowToNameMe/micro-expression
Alleviating Style Sensitivity then Adapting

Recently, source-free domain adaptation (SFDA) has attracted extensive attention in medical image segmentation due to the ability of knowledge transfer without accessing source data. However, existing SFDA methods suffer from severe performance degradation since the style of the target data shifts from the source. Although traditional unsupervised domain adaptation (UDA) methods are capable of addressing the style shifts issue using both domain data, they fail to extract the source style due to a lack of source data in source-free scenarios. In this paper, we propose a novel style-insensitive source-free domain adaptation framework (SI-SFDA) for medical image segmentation to reduce the impacts of style shifts. The proposed framework first pretrains a generalized source model and then adapts the source model in a source data-free manner. Towards the former, a cross-patch style generalization (CPSG) mechanism is introduced to reduce the style sensitivity of the source model via a self-training paradigm with Transformer structure. Towards the latter, an adaptive confidence regularization (ACR) loss with dynamic scaling strategy is developed to further reduce the classification confusion caused by style shifts. The proposed ACR loss is model-independent so that it can be used with other methods to improve the segmentation performance. Extensive experiments are conducted on five public medical image benchmarks, the promising performance on organ and fundus segmentation tasks demonstrates the effectiveness of our framework.
GCL

Despite the great success of Graph Neural Networks (GNNs), the trustworthiness is still lack-explored. A very recent study suggests that GNNs are under-confident on the predictions which is opposite to deep neural networks. In this paper, we investigate why this is the case. We discover that the "shallow" network of GNNs is the central cause. To address this challenge, we propose a novel Graph Calibration Loss (GCL), the first end-to-end calibration method for GNNs, which reshapes the standard Cross Entropy loss and is encouraged to assign up-weights loss to high-confidence examples. Through empirical observation and theoretical justification, we discover the GCL's calibration mechanism is to add a minimal-entropy regulariser to KL-divergence to bring down the entropy of correctly classified samples. To evaluate the effectiveness of the GCL, we train several representative GNNs models which use the GCL as loss function on various citation networks datasets, and further apply the GCL to a self-training framework. Compared to the existed methods, the proposed method achieves state-of-the-art calibration performance on node classification task and even improves the standard classification accuracy in almost all cases.
DomainPlus: Cross Transform Domain Learning towards High Dynamic Range Imaging

High dynamic range (HDR) imaging by combining multiple low dynamic range (LDR) images of different exposures provides a promising way to produce high quality photographs. However, the misalignment between the input images leads to ghosting artifacts in the reconstructed HDR image. In this paper, we propose a cross-transform domain neural network for efficient HDR imaging. Our approach consists of two modules: a merging module and a restoration module. For the merging module, we propose a Multiscale Attention with Fronted Fusion (MAFF) mechanism to achieve coarse-to-fine spatial fusion. For the restoration module, we propose fronted Discrete Wavelet Transform (DWT) and Discrete Cosine Transform (DCT)-based learnable bandpass filters to formulate a cross-transform domain learning block, dubbed DomainPlus Block (DPB) for effective ghosting removal. Our ablation study and comprehensive experiments show that DomainPlus outperforms the existing state-of-the-art on several datasets.
Recurrent Meta-Learning against Generalized Cold-start Problem in CTR Prediction

During the last decades, great success has been witnessed along the course of accurate Click-Through-Rate (CTR) prediction models for online advertising. However, the cold-start problem, which refers to the issue that the standard models can hardly draw accurate inferences for unseen users/ads, is still yet to be fully understood. Most recently, some related studies have been proposed to tackle this problem with only the new users/ads being considered. We argue that such new users/ads are not the only sources for cold-start. From another perspective, since users might shift their interests over time, one's recent behaviors might vary greatly from the records long ago. In this sense, we believe that the cold-start problem should also exist along the temporal dimension. Motivated by this, a generalized definition of the cold-start problem is provided where both new users/ads and recent behavioral data from known users are considered. To attack this problem, we propose a recursive meta-learning model with the user's behavior sequence prediction as a separate training task. Specifically, a time-series CTR model with the MAML (Model-Agnostic Meta-Learning)-like meta-learning method is proposed to make our model adapt to new tasks rapidly. Besides, we propose a parallel structure for extracting the feature interactions to efficiently fuse attention mechanisms and the RNN layer. Finally, experiments on three public datasets demonstrate the effectiveness of the proposed approaches.
High-Quality 3D Face Reconstruction with Affine Convolutional Networks

Recent works based on convolutional encoder-decoder architecture and 3DMM parameterization have shown great potential for canonical view reconstruction from a single input image. Conventional CNN architectures benefit from exploiting the spatial correspondence between the input and output pixels. However, in 3D face reconstruction, the spatial misalignment between the input image (e.g. face) and the canonical/UV output makes the feature encoding-decoding process quite challenging. In this paper, to tackle this problem, we propose a new network architecture, namely the Affine Convolution Networks, which enables CNN based approaches to handle spatially non-corresponding input and output images and maintain high-fidelity quality output at the same time. In our method, an affine transformation matrix is learned from the affine convolution layer for each spatial location of the feature maps. In addition, we represent 3D human heads in UV space with multiple components, including diffuse maps for texture representation, position maps for geometry representation, and light maps for recovering more complex lighting conditions in the real world. All the components can be trained without any manual annotations. Our method is parametric-free and can generate high-quality UV maps at resolution of 512 x 512 pixels, while previous approaches normally generate 256 x 256 pixels or smaller. Our code will be released once the paper got accepted.
Camera-specific Informative Data Augmentation Module for Unbalanced Person Re-identification

Person re-identification~(Re-ID) aims at retrieving the same person across the non-overlapped camera networks. Recent works have achieved impressive performance due to the rapid development of deep learning techniques. However, most existing methods have ignored the practical unbalanced property in real-world Re-ID scenarios. In fact, the number of pedestrian images in different cameras vary a lot. Some cameras cover thousands of images while others only have a few. As a result, the camera-unbalanced problem will reduce intra-camera diversity, then the model cannot learn camera-invariant features to distinguish pedestrians from "poor" cameras. In this paper, we design a novel camera-specific informative data augmentation module~(CIDAM) to alleviate the proposed camera-unbalanced problem. Specifically, we first calculate the camera-specific distribution online, then refine the "poor" camera-specific covariance matrix with similar cameras defined in the prototype-based similarity matrix. Consequently, informative augmented samples are generated by combining original samples with sampled random vectors in feature space. To ensure these augmented samples can better benefit the model training, we further propose a dynamic-threshold-based contrastive loss. Since augmented samples may not be as real as original ones, we calculate a threshold for each original one dynamically and only push hard negative augmented samples away. Moreover, our CIDAM can be compatible with a variety of existing Re-ID methods. Extensive experiments prove the effectiveness of our method.
IVT: An End-to-End Instance-guided Video Transformer for 3D Pose Estimation

Video 3D human pose estimation aims to localize the 3D coordinates of human joints from videos. Recent transformer-based approaches focus on capturing the spatiotemporal information from sequential 2D poses, which cannot model the contextual depth feature effectively since the visual depth features are lost in the step of 2D pose estimation. In this paper, we simplify the paradigm into an end-to-end framework, Instance-guided Video Transformer (IVT), which enables learning spatiotemporal contextual depth information from visual features effectively and predicts 3D poses directly from video frames. In particular, we firstly formulate video frames as a series of instance-guided tokens and each token is in charge of predicting the 3D pose of a human instance. These tokens contain body structure information since they are extracted by the guidance of joint offsets from the human center to the corresponding body joints. Then, these tokens are sent into IVT for learning spatiotemporal contextual depth. In addition, we propose a cross-scale instance-guided attention mechanism to handle the variational scales among multiple persons. Finally, the 3D poses of each person are decoded from instance-guided tokens by coordinate regression. Experiments on three widely-used 3D pose estimation benchmarks show that the proposed IVT achieves state-of-the-art performances.
Few-Shot Model Agnostic Federated Learning

Federated learning has received increasing attention for its ability to collaborative learning without leaking privacy. Promising advances have been achieved under the assumption that participants share the same model structure. However, when participants independently customize their models, models suffer communication barriers, which leads the model heterogeneity problem. Moreover, in real scenarios, the data held by participants is often limited, making the local models trained only on private data present poor performance. Consequently, this paper studies a new challenging problem, namely few-shot model agnostic federated learning, where the local participants design their independent models from their limited private datasets. Considering the scarcity of the private data, we propose to utilize the abundant public available datasets for bridging the gap between local private participants. However, its usage also brings in two problems: inconsistent labels and large domain gap between the public and private datasets. To address these issues, this paper presents a novel framework with two main parts: 1) model agnostic federated learning, it performs public-private communication by unifying the model prediction outputs on the shared public datasets; 2) latent embedding adaptation, it addresses the domain gap with an adversarial learning scheme to discriminate the public and private domains. Together with theoretical generalization bound analysis, comprehensive experiments under various settings have verified our advantage over existing methods. It provides a simple but effective baseline for future advancement. The code is available at https://github.com/WenkeHuang/FSMAFL.
KAConv: Kernel attention convolutions

Most of the previous network optimization works applied attention mechanism to feature maps, but neglected to embed attention into convolution kernel of the end-to-end network that is convenient to deploy. To address this issue, we present a novel attention convolution method named Kernel Attention Convolution (KAConv) to enhance the flexibility of convolution. The proposed KAConv generates different attention weights for different spatial positions of convolution kernels based on the input features, so as to dynamically adjust the parameters of convolution kernels during the forward propagation to enhance the flexibility of convolution. We decompose the convolution kernels into subkernels spatially, and generate the corresponding feature maps through which attention weights are obtained. The final refined feature maps are aggregated by the attention weighted feature maps corresponding to each subkernel. KAConv is a computationally lightweight convolution method, which not only incorporates attention into kernels but also enhances informative representations. By replacing the standard convolution with the proposed KAConv in convolutional neural networks (CNNs), the networks yield significant performance improvement. Extensive experiments on the ImageNet-1K benchmark demonstrate that KAConv outperforms existing attention mechanism-based methods. We also carry out experiments on the MS COCO and PASCAL VOC datasets to show the generalization ability of our method.
DavarOCR: A Toolbox for OCR and Multi-Modal Document Understanding

This paper presents DavarOCR, an open-source toolbox for OCR and document understanding tasks. DavarOCR currently implements 19 advanced algorithms, covering 9 different task forms. DavarOCR provides detailed usage instructions and the trained models for each algorithm. Compared with the previous open-source OCR toolbox, DavarOCR has relatively more complete support for the sub-tasks of the cutting-edge technology of document understanding. In order to promote the development and application of OCR technology in academia and industry, we pay more attention to the use of modules that different sub-domains of technology can share. DavarOCR is publicly released at https://github.com/hikopensource/Davar-Lab-OCR.