Reviewerly/exHarmony huggingface dataset:
DTT-Net: Dual-Domain Translation Transformer for Semi-Supervised Image Deraining

Domain gap between synthetic and real rain has impeded advances in natural image deraining task. Existing methods are mostly built on convolutional neural networks (CNNs) and the receptive field of CNNs is limited, thereby resulting in poor domain adaptation. This paper designs a dual-domain translation Transformer network (termed DTT-Net) for semi-supervised image deraining. By leveraging Transformer architecture, the proposed DTT-Net can significantly mitigate the domain gap, greatly boosting the performance on real-world rainy images. Meanwhile, DTT-Net integrates three loss functions including adversarial, cycle-consistency, and MSE losses to adversarial training to further improve the visual quality of the derained images. Extensive experiments are conducted on synthetic and real-world rain datasets. Experimental results show that our DTT-Net outperforms the state-of-the-art by more than 2 dB PSNR. The source code is available at https://github.com/GZHU-DVL/DTT-Net.
For the Sake of Privacy: Skeleton-Based Salient Behavior Recognition

Authorities as well as emergency and rescue services have an increasing interest in smart support systems to ensure public safety which includes in particular behavioral analysis of pedestrians by using video surveillance systems. In order to accommodate concerns of citizens regarding their personal rights, the demand for data privacy friendly approaches, using as few information as possible, arises. In this paper, we examine existing approaches tackling the recognition of anomalous or salient behavior based solely on person pose information within the context of real-world surveillance applications. Particularly, we chose two existing state-of-the-art approaches and evaluate them on two public and an internal dataset in order to examine the overall performance of these methods for the desired task. Furthermore, we present our own approach achieving comparable results to these methods. Finally, we extend the aforementioned methods with a memory extension for modeling normal behavior, which yields on average a 4.3% higher recognition performance.
Conmw Transformer: A General Vision Transformer Backbone With Merged-Window Attention

Recently, the application of Transformer in computer vision has shown us the potential of this new paradigm. However, standard multi-head Attention (MSA) faces an explosion of computational cost as the input changes from a sequence of text to an image, and MSA is computationally redundant for images. In this paper, we propose a new backbone network combining window-based attention and convolutional neural networks named ConMW Transformer, introducing convolution into the Transformer to help it converge quickly and improve accuracy. ConMW Transformer use a hierarchical architecture, an inductive bias is incorporated during tokenization and feature projection. We reduce the computational cost by performing the attention operation within windows after partition the feature map, while allowing connections between multiple heads for a more appropriate joint representation. We also use large kernel convolution after the window-based attention to merge features between windows, which help maintaining the superiority of attention in global context modelling. With only ImageNet-1K pre-training using 224 × 224 resolution, our base model can achieve 83.7% top-1 accuracy on ImageNet-1K and 49.9 mIoU for semantic segmentation on ADE20K.
Context Relation Fusion Model for Visual Question Answering

Traditional VQA models tend to rely on language priors as a shortcut to answer questions and neglect visual information. To solve this problem, the latest approaches divide language priors into "good" language context and "bad" language bias through global features to benefit the language context and suppress the language bias. However, language priors cannot be meticulously divided by global features. In this paper, we propose a novel Context Relation Fusion Model (CRFM), which produces comprehensive contextual features forcing the VQA model to more carefully distinguish language priors into "good" language context and "bad" language bias. Specifically, we utilize the Visual Relation Fusion Model (VRFM) and Question Relation Fusion Model (QRFM) to learn local critical contextual information and then perform information enhancement through the Attended Features Fusion Model (AFFM). Experiments show that our CRFM achieves state-of-the-art performance on the VQA-CP v2 dataset.
A CNN-Based Post-Processor for Perceptually-Optimized Immersive Media Compression

In recent years, resolution adaptation based on deep neural networks has enabled significant performance gains for conventional (2D) video codecs. This paper investigates the effectiveness of spatial resolution resampling in the context of immersive content. The proposed approach reduces the spatial resolution of input multi-view videos before encoding, and reconstructs their original resolution after decoding. During the up-sampling process, an advanced CNN model is used to reduce potential re-sampling, compression, and synthesis artifacts. This work has been fully tested with the TMIV coding standard using a Versatile Video Coding (VVC) codec. The results demonstrate that the proposed method achieves significant rate-quality performance improvement for the majority of the test sequences, with an average BD-VMAF improvement of 3.07 over all sequences.
Partition and Reunion: A Viewpoint-Aware Loss for Vehicle Re-Identification

Vehicle Re-Identification (ReID) aims to retrieve images of vehicles with the same identity from different scenarios. It is a challenging task due to the large intra-identity discrepancy caused by viewpoint variations and the subtle inter-identity difference produced by similar appearances. In this paper, we propose a Viewpoint-Aware Loss (VAL) function to deal with these challenges. Specifically, we propose partition and reunion operations in VAL, which significantly shrinks the intra-identity distance and acquires viewpoint-invariant representations. In addition, we embed a multi-decision boundary mechanism in VAL. It contributes to enlarging the inter-identity distance. A comprehensive evaluation on two benchmarks shows the superiority of our method in contrast to a series of existing state-of-the-arts.
Semi-Supervised Ranking for Object Image Blur Assessment

Assessing the blurriness of an object image is fundamentally important to improve the performance for object recognition and retrieval. The main challenge lies in the lack of abundant images with reliable labels and effective learning strategies. Current datasets are labeled with limited and confused quality levels. To overcome this limitation, we propose to label the rank relationships between pairwise images rather their quality levels, since it is much easier for humans to label, and establish a large-scale realistic face image blur assessment dataset with reliable labels. Based on this dataset, we propose a method to obtain the blur scores only with the pairwise rank labels as supervision. Moreover, to further improve the performance, we propose a self-supervised method based on quadruplet ranking consistency to leverage the unlabeled data more effectively. The supervised and self-supervised methods constitute a final semi-supervised learning framework, which can be trained end-to-end. Experimental results demonstrate the effectiveness of our method. Source of labeled datasets: https://github.com/yzliangHIK2022/SSRanking-for-Object-BA
Dual Path Cross-Scale Attention Network For Image Inpainting

Existing deep inpainting methods often generate contents with blurry textures and distorted structures, which manifests to well fuse the high-level structures and the low-level textures remains challenging. Meanwhile, current inpainting networks just process features at the encoder side but neglect the strong generative abilities in decoder pathway. To address above challenges, we propose a novel Dual Path Cross-Scale Attention Network (DPCSAN). It adopts U-Net architecture as backbone. Based on contextual attention, a Cross-Scale Attention Module (CSAM) is designed to transfer high-level structure information to adjacent low-level features across scales. CSAMs are equipped between each encoder layer and the symmetric decoder layer at multiple levels. The purpose of this design is to fill holes in shallower feature maps and to refine features generated by upsampling (with higher resolution). Experiments on Facade and CelebA-HQ demonstrate that our model performs better than state-of-the-art methods, while generating richer details and keeping original structures.
Stacking More Linear Operations with Orthogonal Regularization to Learn Better

How to improve the generalization of CNN models has been a long-lasting problem in the deep learning community. This paper presents a runtime parameter/FLOPs-free method to strengthen CNN models by stacking linear convolution operations during training. We show that overparameterization with appropriate regularization can lead to a smooth optimization landscape that improves the performance. Concretely, we propose to add a 1 × 1 convolutional layer before and after the original k × k convolutional layer respectively, without any non-linear activations between them. In addition, Quasi-Orthogonal Regularization is proposed to maintain the added 1 × 1 filters as orthogonal matrixes. After training, those two 1 × 1 layers can be fused into the original k × k layer without changing the original network architecture, leaving no extra computations at inference, i.e. parameter/FLOPs-free.
Dynamic Template Update for Visual Object Tracking

Siamese-based trackers have recently demonstrated impressive performance and high speed. Despite their great success, conventional siamese trackers are prone to be fooled when facing appearance variations of target objects because they refer to fixed templates captured from first frames to track target objects in the rest of videos. To address this issue, we propose a novel siamese-based tracking framework utilizing a dual template which consists of a static template and a dynamic template. The dynamic template is updated every update interval and allows the tracker to catch appearance variations of the target over time. Furthermore, we introduce a reliability score which prevents incorrect dynamic templates from degrading tracking performance to ensure reliable dynamic template updates. Experimental results show that our method possesses better discriminability and robustness than the baseline, which utilizes a single static template.
Self-Supervised Learning for Texture Classification Using Limited Labeled Data

Fine-grained texture classification differentiates between similar materials. When there are large unlabeled datasets available, representation learning is useful to distinguish between classes. In this paper, we show that harnessing contrastive self-supervised learning (SSL) for visual representations leads to performance gains for fine-grained texture classification. We demonstrate that, in the absence of sufficient labeled training data, SSL pre-training provides better representation for classification, when compared to supervised methods. We propose a novel pretext task, part-to-whole, in which we use the property of textures that a randomly cropped patch is similar in structure to the whole image. We also propose the usage of representations that are tapped from multiple layers of a convolutional neural network (CNN) and show the effectiveness of combining high-level and low-level features in improving discriminability. We present extensive experiments on the ground-terrain outdoor scenes (GTOS) dataset and show that multi-layer global average pooling (multi-GAP) representations from EfficientNet-B4 model trained using part-to-whole pretext task, beats the current state-of-the-art (SOTA) methods on single-view material classification in limited labeled data settings.
Multi Object Tracking Based on Uncertainty-Aware RE-ID

In multi-object tracking (MOT), many tracking-by-detection methods have been proposed, and intersection over union (IoU) is the most common re-identification (re-ID) method. However, IoU does not care about target motion and so that weak for large change of position, size, or disturbances that often occur in MOT scenes. In addition, only detections with a certain level of confidence are used for re-ID to avoid noise, which also exclude some positive detections and lead to fragmentation of a tracklet. In this paper, we propose a robust MOT method that represents the target motion by a combining multiple indices and performing re-ID by allowing uncertainty in each index which means one of them does not have to be satisfied for re-ID to deal with such changes and disturbances. Our re-ID also makes it possible to effectively utilize positive detections that were previously excluded, achieving top performance in comprehensive metrics MOTA, IDF1, and HOTA, and in robustness metrics such as AssA, Frag, and MT comparing to the state-of-the-art methods in MOT17 benchmark.
Exploring the Solution Space of Linear Inverse Problems with GAN Latent Geometry

Inverse problems consist in reconstructing signals from incomplete sets of measurements and their performance is highly dependent on the quality of the prior knowledge encoded via regularization. While traditional approaches focus on obtaining a unique solution, an emerging trend considers exploring multiple feasibile solutions. In this paper, we propose a method to generate multiple reconstructions that fit both the measurements and a data-driven prior learned by a generative adversarial network. In particular, we show that, starting from an initial solution, it is possible to find directions in the latent space of the generative model that are null to the forward operator, and thus keep consistency with the measurements, while inducing significant perceptual change. Our exploration approach allows to generate multiple solutions to the inverse problem an order of magnitude faster than existing approaches; we show results on image super-resolution and inpainting problems.
MONet: Multi-Scale Overlap Network for Duplication Detection in Biomedical Images

Manipulation of biomedical images to misrepresent experimental results has plagued the biomedical community for a while. Recent interest in the problem led to the curation of a dataset and associated tasks to promote the development of biomedical forensic methods. Of these, the largest manipulation detection task focuses on the detection of duplicated regions between images. Traditional computer-vision based forensic models trained on natural images are not designed to overcome the challenges presented by biomedical images. We propose a multi-scale overlap detection model to detect duplicated image regions. Our model is structured to find duplication hierarchically, so as to reduce the number of patch operations. It achieves state-of-the-art performance overall and on multiple biomedical image categories.
Scene Context Enhanced Network for Person Search

Person search is a practical but challenging task aiming to locate and identify the target person in unconstrained images. Most existing works are focused on people’s features while ignores scene features, which can help generate effective features for person identities. To address this issue, we present a Scene Contextual Enhanced Network(SCENet), by introducing attention refined scene contextual features to enhance person embedding. Also, a Scene Person Dissimilarity Loss is proposed to relieve the negative effects of irrelevant people in scene features. Experimental results on two person search benchmarks, i.e., CUHK-SYSU and PRW, demonstrate that our proposed model outperforms existing methods and achieves state-of-the-art performances.
Adversarial Training of Anti-Distilled Neural Network with Semantic Regulation of Class Confidence

Knowledge distillation (KD) has been identified as an effective knowledge transfer approach. By learning from the outputs of a pre-trained, over-parameterized teacher network, a compact student network can be trained efficiently to achieve superior performance. Although KD has gained substantial successes, exposure to pre-trained models usually causes potential risks of intellectual property leaks. From a model stealing attacker’s perspective, one can easily mimic the model functionality via KD, resulting in huge financial loss. In this paper, we propose a novel adversarial training framework called semantic nasty teacher, which prevents the teacher model from being copied by the attacker. In specific, we disentangle the semantic relationship in the output logits when training the teacher model, which is the key to success in KD. Experiment results show that neural networks trained with our approach only sacrifices little performance while canceling out the probability of KD-based model stealing.
Graph Refinement with Regression Prior for 3D Face Reconstruction

We propose the Graph Refinement with Regression Prior (GRRP) framework for 3D face reconstruction. The GRRP framework is composed of two component networks, namely the 3D Face Regressor and the Graph-Convolutional Refining (GCR) Network. The 3D Face Regressor is made of a face reconstruction pre-trained model, which is not updated during training and offers a pseudo target to guide the training of the GCR. The GCR proposed to design the graph-refining architecture with appropriate graph convolutional settings for enhancing 3D reconstruction accuracy. Apart from the network design, we also highlight the influences of two objective functions, the adaptive vertex loss and surface normal loss, and point out the difficulty of generating 3D faces of local details. The adaptive vertex loss is designed to adaptively optimize the local facial meshes so that the local undesired vertices can be rectified while the global mesh can be kept, and the surface normal loss is designed to eliminate the flying vertices problem of the generated shape surface. The proposed framework is verified through the experiments on the AFLW2000-3D and MICC Florence datasets and compared with contemporary approaches for evaluation.
DEKRV2: More Accurate or Fast than DEKR

Bottom-up human pose estimation has raised more investigation in recent years, especially 2D keypoints regression. However, the state-of-art DEKR [1] still has some aspects (e.g., speed and accuracy) to be improved. In this paper, we propose a new framework named DEKRv2, which has been enhanced compared to DEKR. When DEKR calculates the offset of each keypoint, it only considers the features of the current keypoint and neglects the constraints between the adjacent keypoints. We adopt a coarse-to-fine feature extraction method to obtain a more accurate feature location of keypoints for this problem. We also find that the multibranch network in DEKR is very time-consuming because it is serial. We designed a more effective module based on Group Convolution to replace the multi-branches network in DEKR, and it can reduce reasoning time. Experiments on the CrowdPose dataset show that our method achieves superior compared with DEKR in speed or accuracy, respectively. In the single-scale test, our method obtains 66.6 mAP, 0.6 higher than DEKR. The codes and models are available at https://github.com/chaowentao/DEKRv2.
Analysis of Video Quality Induced Spatio-Temporal Saliency Shifts

Human viewers’ eye movements reflect their perceptual responses to visual signals. Previous research has shown that distortions in videos cause spatio-temporal gaze shifts, which means gaze behaviour is related to video quality perception. It would be highly beneficial to understand gaze behaviour of viewing videos of varying perceived quality. However, little is known about the interactions between gaze, video content and distortions. In this paper, based on our eye-tracking database for video quality (SVQ160), we perform systematic analyses to reveal the impact of video content (VC) and time order (TO) on gaze shifts. Findings and quantitative methods for gaze behaviour can be used to develop advanced video quality metrics and video processing algorithms.
Image Data Augmentation with Unpaired Image-to-Image Camera Model Translation

Many image datasets are built from web searches, with images taken by various cameras. The variance of camera sources can lead to different camera signals and colors within images of the same class, which may impede neural networks from fitting the data. To generalize neural networks to different camera sources, we propose an augmentation method using unpaired image-to-image translation to transfer training images into another camera model domain. Our approach utilizes CycleGAN to create a translation mapping between two different camera models. We show that such a mapping can be applied to any image as a form of data augmentation and is able to outperform traditional color-based transformations. Additionally, this approach can be further enhanced with geometric transformations.
Transformers for Workout Video Segmentation

Temporal analysis of workout routines comes with a major difficulty: the observed actions or processes are naturally sequences of variable length. Besides being highly dependent on the exercise itself, the faster or slower repetition of the pattern’s temporal dynamics is also determined by the individual’s performance. In this paper, we present a Transformer-based Deep Neural Network to perform classification over 19 phases of 5 exercises, common to CrossFit routines. From this baseline, we aim to perform workout video segmentation, while creating a descriptor that enables repetition counting or feedback on the exercise execution. More specifically, a 2D Human Pose Network creates heatmap-based features that capture the human body pose, which are fed into the Transformer Encoder with 4 attention heads. A final Multilayer Perceptron with 3 Dense layers performs the phase classification task. To this end, we have trained our model using a previously acquired dataset that is naturally imbalanced, e.g. 6 classes have less than 8k samples and 9 classes have more than 24k. Finally, the obtained results show that we are able to divide videos in a temporally consistent manner, outperforming a state-of-the-art model that counts repetitive actions, specifically for 4 out of the 5 exercises.
Egnet: A Novel Edge Guided Network for Instance Segmentation

Edge information plays a significant role in instance segmentation. However, many instance segmentation methods directly perform pixel-wise classification via fully convolutional networks, which may ignore object edges. In this paper, we propose a novel Edge Guided Network (EGNet), which exploits edge information to improve the mask accuracy, for instance segmentation. Specifically, we propose an edge branch to extract edge information. Then, we use edge information as guidance and fuse it with mask features, in order to enrich the mask features. Furthermore, we propose a Spatial Attention (SA) module and add it to the backbone of our EGNet, enabling the network to focus more on foreground objects. In addition, we incorporate a Semantic Enhancement (SE) module into the edge branch, aiming to obtain additional global context information. Experimental results on the COCO 2017 dataset show the effectiveness of the proposed EGNet.