Reviewerly/exHarmony huggingface dataset:
Channel-Position Self-Attention with Query Refinement Skeleton Graph Neural Network in Human Pose Estimation

Human Pose Estimation (HPE) is a long-standing yet challenging task in computer vision. The nature of the problem requires comprehensive global contextual reasoning among joints in different locations. In this work, we explore how to incorporate two popular and effective concepts, self-attention and Graph Neural Network (GNN), to model long-range information in HPE. Three different ways to implement self-attention in 3D feature maps are studied, where the best result is achieved via the channel-position version. Accuracy is further improved by refining the queries via an efficient channel-wise parallel GNN that explicitly models the human joint graphical relationships. We are able to improve prediction accuracy on strong baseline models and achieve state-of-the-art results.
EANET: Efficient Attention-Augmented Network for Real-Time Semantic Segmentation

Real-time semantic segmentation plays a significant role in many real-world applications. However, existing methods usually neglect the importance of aggregating global scene clues and multi-level semantics due to computational limits of mobile devices. To address the above challenges and maintain higher accuracy, we propose an efficient attention-augmented network, namely EANet. Specifically, we first leverage an extremely lightweight attention module called sparse strip attention module (SSAM) to retain global contextual information while greatly reducing computation cost. Moreover, the meticulously designed joint attention fusion module (JAFM) follows an attention strategy to efficiently integrate semantics and details from multi-level features. On Cityscapes test set, our network achieves 74.6% mIoU at 35.4 FPS on a single GTX1080Ti GPU with a 1024×2048-pixel image. Extensive experiments show that our EANet achieves promising results on Cityscapes dataset.
Fully Shareable Scene Text Recognition Modeling for Horizontal and Vertical Writing

This paper proposes a simple and efficient method of joint scene text recognition for both horizontal and vertical writing. Recently, end-to-end scene text recognition using the Transformer-based autoregressive encoder-decoder model offers high recognition accuracy. Research into this method has mainly focused on horizontally written text, but in several Asian countries, texts are also written vertically. To efficiently train a recognition model for jointly recognizing horizontal and vertical writing, several methods have been proposed that partially share model components between each writing direction. However, this approach lowers training efficiency because non-shareable components are trained only on just horizontal or vertical writing data. To increase training efficiency, our key idea is to consider writing direction in the continuous space obtained by a fully shareable model for horizontal and vertical writing. To this end, our proposed method gives the writing direction as an initial token to the autoregressive decoder while sharing all components for each writing direction. Furthermore, to incorporate common features between each writing into the model, the proposed method predicts character count before predicting the character string. Experiments on Japanese scene text recognition demonstrate the effectiveness of the proposed method.
PGTNet: Prototype Guided Transfer Network for Few-Shot Anomaly Localization

Anomaly localization is pixel-level regions detection in the image. The challenge is how to generate accurate representations of the novel anomaly types which are multifarious. Besides, the anomaly sample size is often not enough to support model learning to detection because of the limitations of real conditions. In this work, we present a novel few-shot setting for anomaly detection and reorganize the defective datasets. Based on the few-shot learning, we transfer the idea of metric learning and propose the prototype-guided transfer network (PGTNet). Extensive experiment results suggest that PGT-Net outperforms current SOTA methods and provides a novel perspective for the anomaly localization task.
Barycentric Defense

Wasserstein metric based adversarial attacks have attracted a great interest in the recent past. Even though they exhibit strong attacks, surprisingly, they have not been investigated for defense. In this work, we demonstrate that barycenters computed in Wasserstein space can act as a measure of defense against adversarial attacks. We compute the barycenter using marginals obtained from the given image and demonstrate its effectiveness in defense even without any adversarial training. We further analyse the barycenters using GradCam to understand their defensive characteristics. Elaborate experiments on MNIST, Fashion-MNIST, CIFAR-10 and CIFAR-100 demonstrate a significant increase in the robustness of victim classifiers. Our code is available at https://github.com/Visual-Conception-Group/Barycentric-Defense.
RAPID: A Single Stage Pruning Framework

Conventional network pruning methods require multiple stages to identify and train a single compact pruned model. This approach has a high computational overhead and requires multiple iterations to train multiple pruned models increasing the total computational cost. In this work, we present a single-stage Random Pruning Online Distillation (RAPID) framework to identify and train multiple pruned models at once. We randomly prune the original network at scratch and then leverage distillation at the later training epochs to transfer information from the original network(teacher) to multiple pruned models(students) online. Extensive experiments demonstrate the effectiveness of the proposed RAPID framework on several datasets and architectures when com-pared with other pruning methods.
Multi-Scale Transformer-Based Feature Combination for Image Retrieval

Image retrieval consists in the selection of a set of images from a database, based on their visual similarity to a given query image. To compute this visual similarity, a global or multiple local features are required per image.In this work, we present a system that uses multiple high-level semantic feature maps from a single image for global feature generation. Our method uses feature maps extracted from a backbone architecture at different spatial resolutions, enhancing the semantic information relevant at each scale using a transformer-based approach. Then, these feature maps are combined using a self-attention mechanism generating a global feature.The main contribution of this work is a novel global feature generation method, which outperforms current state-of-the-art techniques that use either global-only or a combination of global and local features. This result is assessed using the Google Landmarks v2 resulting in an improvement of 3.8% in mAP@100. Also, we assess our system using Revisiting Oxford and Paris datasets, obtaining an improvement up to 3.7% in mAP compared with current methods.
Adding Non-Linear Context to Deep Networks

Enormous success has been achieved with deep neural networks consisting of standard linear-convolutions followed by simple non-linear mapping functions. In this paper, we add easily-computed non-linear local and global statistics to deep architectures, augmenting the information available at each layer. This additional information is then used in an identical manner to current processing. The summary statistics, which can be as simple as calculating within-channel variance, introduces little run-time computational overhead and can be instantiated with few extra parameters. All standard training procedures can be used without modification for training these augmented networks. We show, through extensive testing with ResNet on ImageNet, performance improvements across a wide range of network sizes. Additionally, we provide a detailed study of where within the deep networks these statistics are most effective.
Moving Object Detection in Noisy Video Sequences Using Deep Convolutional Disentangled Representations

Noise robustness is crucial when approaching a moving detection problem since image noise is easily mistaken for movement. In order to deal with the noise, deep denoising autoencoders are commonly proposed to be applied on image patches with an inherent disadvantage with respect to the segmentation resolution. In this work, a fully convolutional autoencoder-based moving detection model is proposed in order to deal with noise with no patch extraction required. Different autoencoder structures and training strategies are also tested to get insights into the best network design approach.
Background-Tolerant Object Classification With Embedded Segmentation Mask For Infrared And Color Imagery

Even though convolutional neural networks (CNNs) can classify objects in images very accurately, it is well known that the attention of the network may not always be on the semantically important regions of the scene. It has been observed that networks often learn background textures, which are not relevant to the object of interest. In turn this makes the networks susceptible to variations and changes in the background which may negatively affect their performance.We propose a new three-step training procedure called split training to reduce this bias in CNNs for object recognition using Infrared imagery and Color (RGB) data. Our split training procedure has three steps. First, a baseline model is trained to recognize objects in images without background, and the activations produced by the higher layers are observed. Next, a second network is trained using Mean Square Error (MSE) loss to produce the same activations, but in response to the objects embedded in background. This forces the second network to ignore the background while focusing on the object of interest. Finally, with layers producing the activations frozen, the rest of the second network is trained using cross-entropy loss to classify the objects in images with background. Our training method outperforms the traditional training procedure in both a simple CNN architecture, as well as for deep CNNs like VGG and DenseNet, and learns to mimic human vision which focuses more on shape and structure than background with higher accuracy.
Non-Separable Filtering with Side-Information and Contextually-Designed Filters for Next Generation Video Codecs

We propose two new in-loop filtering tools to augment the loop restoration process in AV1. Targeting challenging future compression scenarios that will be faced by the emerging AV2 standard, the proposed tools are designed to improve performance at both high and low bit-rates. Our non-separable Wiener filtering proposal aims to increase quality especially over directional features and textures in the decoded picture with the help of side-information at higher rates. The proposed pixel-adaptive filter complements at lower rates by improving performance without side-information relying solely on finely characterized pixel contexts. Simulation results show the rate-distortion efficacy of the proposed tools.
Mixed Membership Generative Adversarial Networks

GANs are designed to learn a single distribution, though multiple distributions can be modeled by treating them separately. However, this naive implementation does not consider overlapping distributions. We propose Mixed Membership Generative Adversarial Networks (MMGAN) analogous to mixed-membership models that model multiple distributions and discover their commonalities and particularities. Each data distribution is modeled as a mixture over a common set of generator distributions, and mixture weights are automatically learned from the data. Mixture weights can give insight into common and unique features of each data distribution. We evaluate our proposed MMGAN and show its effectiveness on MNIST and Fashion-MNIST with various settings.
Boosting Supervised Learning in Small Data Regimes with Conditional GAN Augmentation

In many applied computer vision tasks, training data is a scarce resource. This can result in poor performance of deep neural networks trained via supervised learning. We propose an approach to help boost performance in small training data regimes. Our method, called "CONGA", uses a Conditional GAN to Augment training data. Unlike previous work it can be added to any target discriminative model and allows the trade-off of computational cost for improved training sample efficiency. To further improve the quality of generated images and our method’s performance, and distinct from normal conditional GANs, we also propose to supervise the generator’s output via the target model. We compare our approach to similarly motivated methods on various image classification datasets (CIFAR-10, CIFAR-100, and Street View House Numbers), showing significant quantitative improvements.
Unsupervised Domain Adaptation Person Re-Identification by Camera-Aware Style Decoupling and Uncertainty Modeling

Unsupervised domain adaptation (UDA) person re-identification (re-ID) aims to transfer knowledge learned from labeled source domain to unlabeled target domain and has been successfully applied into a wide range of real-world scenarios. However, existing methods are mainly ineffective at handling domain shift as well as being sensitive to camera styles due to the unannotated target domain. In this paper, we pro-pose a Camera-style Separation and Uncertainty Estimation (CSUE) model to address the problem from two perspectives. To alleviate the negative effect of cross-camera variation, we introduce the Camera-aware Style Decoupling module to im-pose inter-and-intra camera constraints on the feature extracting stage. It can better mine and describe the latent camera invariant features. Moreover, to avoid the inherent defect of clustering, an Uncertainty Modeling module is constructed via estimating the certainty, which helps progressively refine the pseudo labels. Extensive experiments on widely used datasets demonstrate the state-of-the-art performance of our model under the UDA re-ID setting.
Component-Based Transformation for Person Image Generation

Person image generation has attracted more and more attention because it has many applications. This paper focuses on pose transfer and virtual try-on. We propose a flow-based method called attribute-decomposited spatial transformation network. It warps different body parts separately using segmentation and then fuses different body parts and generates images. The flow-based technique enables the model to generate high-quality person images for pose transfer and the attribute-decomposited technique enables the model to support virtual try-on. The proposed method was evaluated on Deepfashion dataset. Quantitative and qualitative experimental results show the superiority of the method.
Multi-View Feature Boosting Network for Deep Subspace Clustering

Subspace clustering is widely used to find clusters in different subspaces within a dataset. Autoencoders are popular deep subspace clustering methods using feature extraction and dimensional reduction. However, neural networks are vulnerable to overfitting, and therefore have limited potential for unsupervised subspace clustering. This paper proposes a deep multi-view subspace clustering network with feature boosting module to successfully extract meaningful features in different views and to fuse multi-view representations in a complementary manner for enhanced clustering results. The multi-view boosting provides the robust features for unsupervised clustering by emphasizing the features and removing the redundant noise. Quantitative and qualitative analysis on various benchmark datasets verifies that the proposed method outperforms state-of-the-art subspace clustering methods.
Recurrent Attentive Decomposition Network for Low-Light Image Enhancement

This paper aims to solve the problems of Low-light image enhancement based on classical method RetinexNet. Given the problems of original results with lots of noise and color distortion, this paper proposes a novel recurrent attentive decomposition network, which combines spatial attention mechanism and Encoder-Decoder structure to better capture the key information of images and make a thorough image decomposition process. Furthermore, another network based on attention mechanism is added to denoise the reflection image and improve the restoration effect of image details. Compared with RetinexNet and other popular methods, the overall style of images processed by our method is more consistent with that of the real scene. Both visual comparison and quantity comparison of Structural Similarity(SSIM) and Peak Signal to Noise Ratio(PSNR) demonstrate that our method is with superiority to several state-of-the-art methods.
Deeply Learned Structure-Aware Transmission for Image Haze Removal

Atmospheric haze degenerates image visibility or visual quality. Unfortunately, characterizing scene depth and scene radiance uncertainties to precisely recover the transmission still remains challenging. This paper proposes a new structure-aware transmission recovery strategy for single image dehazing. Specifically, such a new strategy is a coarse-to-refined dehazing approach that integrates multiscale convolutional neural networks and structure-aware retinex modeling into filtering-based fusion to accurately recover the transmission map. The proposed method was evaluated on benchmark data and compared with currently available dehazing approaches by quantifying colorfulness, contrast, and structural fidelity information of dehazed photographs. The experimental results demonstrate that the robust structure-preserving dehazing method outperforms other approaches, with significantly improving the average visual quality or content of the colorfulness, contrast, structural sharpness, and visibility from (0.63, 0.72, 0.15, 0.62) to (0.81, 0.84, 0.18, 0.78).
3D End-to-End Boundary-Aware Networks for Pancreas Segmentation

Accurate pancreas segmentation is crucial for computer aided pancreas diagnosis and surgery. It still remains challenging to precisely extract the pancreas due to its small size, unclear boundary, and shape variations on CT images. This work proposes a new 3D end-to-end boundary-aware network architecture for automatic accurate pancreas segmentation from CT images. Specifically, this architecture introduces four hybrid blocks for feature extraction in accordance with 3D fully convolutional neural networks so that it can successfully extract and perceive spatial and contextual information from 3D CT data. Simultaneously, a reverse attention block and a boundary enhancement block are embedded into this architecture to enhance the ability to learn and extract feature maps with more context and boundary information. We evaluate our proposed method on publicly available pancreas data using 4-fold cross-validation, with the experimental results showing that our network model can obtain more accurate or comparable segmentation than other existing methods.
Weighted Supervised Contrastive Learning and Domain Mixture for Generalized Person Re-Identification

Domain generalization(DG) in person re-identification (ReID) attracts increasing attention due to its practical applications. It aims to learn a model that, after training on multiple source domains, can be applied directly to the unseen domains without further training. In order to develop a domain-robust model for unseen domains, we propose the Memory-based Anti-Hard-instances Domain-Mix(MAD) framework. Specifically, a memory-based non-parametric contrastive loss is adopted to replace the traditional parametric cross-entropy loss. To prevent overfitting on the source domains, we present an Anti-Hard-instances module to mitigate the effect of hard instances. We also introduce a Domain-Mix module to diversify the features in the source domains, further enhancing the generalization ability of our model. Extensive experiments on four large-scale ReID datasets fully demonstrate the strong generalization and competitiveness of our framework.
Real-World Image Super-Resolution Via Kernel Augmentation And Stochastic Variation

Deep learning (DL) based single image super-resolution (SISR) algorithms have now achieved highly satisfactory evaluation and visualization results on synthetic datasets. However, in some practical applications, especially when restoring some real-world low-resolution (LR) photos, the limitation and unicity of the most commonly used bicubic down-sampling kernel often lead to significant performance degradation of models trained under ideal conditions. Thus, we first propose a kernel augmentation (KA) strategy based on generative adversarial networks (GANs) to improve the generalization ability and robustness of current SISR models. Then, we intend to reconstruct the stochastic variation (SV) features that are widely present in natural images to obtain a more realistic feature representation. In the end, extensive experiments demonstrate the feasibility and effectiveness of our approach in dealing with real-world SISR problems.
NBD-GAP: Non-Blind Image Deblurring without Clean Target Images

In recent years, deep neural network-based restoration methods have achieved state-of-the-art results in various image deblurring tasks. However, one major drawback of deep learning-based deblurring networks is that large amounts of blurry-clean image pairs are required for training to achieve good performance. Moreover, deep networks often fail to perform well when the blurry images and the blur kernels during testing are very different from the ones used during training. This happens mainly because of the overfitting of the network parameters on the training data. In this work, we present a method that addresses these issues. We view the non-blind image deblurring problem as a denoising problem. To do so, we perform Wiener filtering on a pair of blurry images with the corresponding blur kernels. This results in a pair of images with colored noise. Hence, the deblurring problem is translated into a denoising problem. We then solve the denoising problem without using explicit clean target images. Extensive experiments are conducted to show that our method achieves results that are on par to the state-of-the-art non-blind deblurring works.
Improved DC Estimation for JPEG Compression Via Convex Relaxation

Mass image transmission has undergone an explosion of growth with the development of the internet, DCT-based lossy image compression like JPEG is pervasively conducted to save the transmission bandwidth. Recently, DCT-domain coefficient estimation approaches have been proposed to further improve the compression ratio by discarding DC coefficients at the sender’s end while recovering them at the receiver’s end via DC estimation. However, known DC estimation needs to enumerate all possible DC coefficients. Consequently, they are limited and resource-consuming due to the low delay requirements in real-time transmission. In this paper, we propose an improved DC estimation method via convex relaxation, which achieves state-of-the-art performance in terms of both recovery image quality and time complexity. Extensive experiments across various data sets demonstrate the advantages of our method.