Reviewerly/exHarmony huggingface dataset:
Towards Efficient Variational Auto-Encoder Using Wasserstein Distance

VAE, or variational auto-encoder, compresses data into latent attributes and generates new data of different varieties. VAE with KL divergence loss has been considered an effective technique for data augmentation. In this paper, we propose using Wasserstein distance as a measure of distributional similarity for the latent attributes and show its superior theoretical lower bound (ELBO <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">W</inf> ) compared with that of KL divergence (ELBO <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">KL</inf> ) under mild conditions. Using multiple experiments, we demonstrate that the new loss function converges faster and generates better quality data to aid image classification tasks. We also propose implementing a dynamically changing hyper-parameter tuning schedule to avoid the potential overfitting of ELBO <inf xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">W</inf> .
Distilling DETR-Like Detectors with Instance-Aware Feature

DEtection TRansformer (DETR) has achieved great success in object detection but suffers from slow convergence during training process. Knowledge distillation (KD) can speed up model training but has the problem of locating knowledgeable regions in object detection. Related methods mainly locate knowledgeable regions empirically. To address those challenges, we propose a novel distillation framework for DETR-like transformer-based detectors. The key idea is to connect each instance with its corresponding response region in the feature map through cross attention. To better fuse the attention maps between different queries and heads, we introduce an attention fusion module to balance instances of different scales. Extensive experiments on DETR and Conditional DETR are conducted to verify our proposed method. Our method improves the mAP by 3.19% for Conditional DETR with ResNet-50 backbone trained for 50 epochs, which outperforms the strong teacher trained for 108 epochs. We also boost DETR with ResNet-50 backbone from 33.97% to 42.13% mAP (+8.16%) under 50 epochs.
An Empirical Approach for Optimising the Impact of a Preprocessor in a Transcoding Pipeline

The volume of User Generated Content (UGC) on the internet has exploded throughout the pandemic. The relatively low quality of that content generally implies an increased bitrate and much reduced quality after transcoding. Preprocessing e.g. using a noise reducer, is one approach for reducing bitrate and increasing quality. The impact of the noise reducer is however affected by the target bitrate of the encoder. That relationship is known but not previously quantitatively examined. In this paper we present a methodology and new metric for measuring this impact based on the Rate-Distortion curves before and after pre-processing. The metric is used as a cost function for estimating the optimal filter parameter for our chosen denoiser. Our experiments show that optimising the filter parameter in this way yields as much as 4-5dB improvement in PSNR at 3 Mbps.
Relation Enhanced Vision Language Pre-Training

In this paper, we propose a relation enhanced vision-language pre-training (VLP) method for a transformer model (TM) to improve performance in vision-language (V+L) tasks. Current VLP studies attempted to generate a multimodal representation with individual objects as input and relied on a self-attention to learn semantic representation in a brute force manner. However, the relations among objects in an image are largely ignored. To address the problem, we generate a paired visual feature (PVF) that is organized to express the relations between objects. Prior knowledge that reflects co-occurrences of paired objects and a pair-wise distance matrix adjusts the relations, and a triplet is used for sentence embedding. Experimental results demonstrate that the proposed method is efficiently used for VLP by bridging relations between objects, and thus improves performance on V+L downstream tasks.
Utilizing Excess Resources in Training Neural Networks

In this work, we suggest Kernel Filtering Linear Overparameterization (KFLO), where a linear cascade of filtering layers is used during training to improve network performance in test time. We implement this cascade in a kernel filtering fashion, which prevents the trained architecture from becoming unnecessarily deeper. This also allows using our approach with almost any network architecture and let combining the filtering layers into a single layer in test time. Thus, our approach does not add computational complexity during inference. We demonstrate the advantage of KFLO on various network models and datasets in supervised learning.
F2RNET: A Full-Resolution Representation Network for Biomedical Image Segmentation

In this paper, we are interested in exploring the problem of full-resolution image segmentation, with the focus placed on learning full-resolution representations for biomedicine images. We divide the original resolution image into patches of different sizes in different stages and then extracte local features from large to small patches using efficient and flexible components in modern convolutional neural networks (CNN). Meanwhile, a multilayer perceptron (MLP) block intended for modeling long-range dependencies between patches is designed to compensate for the inherent inductive bias caused by convolution operations. In addition, we perform multi-scale fusion and receive representation information from parallel paths at each stage, resulting in a rich full-resolution representation. We evaluate the proposed method on different biomedical image segmentation tasks and it achieves a competitive performance compared to the latest deep learning segmentation methods. It is hoped that this method will serve as a useful alternative to biomedical image segmentation and provide an improved idea for the research based on full-resolution representation.
Le-BEiT: A Local-Enhanced Self-Supervised Transformer for Semantic Segmentation of High Resolution Remote Sensing Images

Semantic segmentation for remote sensing images (RSI) has been a thriving research topic for a long time. Existing supervised learning methods usually require a huge amount of labeled data. Meanwhile, large size, variation in object scales, and intricate details in RSI make it essential to capture both long-range context and local information. To address these problems, we propose Le-BEiT, a self-supervised Transformer with an improved positional encoding Local-Enhanced Positional Encoding (LePE). Self-supervised learning relieves the demanding requirement of a large amount of labeled data. The self-attention mechanism in Transformer has remarkable capability in capturing long-range context. Meanwhile, we use LePE as a substitution for Relative Positional Encoding (RPE) to represent local information more effectively. Moreover, considering the domain difference between natural images and RSI, instead of ImageNet-22K, we pre-train Le-BEiT on a very small high-resolution RSI dataset—GID. To investigate the influence of pre-training dataset size on segmentation accuracy, we furtherly conduct experiments on a larger pre-training dataset called GID-DOTA, which is 1/100 of ImageNet-22K, and have observed considerable accuracy improvements. The result of our method, which relies on a much smaller pretrained dataset, achieves competitive accuracy compared to the counterpart on ImageNet-22K.
Accurate Head Pose Estimation Based on Multi-Stage Regression

This paper proposes a method for head pose estimation from a single image. We employ a multi-stage regression strategy. To overcome the discontinuity of Euler angles and quaternions and avoid the additional constraints required to directly regress the rotation matrix, we apply a continuous 6D representation to the head pose estimation problem. Each stage of the network regresses two 1 × 3 vectors, which are then transformed into a 3 × 3 rotation matrix by this continuous 6D representation. To better perceive the difference in rotation angles, we adopt the Riemann distance to measure the closeness between the network-estimated rotation matrix and the ground truth rotation matrix corresponding to the head pose. Experiments show that our method achieves the state-of-the-art on BIWI dataset and performs favorably on AFLW2000 dataset.
Multi-label Aerial Image Classification Based on Image-Specific Concept Graphs

Multi-label aerial image classification (MAIC) is a fundamental but challenging task for computer vision-based remote sensing applications. Existing MAIC models suffer from the insufficient semantic information of image and label representations. To this end, we integrate commonsense knowledge into the MAIC task and propose a novel Knowledge-augmented Concept Graph Learning (KCGL) framework. KCGL first collects relevant semantic concepts for each label from a commonsense knowledge graph ConceptNet. With the guidance of semantic concepts, an image decoupling module is employed to extract concept-specific image features from the input image. Then, KCGL constructs an individual concept graph for each image, in which nodes are corresponding to concept-specific image features and edges are their relations extracted from ConceptNet. Finally, the classification probability on each label is computed in the specific concept graph via a GCN-based encoder-decoder model. Experimental results prove that the proposed KCGL outperforms existing state-of-the-art MAIC models on two aerial image datasets.
Adaptive Multi-Scale Progressive Probability Model for Lossless Image Compression

Domain adaptation is an efficient technique to improve the performance of a system by adapting a pre-trained model to the given input data. The adaptation technique has been generally applied in conventional video codecs. For neural network-based systems, the encoder may adapt the decoder to the input data by fine-tuning a pre-trained model present at the decoder side. The weight update is then transferred to the decoder and the updated model is used to decode the bitstream. However, due to the large number of parameters in deep neural networks, the overhead of the weight update may diminish the gain from the adaptation technique. In recent years, various methods have been proposed to reduce the overhead without significantly compromising the gain. In this paper, we propose an adaptive multi-scale progressive probability model for lossless image compression. The proposed method uses the data that has already been processed at the inference stage to fine-tune the probability model. Importantly, the decoder can apply the fine-tuning by itself resulting a small adaptation overhead to help the decoder in performing the fine-tuning. The proposed method achieves up to 0.28 bits-per-pixel (BPP) reduction on four benchmark datasets compared to the state-of-the-art method.
Hardware-Oriented Shallow Joint Demosaicing and Denoising

Demosaicing and denoising are two important procedures in image signal processing. Although they can be performed separately, the better approach is to perform demosaicing and denoising jointly. From traditional methods to deep learning methods, many joint demosaicing and denoising algorithms are available, but few of them can achieve good image quality while maintaining a low cost. In this paper, we propose a new high-quality, low-cost, and hardware-oriented joint demosaicing and denoising algorithm. First, we introduce an efficient interpolation-based demosaicing algorithm. Then, we develop a new edge-preserving wavelet denoising algorithm with a novel voting-based direction estimation method. Finally, we present our innovative joint demosaicing and denoising algorithm. Our method conducts demosaicing and denoising alternatively, sharing the same direction information. The obtained experimental results show that the new algorithm can achieve better performance with less than 1/40 of the processing time required by other traditional algorithms.
Exploiting Doubly Adversarial Examples for Improving Adversarial Robustness

Deep neural networks have shown outstanding performance in various areas, but adversarial examples can easily fool them. Although strong adversarial attacks have defeated diverse adversarial defense methods, adversarial training, which augments training data with adversarial examples, remains an effective defense strategy. To further improve adversarial robustness, this paper exploits adversarial examples of adversarial examples. We observe that these doubly adversarial examples tend to return to the original prediction on the clean images but sometimes drift toward other classes. From this finding, we propose a regularization loss that prevents these drifts, which mitigates the vulnerability against multi-targeted attacks. Experimental results on the CIFAR-10 and CIFAR-100 datasets empirically show that the proposed loss improves adversarial robustness.
Font Watermarking Network for Text Images

With the popularization of online services, a lot of text watermarking algorithm have been proposed to protect the digital documents. However, most of them require extra manual design or text samantic modification. This paper proposes an end-to-end font watermarking network which is capable of automatic watermark embedding and extraction without changing the text content. In our scheme, a watermark is generated by changing the font attributes slightly and then embedded as a tiny perturbation. And we extract the watermark by detecting the attribute values of the font image. Experimental results highlight the superiority of the proposed watermarking scheme in terms of imperceptibility and speed comparing to the existing work.
Panoptic-Deeplab-DVA: Improving Panoptic Deeplab with Dual Value Attention and Instance Boundary Aware Regression

Panoptic DeepLab is a state-of-the-art framework that has showed good tradeoff between performance and complexity. In this paper, we focus on improving it to increase wide deployment of panoptic segmentation on mobile devices with low complexity. Specifically, we first present a novel Dual Value Attention (DVA) module to enable context information exchange between the semantic segmentation branch and the instance segmentation branch. Second, we further propose a new instance Boundary Aware Regression (iBAR) loss that assigns more emphasis on the instance boundary during instance regression. To assess the effectiveness of our proposed approach, we evaluate the performance on MSCOCO dataset for panoptic segmentation task, to show that our approach can improve upon the state-of-the-art Panoptic DeepLab with both the light-weight backbone network MobileNetV3 and the heavy-weight backbone network HRNetV2.
Improving IQA Performance Based on Deep Mutual Learning

In this paper, we propose a novel solution, termed DML-IQA, for the image quality assessment (IQA) tasks. DML-IQA holds a dual-branch network architecture and builds the IQA model through a deep mutual learning (DML) strategy. Specifically, the two branches extract stable feature representations by feeding different transformed images into the classical CNNs. The DML strategy first calculates the prediction loss of each branch and the consistency loss across two branches, followed by updating the network iteratively to converge. Overall, DML-IQA has the following advantages: 1) It is flexible to adapt to diverse backbones for tackling the IQA issues in both the laboratory and wild; 2) It improves the baseline’s performance by approximately 1%~2%, especially performs well in the case of small samples. Extensive experiments on four public datasets show that the proposed DML-IQA can handle the IQA tasks with considerable effectiveness and generalization.
DARTS-PD: Differentiable Architecture Search with Path-Wise Weight Sharing Derivation

With the advent of Neural Architecture Search (NAS), auto-designing of artificial neural networks has been made possible. Among various NAS methods, Differentiable Architecture Search (DARTS) has achieved significant progress due to its high calculating efficiency. However, it suffers from poor stability and obvious performance drop because of bi-level optimization and hard pruning. Besides, it only generates one best architecture at once. To alleviate the problems above, we design a three-stage framework with a path-wise weight sharing derivation. We first prune the supernet with differentiable methods to keep top-k operations on each edge instead of one. Then the pruned supernet is trained with our path-wise weight sharing method. At the derivation stage, the best candidate operations are selected with Evolutionary Search based on the validation accuracy of paths. Our weight sharing derivation is proved effective in improving searching stability as well as alleviating the performance drop. Furthermore, it also allows us to search for a large number of architectures with different parameter sizes at one time. Comprehensive experiments on CIFAR-10 and ImageNet show that we manage to find a group of state-of-the-art architectures (97.61% on CIFAR-10 and 76.4% on ImageNet).
Low SNR Multiframe Registration for Cubesats

We present a registration algorithm which jointly estimates motion and the ground truth image from a set of noisy frames under rigid, constant translation. The algorithm is non-iterative and needs no hyperparameter tuning. It requires a fixed number of FFT, multiplication, and downsampling operations for a given input size, enabling fast implementation on embedded platforms like cubesats where on-board image fusion can greatly save on limited downlink bandwidth. The algorithm is optimal in the maximum likelihood sense for additive white Gaussian noise and non-stationary Gaussian approximations of Poisson noise. Accurate registration is achieved for very low SNR, even when visible features are below the noise floor.
Deep Feature Compression using Rate-Distortion Optimization Guided Autoencoder

Collaborative intelligence (CI) has been proposed to efficiently utilize computational resources on edge devices over edge-cloud environments. To realize this, a deep neural network (DNN) model is divided into two parts, which are deployed to the edge device and cloud separately so that they can perform DNN inference collaboratively. Feature compression methods have been investigated to reduce the traffic volume between the edge and cloud because the data size of a feature map, which is the output of the edge device in CI, is relatively larger than that of the compressed original image. Although feature compression methods can achieve a good compression rate, the traffic volume is not small enough because the data size of a feature map might exceed that of the original image.Herein, we propose a novel feature compression method that utilizes an autoencoder (AE) to compress the feature map. In image compression, distortion and data size are minimized, whereas in our method, the AE is trained to minimize the prediction error and data size. In this way, the AE can extract only the data needed for the task of the DNN model and considerably reduce the data size while maintaining the prediction accuracy. In performance evaluation, we apply the proposed method to widely used convolutional neural network models: VGG16, ResNet50, and YOLOv3. Our results confirm that the proposed method drastically reduces the traffic volume while utilizing the resources of edge devices in CI. Our proposed method reduces the traffic volume by 62–99%, 16–99%, and 35–53% compared to versatile video coding in each model.
Temporal Axial Attention For Lidar-Based 3d Object Detection In Autonomous Driving

3D object detection is a core problem of the perception systems of autonomous vehicles. Despite recent progress in the field, the temporal aspect of LiDAR data has not been fully explored in current state-of-the-art detectors. This work proposes a modified CenterPoint architecture that uses temporal axial attention to exploit the sequential nature of autonomous driving data for 3D object detection. The last ten LiDAR sweeps are split into three groups of frames, and the axial attention transformer block captures both spatial and temporal dependencies among the features extracted from each group. Our proposal is evaluated using the nuScenes dataset. With this novel approach, we obtain an average mAP improvement of 3.8 and 2.3 points over the original CenterPoint in the fine/coarse pillar settings, respectively.
SRK-Net: Learning to Detect Repeatable Keypoints with Local Saliency Knowledge

The dominant approach for learning keypoint detectors relies on the covariance constraint. However, existing learned detectors sometimes extract unstable keypoints from edges. To solve this problem, we propose a novel method that exploits local saliency knowledge to train a keypoint detector, and obtain a keypoint detector, called as SRK-Net, which can extract stable and repeatable keypoints. Firstly, given an image, we propose a General Local Saliency Measure method (GLSM) to assess the local saliency value for each pixel and generate a local saliency map for this image. Then we propose a Local Salient Structure Maintaining loss (LSSM) and a two-stage progressive training manner tailored for leveraging the supervision of the covariance constraint and the local saliency maps provided by our GLSM. Experimental results show that the proposed SRK-Net performs better than all the existing keypoint detectors on HPatches dataset.
Parallel Partitioning: Path Reducing and Union–Find Based Watershed for the GPU

The watershed transform is a common step in different image processing tasks. With the fast development of general-purpose computing on GPUs, a number of parallel watershed algorithms have been introduced for that setup. We propose two novel parallel watershed algorithms: one intended for relatively larger images and another for smaller images. Our algorithms are based on the combination of the parallel path reduction technique, introduced in our previous paper on the topic, and the parallel version of the Union–Find algorithm. We show through multiple experiments, both in 2D and 3D, that our algorithms achieve unmatched execution times. On a typical gaming GPU, our algorithm processes an 800 megavoxel image in around 2.5 seconds.
Occlusion-Invariant Representation Alignment for Entity Re-Identification

Entity re-identification is the foundation of tracking- and matching-based computer vision tasks, which are widely employed in a variety of applications. However, when trained exclusively on clear images, the models capacity to generalize is significantly affected by the presence of occlusion at referencing time, whereas data argumentation-based approaches are costly to construct without guaranteeing a test-time improvement. To tackle this problem, we propose a domain adaptation framework based on learning representations that generates occlusion-invariant feature representations by aligning the clean image embedding distribution with the occluded one, using a disparity discrepancy metric derived from the siamese network architecture. Without the need for additional processing modules during the inference stage or an expensive occlusion-augmentation-enlarged dataset during the training stage, we could obtain occlusion invariant embeddings that are free of the impact of occluders. Extensive experimental results for two tasks across three datasets indicate the proposed method’s robustness and effectiveness to a variety of occlusions at all levels.