Reviewerly/exHarmony huggingface dataset:
CSTNET: Enhancing Global-To-Local Interactions for Image Captioning

Image captioning aims to generate descriptions of images, which requires capturing complex interactions between local regions and global context within image. However, effective global context modeling from image remains a challenging research topic. Existing approaches incorporate global-level information into the initialized input mainly based on transformer architecture. Unlike previous methods that may not be able to capture rich global contextual information, we propose a novel method named Context-Sensitive Transformer (CSTNet), which can discover the inherent global context and further empower the global-to-local interactions. Experimental results on the MSCOCO dataset show that the proposed model can significantly improve the performance of image captioning.
Breakpoint Dependent Scalable Coding of Optical Flow Volume

Motion representations that describe motion flow from one single anchor frame to multiple reference frames can be useful for many tasks such as motion compensation (MC) and frame-rate up-sampling. We construct an anchored, multi-flow representation which we refer to as an Optical Flow Volume (OFV). We explore the use of recent breakpoint dependent DWT (BD-DWT) being considered as part of extensions to JPEG 2000 for coding discontinuous media. Breakpoints describe discontinuity boundary geometry and we estimate a single set of breakpoints that can be shared by all individual flows of the OFV. Additionally, as flows are anchored at a common frame, we are able to readily explore inter-flow transforms. Rate scalable results show significant rate-distortion gains for BD-DWT over the 5/3 DWT commonly used with JPEG 2000. The validity and utility of OFV coding are confirmed with accompanying MC results.
Improving Generalization of Reinforcement Learning Using a Bilinear Policy Network

In deep reinforcement learning (DRL), the agent is usually trained on seen environments by optimizing a policy network. However, it is difficult to be generalized to unseen environments properly, even when the environmental variations are insignificant. This is partly because the policy network cannot effectively learn the representation of visual difference that is subtle among highly similar states in the environments. Because a bilinear structured model containing two feature extractors allows pairwise feature interactions in a translation-ally invariant manner which makes it particularly useful for subtle difference recognition among highly similar states, in this work, a bilinear policy network is employed to enhance representation learning, and thus to improve generalization of the DRL. The proposed bilinear policy network is tested on various DRL task, including a control task on path planning for active object detection, and Grid World, an AI game task. The test results show that the generalization of DRL can be improved by the proposed network.
A Structure From Motion Pipeline for Orthographic Multi-View Images

Structure from Motion (SfM) plays a crucial role in unstructured capturing. While images are usually taken by perspective cameras, orthographic camera projections do not suffer from the foreshortening effect, that leads to varying capturing quality in image regions. Most contributions to orthographic image SfM assume a perspective setup with nearly infinite focal length. These assumptions lead to potentially sub-optimal camera pose estimation. Therefore, we propose a SfM pipeline that is optimized for orthographically projected images. For this, we estimate initial camera poses using the factorization method by Tomasi and Kanade. These poses are further refined by a specialized bundle adjustment implementation for orthographic projections. The proposed pipeline surpasses the precision of state-of-the-art work by an order of magnitude, while consuming considerably less computational resources.
Variational Depth Estimation on Hypersphere for Panorama

Depth estimation for panorama is a key part of 3D scene understanding, and adopting discriminative models is the most common solution. However, due to the rectangular convolution kernel, these existing learning methods cannot efficiently extract the distorted features in panoramas. To this end, we propose OmniVAE, a generative model based on Conditional Variational Auto-Encoder (CVAE) and von Mises-Fisher (vMF) distribution, to strengthen the exclusive generative ability for spherical signals by mapping panoramas to hypersphere space. Further, to alleviate the side effects of manifold-mismatching caused by non-planar distribution, we put forward the Atypical Receptive Field (ARF) module to slightly shift the receptive field of the network and even take the distribution difference into account in the reconstruction loss. The quantitative and qualitative evaluations are performed on real-world and synthetic datasets, and the results show that OmniVAE outperforms the state-of-the-art methods.
Unrolling Graph Total Variation for Light Field Image Denoising

A light field (LF) image is composed of multiple sub-aperture images (SAIs) from slightly offset viewpoints. To denoise a noise-corrupted LF image, leveraging recent development in deep algorithm unfolding, we pursue a hybrid graph-model-based / data-driven approach. Specifically, we first connect each pixel in a target patch of an SAI to neighboring pixels within the patch, and to pixels in co-located "similar" patches in adjacent SAIs. Given graph connectivity, we formulate a maximum a posteriori (MAP) problem using graph total variation (GTV) as signal prior. We then unroll the iterations of a corresponding optimization algorithm into a sequence of neural layers. In each unrolled layer, we learn relevant features per pixel from data using a convolutional neural net (CNN) in a supervised manner, so that edge weights can be computed as functions of feature distances. Each neural layer can be interpreted as a graph low-pass filter for a 4D LF image patch. Experiments show that our proposal outperformed two model-based and two deep-learning-based implementations in numerical and visual comparisons.
Learning from Noisy Labels via Meta Credible Label Elicitation

Deep neural networks (DNNS) can easily overfit to noisy data, which leads to a significant degradation of performance. Previous efforts are primarily made by label correction or sample selection to alleviate supervision problem. To distinguish between noisy labels and clean labels, we propose a meta-learning framework which could gradually elicit credible labels via the meta-gradient descent step under the guidance of potentially non-noisy samples. Specifically, by exploiting the topological information of feature space, we can automatically estimate label confidence with a meta-learner. An iterative procedure is designed to select the most trustworthy noisy-labeled instances to generate pseudo labels. Then we train DNNs with pseudo supervision and original noisy super vision, which learns sufficiency and robustness properties in a joint learning objective. Experimental results on benchmark classification datasets show the superiority of our approach against the state-of-the-art methods.
The BRIO-TA Dataset: Understanding Anomalous Assembly Process in Manufacturing

In this paper, we introduce a new video dataset for action segmentation, the BRIO-TA (BRIO Toy Assembly) dataset, which is designed to simulate operations in factory assembly. In contrast with existing datasets, BRIO-TA consists of two types of scenarios: normal work processes and anomalous work processes. Anomalies are further categorized into incorrect processes, omissions, and abnormal durations. The subjects in the videos are asked to perform either normal work or one of the three anomalies, and all video frames are manually annotated into 23 action classes. In addition, we propose a new metric called anomaly section accuracy (ASA) for evaluating the detection accuracy of anomalous segments in a video. With the new dataset and metric, we report that the state-of-the-art methods show a significantly low ASA, while they work for normal work segments. Demo videos are available at https://github.com/Tarmo-moriwaki/BRIO-TA_sample and the full dataset will be released after publication.
Toward Snow Removal Via the Diversity and Complexity of Snow Image

Existing snow removal approaches ignore specific characteristics of the snowflake itself, leading to insufficient snowflake feature representation and further incomplete snow removal. We propose a novel snow removal algorithm considering the diversity and complexity of snow, named as DCSNet. For diversity, we construct an adaptive fusion feature pyramid structure, characterizing the four elements of snowflakes(i.e., shape, size, transparency and direction), guiding the following snow removal process with more accurate direction location and salient shape features. We further capture and fuse cross-scale interaction information from the four elements of represented snowflakes more comprehensively. For complexity, we design a progressive recovery module to decompose snowflakes layers stage by stage, allowing the previous feature maps to interact with degraded images for achieving clearer snowflake removal. Extensive experimental results show that DCSNet outperforms the state-of-the-art desnowing algorithms by 3 dB increase in PSNR on three representative datasets. The source code is available at https://github.com/Xjg-0216/DCSNet.
Learnable Pixel Clustering Via Structure and Semantic Dual Constraints for Unsupervised Image Segmentation

Unsupervised image segmentation is a challenge task, since a high-quality segmented image should perceive not only local object structures but also certain semantics without any annotations. In this paper, we propose a novel encoder-decoder pixel clustering framework with dual constraints to incorporate local structure and global semantic information for guiding pixel feature learning in a self-supervised manner. On one hand, a Local Structure Constraint (LStC) is constructed based on fine-grained superpixels, which improves the boundary perception of pixel features by keeping intra-superpixel feature consistency and largening inter-superpixel feature distance. On the other hand, a new Global Semantic Constraint (GSeC) is proposed via adapting the mutual information maximization technique to the single-image setting, and it strengthens the global semantic perception of pixel features and thus improves the segmenting integrity of objects. Finally, based on the learned pixel features, a smoothing component is employed to achieve semantically meaningful pixel clustering. The experimental evaluation on BSDS500 and PASCAL Context datasets show the superiority of our method on region and boundary qualities.
CRAB: Certified Patch Robustness Against Poisoning-Based Backdoor Attacks

Backdoor attacks have been proved to be seriously threatening to deep neural networks. Many defending methods against backdoor attack have been proposed and reduced attack success rate significantly. However, most existing defending methods are empirical, and might be later broken by stronger attack methods. To avoid such a cat-and-mouse game, We proposed CRAB, a defense that can guarantee the robustness of an image classifier against poisoning-based backdoor attack with triggers bounded in a contiguous region. We analyze two ways of adding triggers: fixed-region and randomized-region. For fixed-region setting, we train a set of models on benign dataset for different image ablation positions and give robustness guarantee to both training and testing datasets. Whilst for random position triggers, we train a universal model on dataset with triggers, and give robustness guarantee to testing datasets. Our excellent experimental results demonstrate that CRAB exhibits strong robustness against patched backdoor attack, while maintaining comparable high clean accuracies.
Progressive Training Enabled Fine-Grained Recognition

Organizing training samples in a meaningful order is beneficial for accelerating the convergence rate and enhancing the recognition performance in the CNN model. However, achieving reasonable sample ranking for fine-grained recognition datasets is very challenging because the intra and inter class relation in those datasets is opposite to that in public recognition datasets. In this paper, we propose a general framework for the progressive training of fine-grained recognition models. In particular, we first formulate the training subset selection as a group ranking-oriented submodular optimization problem, where the submodularity is adopted to evaluate the benefit of selected training subsets. This can give theoretical guidance for the consecutive discrimination of difficult and ordinary training subsets. Secondly, we design a training strategy to dynamically adjust the ratio of difficult and ordinary training subsets according to the recognition performance. Extensive experiments on CUB-200-2011 and Stanford Dogs datasets demonstrate that the proposed method outperforms the state-of-the-art curriculum learning methods.
MDNet: Motion Distinction Network for Effective Action Recognition

Motion information is critical for action recognition. Most existing methods perform motion enhancement only from the channel or spatiotemporal dimension. They may fail to achieve fine-grained motion modeling, leading to suboptimal performance. In this paper, we propose a novel motion distinction network (MDNet) to address this challenging problem. Specifically, we first propose a channel-wise motion enhancement (CME) module, which aims to emphasize motion-related channels by leveraging a channel-wise gating mechanism. Then, we propose a cascaded spatiotemporal enhancement (CSTE) module to enhance motion features along the spatiotemporal dimension. Moreover, we design a multi-attention fusion strategy to further refine the enhanced motion features in a moderate manner, enabling the network to focus on discriminative motion regions. The proposed two modules and fusion strategy are complementary for fine-grained motion enhancement. Extensive experiments are conducted on the challenging Something-Something V1 and Kinetics-400 datasets to show the effectiveness of the proposed MDNet.
Batch Size Reconstruction–Distribution Trade–Off In Kernel Based Generative Autoencoders

Most autoencoder-based generative machine learning models use a two-factor cost function composed of reconstruction error and prior distribution distance. The latter is often evaluated with kernel–based methods. We notice that the impact of the batch size is different on each of the factors: kernel distribution profits from larger batches, while the reconstruction term achieves peak performance on smaller ones. Thus, we define a batch size reconstruction-distribution trade-off. Instead of searching for an optimum global size of the batch, we propose to use small batches for the sake of reconstruction together with a vector enhanced with previously computed latent data for the sake of prior distribution optimization. We evaluate our method on standard benchmarks and illustrate that it can improve the model’s generative scores.
Revisiting Spatial Inductive Bias with MLP-Like Model

In recent years, deep convolution neural nets have produced outstanding results in vision tasks compared to previous methods. In the Transformer models with self-attention structure, the introduction of inductive bias of convolution operator has been studied. There is no doubt that the hard inductive bias of the convolution operator, i.e., the receptive field (locality) and spatial invariance (weight sharing) in the spatial dimension, is an essential factor in achieving high performance. We hypothesize that by placing additional constraints on the locality, we can obtain better inductive bias. In this paper, we propose an MLP-like model, content-aware token mixing MLP (CaMLP), in which locality and spatial invariance are tunable. We verify that smooth and outward-decaying locality plays an important role in token mixing of the model with relaxed spatial invariance in the CIFAR-10/ImageNet100 classification task.
Multi-Stage Feature Alignment Network for Video Super-Resolution

Video super-resolution aims at generating high-resolution video frames using multiple adjacent low-resolution frames. An important aspect of video super-resolution is the alignment of neighboring frames to the reference frame. Previous methods directly align the frames either using optical flow or deformable convolution. However, directly estimating the motion from low-resolution inputs is hard since they often contain blur and noise that hinder the image quality. To address this problem, we propose to conduct feature alignment across multiple stages to more accurately align the frames. Furthermore, to fuse the aligned features, we introduce a novel Attentional Feature Fusion Block that applies a spatial attention mechanism to avoid areas with occlusion or misalignment. Experimental results show that the proposed method achieves competitive performance to other state-of-the-art super-resolution methods while reducing the network parameters.
Natural Image Matting with Shifted Window Self-Attention

Natural image matting is a challenging and significant task in computer vision. Recently, image matting achieves fantastic development by introducing deep learning methods. To the best of our knowledge, there is no image matting method using the Transformer. Compared with CNNs, the Transformer pays more attention to the interest points and the relationships of content, which is beneficial to the image matting task. In this paper, we first present a novel Transformer-based image matting method with Shifted Window self-Attention. Specifically, our method contains two encoders, an alpha encoder and a context encoder. The former leverages the Transformer with Shifted Window self-Attention to extract features of details, such as hairs, feathers and porous parts of foreground objects. Shifted Window self-Attention focuses on patches with the size of the window and connections of adjacent patches. With this, the Transformer is capable of dealing with high-resolution images. The context encoder, which takes rescaled images as input, aims to extract the whole structure information of foreground objects. Then, we propose a novel Hierarchical Pyramid Pooling Module (HPPM) which enables the network to have the flexibility to extract features at various resolutions. Experiments show that our method achieves competitive performance on the Composition-1K dataset.
Humans Disagree With the IoU for Measuring Object Detector Localization Error

The localization quality of automatic object detectors is typically evaluated by the Intersection over Union (IoU) score. In this work, we show that humans have a different view on localization quality. To evaluate this, we conduct a survey with more than 70 participants. Results show that for localization errors with the exact same IoU score, humans might not consider that these errors are equal, and express a preference. Our work is the first to evaluate IoU with humans and makes it clear that relying on IoU scores alone to evaluate localization errors might not be sufficient.
Simulator Attack+ for Black-Box Adversarial Attack

Numerous researches on adversarial black-box attacks have proved that deep neural networks have certain insecurity. However, the current black-box attack methods still have shortages in incomplete utilization of query information. The newly proposed Simulator Attack based on meta-learning shows good performance in query-efficiency but still misses some hidden information. For this disadvantage, our research finds the usability of the feature layer output information in a simulator model for the first time. Then we propose an optimized Simulator Attack+ framework based on this discovery. By conducting experiments on the CIFAR-10 and CIFAR-100 datasets, results legibly show that Simulator Attack+ can further reduce the number of consuming queries to improve query-efficiency meanwhile maintaining attack effect. Our code is available at https://github.com/Rain117E/SimulatorAttackplus.
SeqDNet: Improving Missing Value by Sequential Depth Network

Depth estimation is essential for many applications, such as 3D reconstruction and object recognition. In practice, the captured depth images naturally suffer from the noise and invalid depth value, i.e, depth noise and depth hole. Despite making a process on filling a small number (less than 10%) of depth holes, the performance of the prior depth reconstruction systems drops significantly when coping with the larger hole region (up to 25%). To tackle this, we propose a novel Sequential Depth Networks (SeqDNet). Instead of filling holes in coarse scale, our model tends to leverage the information in the spatial structures constructed from depth images. There are two novel components in the proposed SeqDNet: (1) We slice the depth map from near to far, generating the spatial structure, named sequential depth. It provides residual depth information, which was neglected in previous works. (2) The edge awareness loss (EA Loss) matches the gradient of the depth map by controlling the weight with the normal map so as to provide additional instruction (whether the position is the edge or not) when filling depth holes. Experimental results show a significant performance gain on filling images with 25% depth holes using SeqDNet. Moreover, a comprehensive analysis is given to pave a path for future researches addressing the large hole size on the depth maps.
Invited Talk #7: From resilience to Resistive Memory variability in Binarized Neural Networks to exploitation of variability in Bayesian Neural Network

Resistive memory technologies are promising technologies for energy-efficient hardware implementation of intelligent systems that process information locally at the edge.However, current approaches are difficult to implement due to the intrinsic non-idealities of resistive memory.We present on the one hand binarized neural networks that are extremely resilient to errors and which operates at low energy consumption regimes and on the other hand a machine learning scheme that exploits memristor variability to implement Markov chain Monte Carlo sampling configured as a Bayesian machine learning model.Our approaches demonstrates robustness to device variability, degradation, and, based on circuit and systemlevel simulations, the total energy to perform a classification and to train models is estimated to be two order of magnitude lower than in complementary metal-oxidesemiconductor (CMOS)-based approaches.
Reference-Based Blind Super-Resolution Kernel Estimation

Over the past few years, blind image super-resolution kernel estimation has become an emerging research topic. Re-cent SR kernel estimation works KernelGAN and FKP, as well as KernelNet, have shown promising results when applied to real-world SR problems. Among them, KernelNet improves the accuracy of SR kernel estimation while reducing computation time. Despite this, none of these previous studies addressed the estimation of SR kernel for scenarios that incorporate reference higher quality images. Our study presents KernelNet-R, a reference-based SR kernel estimator network. In both synthetically-generated pairs, as well as real-world pairs of images from wide-angle and telephoto camera images, the proposed solution reaches state-of-the-art (SoTa) kernel estimation performance.