Reviewerly/exHarmony huggingface dataset:
Multi-Modal Hierarchical Attention-Based Dense Video Captioning

Most of the existing dense video captioning models use a single modality of features for captioning. A video has a wide variety of information like spatial features, temporal features, audio features, and semantic features. In this paper, we propose a dense video captioning model that captures crossmodal attention between different types of features using an audio-visual attention block in the encoder and a hierarchical attention block in the decoder. The audio-visual attention block applies cross-modal attention between the RGB, flow, and audio features. The hierarchical attention block performs two-level attention between the semantic features and the features from the encoder for generating descriptions. The results show that the proposed approach performs better than the state-of-the-art approaches.
Feature-Aware Prohibited Items Detection for X-Ray Images

Prohibited items detection is a challenging problem in security inspection. Since the commonly used manual security screening is a subjective, inefficient and costly process, some researchers have used existing models designed for natural images and their modifications to replace it. However, these methods do not take full advantage of the unique properties of X-ray images to solve its clutter and occlusion problem in detection. In this paper, we propose a feature-aware prohibited items detection (FAPID) method for X-ray images to detect normal and overlapped heavily prohibited items. The overall framework consists of shape-guided feature enhancement module (SGFE) and prohibited item aware module (PIA). The SGFE module enhances the items’ structural integrity in the learned features by utilizing the strong shapes in X-ray images without losing texture features. And the PIA module learns discriminative fine contour and texture of prohibited items without the influence of overlapping noise by using frequency information and the feedback-like mechanism. Extensive experiments conducted on HiXray and OPIXray datasets verify the effectiveness of the proposed method compared to other SOTA methods.
An Improved Upper Bound on the Rate-Distortion Function of Images

Recent work has shown that Variational Autoencoders (VAEs) can be used to upper-bound the information rate-distortion (R-D) function of images, i.e., the fundamental limit of lossy image compression. In this paper, we report an improved upper bound on the R-D function of images implemented by (1) introducing a new VAE model architecture, (2) applying variable-rate compression techniques, and (3) proposing a novel smoothing function to stabilize training. We demonstrate that at least 30% BD-rate reduction w.r.t. the intra prediction mode in VVC codec is achievable, suggesting that there is still great potential for improving lossy image compression. Code is made publicly available <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Background Clustering Pre-Training for Few-Shot Segmentation

Recent few-shot segmentation (FSS) methods introduce an extra pre-training stage before meta-training to obtain a stronger backbone, which has become a standard step in few-shot learning. Despite the effectiveness, current pre-training scheme suffers from the merged background problem: only base classes are labelled as foregrounds, making it hard to distinguish between novel classes and actual background. In this paper, we propose a new pre-training scheme for FSS via decoupling the novel classes from background, called Background Clustering Pre-Training (BCPT). Specifically, we adopt online clustering to the pixel embeddings of merged background to explore the underlying semantic structures, bridging the gap between pre-training and adaptation to novel classes. Given the clustering results, we further propose the background mining loss and leverage base classes to guide the clustering process, improving the quality and stability of clustering results. Experiments on PASCAL-5 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> and COCO-20 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">i</sup> show that BCPT yields advanced performance. Code will be available at https://github.com/Carboxy/BCPT.
A Penalized Modified Huber Regularization to Improve Adversarial Robustness

Adversarial training (AT) is a learning procedure that trains a deep neural network with adversary examples to improve robustness. AT and its variants are widely considered the most empirically successful against adversary examples. Along the same line, this work proposes a new training objective, PMHR-AT (Penalized Modified Huber Regularization for Adversarial training) for improving adversarial robustness. PMHR-AT minimizes both natural and adversarial risk and introduces a modified Huber loss between the natural and adversarial logits as a regularization with the regularization strength adjusted based on the similarity between the predicted natural and adversarial class probabilities. Experimental results show that the proposed method recorded a better performance than existing methods on strong attacks and offers a better trade-off between the natural accuracy and adversarial robustness.
Multi-Scale Transformer Network for Saliency Prediction on 360-Degree Images

The latest methods for saliency prediction on 360° images show that better results can be obtained using equirectangular (ERP) images as input. Due to the limitation of the receptive field, existing convolution-based networks cannot capture long-range information in complex 360° images. Although the transformer has the innate ability to capture long-range correlations with self-attention, large dataset requirement limit its application in saliency prediction of 360° images. In this paper, we present a novel Multi-scale Transformer framework for Saliency prediction on 360° images (MTSal360). The Multi-scale Transformer Module (MTM) is designed in the network to aggregate the contextual long-range information, which includes a Convolutional Positional Encoder (CPE) to enable the model could train and test on cubic and ERP format separately to address the insufficient data. Experiments on two public datasets illustrate that MTSal360 achieves better results over the state-of-the-art methods.
Flow-Guided Deformable Attention Network for Fast Online Video Super-Resolution

Real-time online video super-resolution (VSR) on resource limited applications is a very challenging problem due to the constraints on complexity, latency and memory foot-print, etc. Recently, a series of fast online VSR methods have been proposed to tackle this issue. In particular, attention based methods have achieved much progress by adaptively aligning or aggregating the information in preceding frames. However, these methods are still limited in network design to effectively and efficiently propagate the useful features in temporal domain. In this work, we propose a new fast online VSR algorithm with a flow-guided deformable attention propagation module, which leverages corresponding priors provided by a fast optical flow network in deformable attention computation and consequently helps propagating recurrent state information effectively and efficiently. The proposed algorithm achieves state-of-the-art results on widely-used benchmarking VSR datasets in terms of effectiveness and efficiency. Code can be found at https://github.com/IanYeung/FastOnlineVSR.
Domain invariant regularization by disentangling content and style Features for visual domain generalization

In this paper, taking the advantage of multiple source domains, we propose a novel approach for visual Domain Generalization (DG). The three key ideas underlying our formulation are (1) leveraging disentangled representations of the images to define different factors of variations, (2) generating perturbed images by changing such factors composing the representations of the images, (3) enforcing the learner (classifier) to be invariant to such changes in the images. We demonstrate the effectiveness of our approach on several widely used datasets for the domain generalization problem, on all of which we achieve competitive results with state-of-the-art models.
Image Translation-Based Deniable Encryption against Model Extraction Attack

In cloud storage applications, data owners’ original images are usually encrypted before being outsourced to the cloud for preserving data owners’ privacy. However, in deep learning model-based image encryption methods, an adversary can conduct the model extraction attack to reveal the model parameters and thus restore the privacy information by obtaining numerous encrypted images. In this paper, we propose an image translation-based deniable encryption (ITDE) scheme to achieve encryption deniability and defend against model extraction attacks. Differing from traditional encryption methods in which encrypted images are visually meaningless, ITDE applies image translation to generate encrypted images in the form of human faces. Moreover, ITDE provides deniability for data owners to keep the encryption parameters private. To defend against model extraction attacks, the defense mechanism is introduced in our proposed ITDE to preserve deep learning models. Experimental results demonstrate the superiority of our proposed methods in terms of encryption deniability and privacy preservation.
Simple Self-Distillation Learning for Noisy Image Classification

In computer vision, considerable success has been achieved in image classification. Yet, real-world applications can be negatively affected by noise corruption, so a method for dealing with noise is crucial. Knowledge distillation that utilizes the knowledge of a teacher model trained on clean images to train a student model on noisy images is a promising technique because it can be applied without special modification of the classifier. However, clean images are typically not available for most practical uses. To address this issue, we proposed a novel knowledge distillation method without clean images. By leveraging a property of the feature extractor in the classifier that naturally removes unrelated features for classification, we perform a simple training of the teacher model on noisy images, under the assumption that such a teacher can provide pseudo-clean features. Our experiments demonstrate that the proposed method can achieve classification performance comparable to conventional methods, even without clean images.
Encoding-Aware Deep Video Super-Resolution Framework

Video super-resolution(VSR) upscales a low-resolution video to the higher one. Most applications require compression of the super-resolved video due to limited internet bandwidth and storage capacity. However, most studies on VSR techniques have focused only on improving image quality, ignoring the impact of the compression process on visual quality. Consequently, even a VSR with good visual quality has a risk of significant loss of quality when serviced online or stored as a file. To address this problem, we propose an encoding-aware VSR framework. In the framework, we created a differentiable virtual codec to estimate the bit rate and used it for the loss function, which optimizes the super-resolved videos by considering the rate-distortion trade-off relationship and eventually leads to the prevention of visual quality degradation. According to the results, our real-time VSR model for x4 upscaling, trained with 1,191K parameters, yields a maximum gain of 13.2% over state-of-the-art VSR models based on the Bjøntegaard delta rate.
Learning Raw Image Denoising Using a Parametric Color Image Model

Deep learning methods for image restoration have produced impressive results over recent years. Nevertheless, they generalize poorly and need large learning image datasets to be collected for each new acquisition modality. In order to avoid the building of such datasets, it has been recently proposed to develop synthetic image datasets for training image restoration methods, using scale invariant dead leaves models. While the geometry of such models can be successfully encoded with only a few parameters, the color content cannot be straightforwardly encoded. In this paper, we leverage the concept of color lines prior to build a light parametric color model relying on a chromaticity/luminance factorization. Further, we show that the corresponding synthetic dataset can be used to train neural networks for the denoising of RAW images from different camera-phones, without using any image from these devices. This shows the potential of our approach to increase the generalization capacity of learning-based denoising approaches in real case scenarios.
FGCVQA: Fine-Grained Cross-Attention for Medical VQA

The application of Visual Question Answering (VQA) in the medical field has significantly impacted traditional medical research methods. A mature medical visual question answering system can greatly help the patients’ diagnosis. The Visual Question Answering Model in the generic domain is not compelling enough for the feature alignment in medical image and text semantics because of the complex diversity of clinical problems and the difficulties in multi-modal reasoning. To solve these, we propose a model called FGCVQA. It is essential to consider the semantic alignment of the medical images and the language features. Specifically, We use the Cross-Modality Encoder to learn the semantic representation of medical images and texts. It improves the reasoning ability of multi-modal by considering the fine-grained property. The experimental results show that FGCVQA outperforms all previous dataset VQA-RAD methods for radiology images. FGCVQA effectively answers medical visual questions and can help doctors to make better clinical analyses and diagnoses. The source codes can be available at https://github.com/wwzziheng/FGCVQA.
Selecting A Diverse Set Of Aesthetically-Pleasing and Representative Video Thumbnails Using Reinforcement Learning

This paper presents a new reinforcement-based method for video thumbnail selection (called RL-DiVTS), that relies on estimates of the aesthetic quality, representativeness and visual diversity of a small set of selected frames, made with the help of tailored reward functions. The proposed method integrates a novel diversity-aware Frame Picking mechanism that performs a sequential frame selection and applies a reweighting process to demote frames that are visually-similar to the already selected ones. Experiments on two benchmark datasets (OVP and YouTube), using the top-3 matching evaluation protocol, show the competitiveness of RL-DiVTS against other SoA video thumbnail selection and summarization approaches from the literature.
Tsanet: Temporal and Scale Alignment for Unsupervised Video Object Segmentation

Unsupervised Video Object Segmentation (UVOS) refers to the challenging task of segmenting the prominent object in videos without manual guidance. In recent works, two approaches for UVOS have been discussed that can be divided into: appearance and appearance-motion-based methods, which have limitations respectively. Appearance-based methods do not consider the motion of the target object due to exploiting the correlation information between randomly paired frames. Appearance-motion-based methods have the limitation that the dependency on optical flow is dominant due to fusing the appearance with motion. In this paper, we propose a novel framework for UVOS that can address the aforementioned limitations of the two approaches in terms of both time and scale. Temporal Alignment Fusion aligns the saliency information of adjacent frames with the target frame to leverage the information of adjacent frames. Scale Alignment Decoder predicts the target object mask by aggregating multi-scale feature maps via continuous mapping with implicit neural representation. We present experimental results on public benchmark datasets, DAVIS 2016 and FBMS, which demonstrate the effectiveness of our method. Furthermore, we outperform the state-of-the-art methods on DAVIS 2016.
Global Balanced Networks for Multi-View Stereo

Existing deep learning-based multi-view stereo (MVS) methods have achieved satisfactory results in terms of accuracy of reconstruction results, but since these methods introduce significant ambiguity in low-texture or even no-texture, or highly reflective regions, this leads to poor depth prediction in low-texture smooth regions, which directly affects the completeness of the reconstruction results. Therefore, how to strike a balance between accuracy and integrity is currently a major challenge for MVS. Based on this problem, we propose a global balanced network(GBNet) for multi-view stereo which aims to maximize completeness with guaranteed accuracy to achieve relative balance. Specifically, our network consists of three parts. The first part is our proposed a feature distillation modular, which mainly serves to reduce feature redundancy and enhance the expressiveness of important distinguishing features. The second part is our hydra attention fusion module, which maximizes the capture of global contextual features to build compact cost-volume. In addition, we propose a non-standard 3D convolution module for aggregating the depth features. Experiments conducted on the DTU and tank temple datasets show that our method outperforms many current state-of-the-art methods in terms of completeness and exhibits a certain degree of generalization performance.
Data Generation with Structure Enforcing Adversarial Learning

Class imbalance issues are very common among real-world datasets. Traditional oversampling approaches are interpolation based and are not well suited for image datasets. These techniques lead to class overlapping and the generation of visually unappealing minority class images. Lately, Generative Adversarial Network (GAN)-based models are used widely for oversampling of image data; however, the learning bias towards the majority classes lead to generation of majority classes in excess and minority classes in rarity. Most of the existing oversampling techniques work on data space, whereas low-dimensional latent space for oversampling is less explored. To tackle these issues, we propose a novel latent space oversampling framework called Structure Enforcing Adversarial Learning (SEAL). In the proposed architecture, the generator is trained by additionally minimizing the structure loss. This boosts the generation of synthetic samples, which retain the covariance structure of each class. The proposed model is evaluated on four image datasets and is compared with the state-of-the-art methods.
A Feature Refinement Module for Light-Weight Semantic Segmentation Network

Low computational complexity and high segmentation accuracy are both essential to the real-world semantic segmentation tasks. However, to speed up the model inference, most existing approaches tend to design light-weight networks with a very limited number of parameters, leading to a considerable degradation in accuracy due to the decrease of the representation ability of the networks. To solve the problem, this paper proposes a novel semantic segmentation method to improve the capacity of obtaining semantic information for the light-weight network. Specifically, a feature refinement module (FRM) is proposed to extract semantics from multi-stage feature maps generated by the backbone and capture non-local contextual information by utilizing a transformer block. On Cityscapes and Bdd100K datasets, the experimental results demonstrate that the proposed method achieves a promising trade-off between accuracy and computational cost, especially for Cityscapes test set where 80.4% mIoU is achieved and only 214.82 GFLOPs are required.
Multi-Scale Temporal Feature Fusion for Few-Shot Action Recognition

The aim of this paper is to recognize actions of interest that are given by a few support videos in testing (query) videos. The focus of our approach is to develop a novel temporal enrichment module where the features describing local temporal contexts in videos are enhanced by collaboratively merging important information in frame-level (no temporal context) features. We call this module a multi-scale temporal feature fusion (MSTFF) module. Utilizing multiple MSTFF modules varying the scope of local temporal context extraction, we can obtain discriminative video representation which is crucial in the few-shot tasks where support videos are not sufficient to describe an action class. For stable learning of a model with MSTFF and the performance boost, we also learn a local temporal context-level auxiliary classifier in parallel with the main classifier. We analyze the proposed components to demonstrate their importance. We achieve state-of-the-art on three few-shot action recognition benchmarks: Something-Something V2 (SSv2), HMDB51, and Kinetics.
Ladder Siamese Network: A Method and Insights for Multi-Level Self-Supervised Learning

In Siamese-network-based self-supervised learning (SSL), multilevel supervision (MLS) is a natural extension to enforce intermediate representations’ consistency against data augmentations. Although existing studies have incorporated MLS to boost their system performances in combination with other ideas, vanilla MLS has not been deeply analyzed. Here, we extensively investigate how MLS works and how much impact it has on SSL performance with various training settings to understand the effectiveness of MLS by itself. For this investigation, we develop a simple Siamese-SSL-based MLS framework Ladder Siamese Network, equipped with multi-level, non-contrastive, and global/local self-supervised training losses. We show that the proposed framework can simultaneously improve BYOL baselines in classification, detection, and segmentation solely by adding MLS. In comparison with the state-of-the-art methods, our Ladder-based model achieves competitive and balanced performances in all tested benchmarks without causing large degradation in any of them, which suggests the usability for building a multi-purpose backbone.
A Multi-Modal Transformer Approach for Football Event Classification

Video understanding has been enhanced by the use of multi-modal networks. However, recent multi-modal video analysis models have limited applicability to sports videos due to their specialised nature. This paper proposes a novel attention-based multi-modal neural network for sports event classification featuring a multi-stage fusion training strategy. The proposed multi-modal neural network integrates three modalities, including an image sequence modality, an audio modality and a newly proposed sports formation modality, to improve the sports video classification performance. Empirical results show that the proposed model outperforms the state-of-the-art transformer-based video method by 4.43% on top-1 accuracy on Soccernet-V2 dataset.
False Correspondence Removal via Revisiting Semantic Context with Position-Attentive Learning

False correspondence removal remains a challenge for many feature-matching-based applications. This paper proposes a solution that revisits the local semantic context via position-attentive learning. First, a cross-divisional module is introduced to extract semantic features from image-patch pairs. Then, through a position-attentive mechanism, the parametric positions and extracted semantic features are jointly utilized to compute the probabilities of a set of putative correspondences. The proposed approach is evaluated on several indoor and outdoor challenging datasets, containing up to 70%+ false correspondences. In most cases, the solution outperforms existing algorithms in terms of matching precision and F1-score. Furthermore, an ablation study indicates that revisiting the semantic context improves precision by nearly 5%. The code is available at the link below <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .