Reviewerly/exHarmony huggingface dataset:
Document Layout Analysis Via Positional Encoding

Document layout analysis plays a vital role in computer vision research. Current document layout analysis methods mostly use pixel-based classification for document layout analysis. However, the method based on pixel classification is insufficient for maintaining the continuity of the classification area. In this paper, we propose a document layout analysis method based on positional encoding and bounding box specification. We maintain the continuity of the analysis area by constructing a document layout analysis framework based on the bounding box. In addition, we also integrate a positional encoding module in the framework to maintain the detailed information in the document layout analysis and modeling process. Experimental results prove that our proposed method has achieved state-of-the-art results.
MLP-Stereo: Heterogeneous Feature Fusion in MLP for Stereo Matching

CNNs’ strong inductive biases of locality and weight sharing provide powerful representation ability and data sample utilization efficiency. However, the weight sharing might smooth out the discrepancy between similar pixels, resulting in the wrong matching between left and right camera-image pair in thin structures region and repetitive texture region. In this paper, we propose a novel Heterogeneous Feature Fusion in MLP (HFF-MLP) for Stereo matching. It employs MLP structure and relaxes the weights sharing in the local spatial region. To this end, pixels in thin structures region and repetitive texture region are dealt with independently using the exclusive weights. Based on HFF -MLP module, we design a real-time network, i.e., MLP-Stereo. Experimental results show that our proposed HFF-MLP achieves competitive results on KITTI 2015 test dataset with the running time of 46 milliseconds. Furthermore, it performs much better than other real-time network in thin structures region and repetitive texture region.
AEBSR: Active-Sampling and Energy-Based Single Image Super-Resolution

Traditionally, single image super-resolution (SISR) methods randomly crop fixed-size patches in both low and high resolution (LR and HR) images as training samples, and obtain reconstruction model through the regression of LR-HR pixels pairs. However, these will lead to two problems. One is the negligence of the essential information of textures and edges leading to redundant and inefficient training. The other is the lack of the overall perception of data distribution causing poor generalization. To mitigate these issues, we propose Active Sampling and Energy-Based Single Image Super-Resolution (AEBSR), which introduces Active Sampling (AS) and Energy-Based Training (EBT) into SISR. Specifically, we first actively sample texture and edge patches through information entropy, and then align different data distributions between SR and HR images through free energy to perceive the overall distribution characteristics. Extensive experiments show that AS and EBT can further improve the SISR effect and our AEBSR also achieves competitive results compared to the current state-of-the-art SISR approaches.
A Study of Shape Modeling Against Noise

Shape modeling is a challenging task with many potential applications in computer vision and medical imaging. There are many shape modeling methods in the literature, each with its advantages and applications. However, many shape modeling methods have difficulties handling shapes that have missing pieces or outliers. In this regard, this paper introduces shape denoising, a fundamental problem in shape modeling that lies at the core of many computer vision and medical imaging applications and has not received enough attention in the literature. The paper introduces six types of noise that can be used to perturb shapes as well as an objective measure for the noise level and for comparing methods on their shape denoising capabilities. Finally, the paper evaluates seven methods capable of accomplishing this task, of which six are based on deep learning, including some generative models.
Stochastic Binary-Ternary Quantization for Communication Efficient Federated Computation

A stochastic binary-ternary (SBT) quantization approach is introduced for communication efficient federated computation; form of collaborative computing where locally trained models are exchanged between institutes. Communication of deep neural network models could be highly inefficient due to their large size. This motivates model compression in which quantization is an important step. Two well-known quantization algorithms are binary and ternary quantization. The first leads into good compression, sacrificing accuracy. The second provides good accuracy with less compression. To better benefit from trade-off between accuracy and compression, we propose an algorithm to stochastically switch between binary and ternary quantization. By combining with uniform quantization, we further extend the proposed algorithm to a hierarchical method which results in even better compression without sacrificing the accuracy. We tested the proposed algorithm using Neural network Compression Test Model (NCTM) provided by MPEG community. Our results demonstrate that the hierarchical variant of the proposed algorithm outperforms other quantization algorithms in term of compression, while maintaining the accuracy competitive to that provided by other methods.
Streaming-Capable High-Performance Architecture of Learned Image Compression Codecs

Learned image compression allows achieving state-of-the-art accuracy and compression ratios, but their relatively slow runtime performance limits their usage. While previous attempts on optimizing learned image codecs focused more on the neural model and entropy coding, we present an alternative method to improving the runtime performance of various learned image compression models. We introduce multi-threaded pipelining and an optimized memory model to enable GPU and CPU workloads’ asynchronous execution, fully taking advantage of computational resources. Our architecture alone already produces excellent performance without any change to the neural model itself. We also demonstrate that combining our architecture with previous tweaks to the neural models can further improve runtime performance. We show that our implementations excel in throughput and latency compared to the baseline and demonstrate the performance of our implementations by creating a real-time video streaming encoder-decoder sample application, with the encoder running on an embedded device.
Context-Aware Hierarchical Transformer for Fine-Grained Video-Text Retrieval

Video-Text Retrieval aims to perform accurate retrieval process that adopts texts to retrieve the corresponding videos, and vice versa. Typically, mainstream methods solve this problem by learning a common joint embedding space, and then measure the similarities between videos and texts. However, these methods lack the ability to represent detailed semantic information. Therefore, we first utilize three pre-trained models to construct the video embeddings of different semantic levels, and then propose a Context-aware Hierarchical Transformer (CHT) model to encode the context information between these levels. More specifically, our model builds fine-grained hierarchical video embeddings of three semantic levels: global, objects, and actions. Attention-based contextual transformers are utilized to establish the context interactions between different semantic levels. Experimental results on two benchmark video-text retrieval datasets demonstrate the superiority of our CHT model. Ablation studies also prove the effectiveness of our proposed model.
DeepSAR: Vessel Detection in SAR Imagery with Noisy Labels

Unlike traditional optoelectronic satellite imaging, Synthetic Aperture Radar (SAR) allows remote sensing applications to operate under all weather conditions. This makes it uniquely valuable for detecting ships/vessels involved in illegal, unreported, and unregulated (IUU) fishing. While recent work has shown significant improvement in this domain, detecting small objects using noisy point annotations remains an unexplored area. In order to meet the unique challenges of this problem, we propose a progressive training methodology that utilizes two different spatial sampling strategies. Firstly, we use stochastic sampling of background points to reduce the impact of class imbalance and missing labels, and secondly, during the refinement stage, we use hard negative sampling to improve the model. Experimental results on the challenging xView3 dataset show that our method outperforms conventional small object localization methods in a large, noisy dataset of SAR images.Source code for our method can be found at: https://github.com/manupillai308/DeepSAR
3D Clues Guided Convolution for Depth Completion

Depth completion is a task that recovers a dense depth map from a sparse depth map with the corresponding color image. Recently, the intensive depth generation guided by image clues in the color map has achieved good results. Color images can provide structural and semantic information as guidance information, but cannot provide the more important information about geometric relationships. In this paper, we propose a novel network to learn latent 3D cues from RGB images and depth images. More specifically, the network contains a 3D clues extractor and a dense depth generator. The extractor is designed to fusion and extract the 3D joint clues from the color image and sparse depth. The generator is trained with the sparse depth map and 3D clues to producing a more accurate dense depth map. Extensive experiments show that our proposed method has a significant improvement over existing image-guided methods.
Emcenet: Efficient Multi-Scale Context Exploration Network for Salient Object Detection

Multi-scale context is crucial for the accurate salient object detection (SOD) in the real-world scenes. Although current contextual information-based SOD methods have achieved great progress, they may fail to generate precise saliency maps due to their seldom considering the correlation of different scale context during the extraction process. To address these issues, we propose an Efficient Multi-Scale Context Exploration Network (EMCENet) for SOD. Specifically, a progressive multi-scale context extraction (PMCE) module is designed to progressively capture strongly correlated multi-scale context by using multi-receptive-field convolution operations. Afterwards, a hierarchical feature hybrid interaction (HFHI) module is introduced to generate powerful feature representations by adaptively aggregating multi-level features in a hybrid interaction strategy. Extensive experimental results on six public datasets demonstrate that the proposed EMCENet method without any post-processing performs favorably against 13 state-of-the-art SOD methods.
A Multi-Scale Content-Insensitive Fusion CNN For Source Social Network Identification

Identification of source social networks of images based on the traces left on images by such platforms is a crucial task in image forensics. The existing techniques provide successful solutions to such a problem. However, we show that the state-of-the-art techniques are adversely affected due to the leaking side-channel information from scene details that convolutional neural networks (CNNs) are prone to exploit. Thus, highly correlated scene details in the training and test sets lead to overestimation of the performance. To address this problem, we develop a data-driven system by parallelizing three CNNs having kernels with different sizes that benefit from learning more relevant forensic traces making the model less susceptible to scene content. The experimental results achieved by the proposed model either trained on images with or without scene overlap show that there is no influence of scene content in the feature learning of the proposed method.
Multi-view 3D Reconstruction from Video with Transformer

Multi-view 3D reconstruction is the base for many other applications in computer vision. Video provides multi-view images and temporal information, which can help us better complete the reconstruction goal. Redundant information handling in video and multi-view feature extraction and fusion become the key issues in the shape prior extraction for reconstruction. In this paper, inspired by the recent great success in Transformer models, we propose a transformer-based 3D reconstruction network. We formulate the multi-view 3D reconstruction into three parts: frame encoder, fusion module, and shape decoder. We apply several special used tokens and perform the fusion progressively in the encoder phase, called patch-level progressive fusion module. These tokens describe which part of the object the frame should focus on and the local structural detail progressively. Then we further design a transformer fusion module to aggregate the structure information. Finally, multi-head attention is utilized to build the transformer-based decoder to reuse the shallow features from encoder. In experiments not only can ours method achieve competitive performance, but it also has low model complexity and computation cost.
Hidden Conditional Adversarial Attacks

Deep neural networks are vulnerable to maliciously crafted inputs called adversarial examples. Research on unprecedented adversarial attacks is significant since it can help strengthen the reliability of neural networks by alarming potential threats against them. However, since existing adversarial attacks disturb models unconditionally, the resulting adversarial examples increase their detectability through statistical observations or human inspection. To tackle this limitation, we propose hidden conditional adversarial attacks whose resultant adversarial examples disturb models only if the input images satisfy attackers’ pre-defined conditions. These hidden conditional adversarial examples have better stealthiness and controllability of their attack ability. Our experimental results on the CIFAR-10 and ImageNet datasets show their effectiveness and raise a serious concern about the vulnerability of CNNs against the novel attacks.
Soli Radar Image-Based Target Localization

Target localization is an important task for a broad range of ubiquitous computing applications. A great deal of previous work has focused on applying advanced machine learning methods to address image-based target detection and localization problems. Some work has also explored the localizing capabilities of radar signals that describe target distances and velocities. This paper proposes a novel machine learning architecture, SoliNet, to localize target positions from radar images alone. Radar data were acquired by Google’s Soli sensors and processed to generate range-Doppler images describing the motion of a radar target. Corresponding camera images of the target were used as validation data. Over 60 experiments including six unique target trajectories were performed using a robotic actuator rig. The SoliNet machine learning architecture was then trained to accomplish localization using the known location of the target over time as dictated by the programmed robot trajectory. SoliNet is an end-to-end trained combination of deep convolutional and recurrent neural networks. Experimental results showed an overall accuracy of 98.6% and equivalent effectiveness to camera-based target localization, demonstrating the promising capabilities of the proposed technique.
Linear Discriminant Analysis Metric Learning Using Siamese Neural Networks

We propose a method for learning the Linear Discriminant Analysis (LDA) using a Siamese Neural Network (SNN) architecture for learning a low dimensional image descriptor. The novelty of our work is that we learn the LDA projection matrix between the final fully-connected layers of an SNN. An SNN architecture is used since the proposed loss maximizes the Kullback-Leibler divergence between the feature distributions from the two branches of an SNN. The network learns an optimized feature space having inherent properties pertaining to the learning of LDA. The learned image descriptors are a) low-dimensional, b) have small intra-class variance, c) large inter-class variance, and d) can distinguish the classes with linear decision hyperplanes. The proposed method has the advantage that LDA learning happens end-to-end. We measured the classification accuracy in the three datasets MNIST, CIFAR-10, and STL-10 and compared the performance with other state-of-the-art methods. We also measured the KL divergence between the class pairs and visualized the projections of feature vectors along the learned discriminant directions.
Object-Aware Self-Supervised Multi-Label Learning

Multi-label Learning on image data has been widely exploited with deep learning models. However, supervised training on deep CNN models often cannot discover sufficient discriminative features for classification. As a result, numerous self-supervision methods are proposed to learn more robust image representations. However, most self-supervised approaches focus on single-instance single-label data and fall short on more complex images with multiple objects. Therefore, we propose an Object-Aware Self-Supervision (OASS) method to obtain more fine-grained representations for multi-label learning, dynamically generating auxiliary tasks based on object locations. Secondly, the robust representation learned by OASS can be leveraged to efficiently generate Class-Specific Instances (CSI) in a proposal-free fashion to better guide multi-label supervision signal transfer to instances. Extensive experiments on the VOC2012 dataset for multi-label classification demonstrate the effectiveness of the proposed method against the state-of-the-art counterparts.
Learning Selective Assignment Network for Scene-Aware Vehicle Detection

Deep learning has shown remarkable success in data-driven vehicle detection, relying on collected training samples from known scenes. A challenging problem arises when these detectors handle agnostic scenes, while keeping the performance of previous ones. To address this issue, a feasible remedy is to learn a set of domain-adaptive detectors by aligning the features from one scene to another. However, the improvement obtained in this way is inflexible despite the progress in object detection. An important reason is that the memory sizes grow massively with deliberately saving all scenes-independent detectors, while ignoring the relationship among different scenes. In this paper, we aim to bridge the gap between scene diversification and object consistency for scene-aware vehicle detection. Specifically, a novel structured network is proposed to integrate selective assignment of scene-specific parameters into the vehicle detection framework. Extensive experiments conducted on different scenes including BDD, Cityscapes-car, CARPK, etc, demonstrate that the proposed method achieves impressive performance, while keeping the performance of previous scenes as the scene changes.
Transformation-Based Adversarial Defense Via Sparse Representation

Deep neural networks are vulnerable to adversarial perturbation. Recent researches indicate that misclassification may result from the distribution mismatch between adversarial examples and clean images. Inspired by a common consensus that the human neural representation is sparse and redundant, we propose an input-transformation-based defense method based on sparse representation to bridge the distribution mismatch. In our method, we first learn a global overcomplete dictionary from a set of image patches extracted from training images, and then, purify input images before feeding them into the neural networks, using the sparse representation technique with the learned dictionary. Compared with other attack-agnostic defenses, our method obtains comparable results on CIFAR-10 and ImageNet in various attack settings.
Gaussian Distributed Graph Constrained Multi-Modal Gaussian Process Latent Variable Model for Ordinal Labeled Data

This paper proposes a Gaussian distributed graph constrained multi-modal Gaussian process latent variable model for ordinal labeled data. Rating data that are used in various real-world applications such as product recommendation can represent user preferences, but the difference between adjacent ratings is often uncertain due to the user’s ambiguity. In order to capture the relationships among multi-modal data including rating data, consideration of the uncertainty is necessary. Therefore, by applying the Gaussian distribution to the rating data, we calculate distributed labels that implicitly include the uncertainty, and thus, the Gaussian distributed graph based on their similarities can be constructed. By introducing a constraint calculated based on the graph Laplacian of the Gaussian distributed graph into the objective function of the multi-modal Gaussian process latent variable model, we can achieve an effective latent space that can consider a label correlation while accounting for the uncertainty. This is the contribution of this paper. The effectiveness of the proposed method is verified by experiments using some open datasets.
Video-Grounded Dialogues with Joint Video and Image Training

In this paper, we propose a multi-modal transformer model for end-to-end training of video-grounded dialogue generation. In particular, LayerScale regularized spatio-temporal self-attention blocks are first introduced to enable us to flexibly train end-to-end from both video and image data, without extracting offline visual features. Further, a pre-trained generative language architecture BART is employed to encode different modalities and perform dialogue generation. Extensive experiments on Audio-Visual Scene-Aware Dialog (AVSD) dataset demonstrate its effectiveness and superiority to the state-of-the-art methods.
Starvqa: Space-Time Attention for Video Quality Assessment

Transformer based on self-attention mechanism is blooming in computer vision nowadays. However, its application to video quality assessment (VQA) has not been reported. Evaluating the quality of in-the-wild videos is challenging due to the unknown of pristine reference and shooting distortion. This paper presents a novel space-time attention network for the VQA problem, named StarVQA. StarVQA builds a Transformer by alternately concatenating the divided space-time attention. To adapt the Transformer architecture for training, StarVQA designs a vectorized regression loss by encoding the mean opinion score (MOS) to the probability vector and embedding a special vectorized label token as the learnable variable. To capture the long-range spatiotemporal dependencies of a video sequence, StarVQA encodes the space-time position information of each patch to the input of the Transformer. Various experiments are conducted on the de-facto in-the-wild video datasets, including LIVE-VQC, KoNViD-1k, LSVQ, and LSVQ-1080p. Experimental results demonstrate the superiority of StarVQA over the state-of-the-art. The source code is available at https://github.com/GZHU-DVL/StarVQA.