Reviewerly/exHarmony huggingface dataset:
Integer Quantized Learned Image Compression

Learned image compression (LIC) has shown remarkable improvement compared to traditional methods, but requires increased computational and memory complexity. Network quantization is an effective way to resolve the issue of complexity, but quantization of LIC has not been explored much. In this paper, we propose an integer quantized LIC (IQ-LIC) via static quantization of both weights and activations as integers. We design a quantized convolution layer involving a new Leaky-Clip module. We also propose a squared quantization error loss to help efficient quantization-aware training. Experimental results show that IQ-LIC achieves better rate-distortion performance compared to existing methods.
Frequency Enhancement Network for Efficient Compressed Video Action Recognition

The existing frequency-based action recognition methods achieve impressive performance in improving efficiency. However, they ignore the low-frequency texture and edge clues, leading to accuracy degradation. To address this problem, we propose a novel frequency enhancement (FE) block for efficient compressed video action recognition, including a temporal-channel two-heads attention (TCTHA) module and a frequency overlapping group convolution (FOGC) module. First, the TCTHA module emphasizes the inter-frame temporal context and the inner-frame informative frequency semantics by attention. Then, the FOGC module groups channels in different frequency bands with overlap, to extract low-frequency texture and edge clues, while maintaining the interaction of groups. We integrate the FE block into 2D-CNNs with frequency I-frame input, termed FENet, focusing on the pivotal low-frequency spatio-temporal semantics for action recognition. Experiments on HMDB-51, UCF-101, Kinetics-400, and Kinetics-700 verify that our FENet achieves comparable accuracy compared with RGB-based methods with high efficiency.
Exploring Self-Supervised Representation Learning for Low-Resource Medical Image Analysis

The success of self-supervised learning (SSL) has mostly been attributed to the availability of unlabeled yet large-scale datasets. However, in a specialized domain such as medical imaging which is a lot different from natural images, the assumption of data availability is unrealistic and impractical, as the data itself is scanty and found in small databases, collected for specific prognosis tasks. To this end, we seek to investigate the applicability of self-supervised learning algorithms on small-scale medical imaging datasets. In particular, we evaluate 4 state-of-the-art SSL methods on three publicly accessible small medical imaging datasets. Our investigation reveals that in-domain low-resource SSL pre-training can yield competitive performance to transfer learning from large-scale datasets (such as ImageNet). Furthermore, we extensively analyse our empirical findings to provide valuable insights that can motivate for further research towards circumventing the need for pre-training on a large image corpus. To the best of our knowledge, this is the first attempt to holistically explore self-supervision on low-resource medical datasets. Source codes are available at: https://github.com/soumitri2001/SmallDataSSL
Dual Transformer Encoder Model for Medical Image Classification

Compared with convolutional neural networks, vision transformer with powerful global modeling abilities has achieved promising results in natural image classification and has been applied in the field of medical image analysis. Vision transformer divides the input image into a token sequence of fixed hidden size and keeps the hidden size constant during training. However, a fixed size is unsuitable for all medical images. To address the above issue, we propose a new dual transformer encoder model which consists of two transformer encoders with different hidden sizes so that the model can be trained with two token sequences with different sizes. In addition, the vision transformer only considers the class token output by the last layer in the encoders when predicting the category, ignoring the information of other layers. We use a Layer-wise Class token Attention (LCA) classification module that leverages class tokens from all layers of encoders to predict categories. Extensive experiments show that our proposed model obtains better performance than other transformer-based methods, which proves the effectiveness of our model.
Learning Spatially-Adaptive Style-Modulation Networks for Single Image Synthesis

Recently there has been a growing interest in learning generative models from a single image. This task is important as in many real world applications, collecting large dataset is not feasible. Existing work like SinGAN is able to synthesize novel images that resemble the patch distribution of the training image. However, SinGAN cannot learn high level semantics of the image, and thus their synthesized samples tend to have unrealistic spatial layouts. To address this issue, this paper proposes a spatially adaptive style-modulation (SASM) module that learns to preserve realistic spatial configuration of images. Specifically, it extracts style vector (in the form of channel-wise attention) and latent spatial mask (in the form of spatial attention) from a coarse level feature separately. The style vector and spatial mask are then aggregated to modulate features of deeper layers. The disentangled modulation of spatial and style attributes enables the model to preserve the spatial structure of the image without overfitting. Experimental results show that the proposed module learns to generate samples with better fidelity than prior works.
Weakly Semi-Supervised Oriented Object Detection with Points

Oriented object detection based on deep learning has received extensive attention while marking the Oriented Bounding Box (OBB) is time-consuming and laborious. For the oriented object detection task, this paper proposes a point-based weakly semi-supervised training strategy, which only requires the training set with an extremely small number (10%) of fully labeled images with OBB and the other with points. Specifically, following the common self-training pipeline, we propose Point to OBB Network (P2ONet) as the teacher model to generate the high-quality pseudo OBB for each point-annotated object. Inspired by channel attention, we introduce Group Attention to P2ONet to better tackle the opposite assignment for similar proposals in the training task leading by the point label assignment strategy. Furthermore, by exploring the constraint in normal self-training pipeline, we propose Confidence-Aware Loss to alleviate the impact of inaccurate pseudo-boxes. Experiments on the DOTA dataset show the close performance between our method and normal oriented object detection training methods with remarkably lower labeling costs.
Hands in Focus: Sign Language Recognition Via Top-Down Attention

In this paper, we propose a novel Sign Language Recognition (SLR) model that leverages the task-specific knowledge to incorporate Top-Down (TD) attention to focus the processing of the network on the most relevant parts of the input video sequence. For SLR, this includes information about the hands’ shape, orientation and positions, and motion trajectory. Our model consists of three streams that process RGB, optical flow and TD attention data. For the TD attention, we generate pixel-precise attention maps focusing on both hands, thereby retaining valuable hand information, while eliminating distracting background information. Our proposed method outperforms state-of-the-art on a challenging large-scale dataset by over 2%, and achieves strong results with a much simpler architecture compared to other systems on the newly released AUTSL dataset [1].
Unsupervised Anomaly Detection with Local-Sensitive VQVAE and Global-Sensitive Transformers

Unsupervised anomaly detection (UAD) has been widely implemented in industrial and medical applications, which reduces the cost of manual annotation and improves efficiency in disease diagnosis. Recently, deep auto-encoder with its variants has demonstrated its advantages in many UAD scenarios. Training on the normal data, these models are expected to locate anomalies by producing higher reconstruction error for the abnormal areas than the normal ones. However, this assumption does not always hold because of the uncontrollable generalization capability. To solve this problem, we present LSGS, a method that builds on Vector Quantised-Variational Autoencoder (VQVAE) with a novel aggregated codebook and transformers with global attention. In this work, the VQVAE focus on feature extraction and reconstruction of images, and the transformers fit the manifold and locate anomalies in the latent space. Then, leveraging the generated encoding sequences that conform to a normal distribution, we can reconstruct a more accurate image for locating the anomalies. Experiments on various datasets demonstrate the effectiveness of the proposed method.
3M3D: Multi-View, Multi-Path, Multi-Representation for 3D Object Detection

3D visual perception tasks based on multi-camera images are essential for autonomous driving systems. The latest work in this field performs 3D object detection by leveraging multi-view images as an input and iteratively enhancing object queries (object proposals) by cross-attending multi-view features. However, individual backbone features are not updated with multi-view features, and it stays as a mere collection of the output of the single-image backbone network. Therefore we propose 3M3D: A Multi-view, Multi-path, Multi-representation for 3D Object Detection where we update both multi-view features and query features to enhance the representation of the scene in both fine panoramic view and coarse global view. Firstly, we update multi-view features by multi-view axis self-attention. It will incorporate panoramic information in the multi-view features and enhance understanding of the global scene. Secondly, we update multi-view features by self-attention of the Region of Interest (ROI) windows which encodes local finer details in the features. It will help exchange the information not only along the multi-view axis but also along the other spatial dimension. Lastly, we leverage the fact of the multi-representation of queries (MRQ) in different domains to further boost performance. Here we use sparse floating queries along with dense Bird’s Eye View (BEV) queries, which are later post-processed to filter duplicate detections. Moreover, we show performance improvements on the nuScenes benchmark dataset [1] on top of our baselines.
Leveraging Optical Flow Features for Higher Generalization Power in Video Object Segmentation

We propose to leverage optical flow features for higher generalization power in semi-supervised video object segmentation. Optical flow is usually exploited as additional guidance information in many computer vision tasks. However, its relevance in video object segmentation was mainly in unsupervised settings or using the optical flow to warp or refine the previously predicted masks. Different from the latter, we propose to directly leverage the optical flow features in the target representation. We show that this enriched representation improves the encoder-decoder approach to the segmentation task. A model to extract the combined information from the optical flow and the image is proposed, which is then used as input to the target model and the decoder network. Unlike previous methods, e.g. in tracking where concatenation is used to integrate information from image data and optical flow, a simple yet effective attention mechanism is exploited in our work. Experiments on DAVIS 2017 and YouTube-VOS 2019 show that integrating the information extracted from optical flow into the original image branch results in a strong performance gain, especially in unseen classes which demonstrates its higher generalization power.
Team DETR: Guide Queries as a Professional Team in Detection Transformers

Recent proposed DETR variants have made tremendous progress in various scenarios due to their streamlined processes and remarkable performance. However, the learned queries usually explore the global context to generate the final set prediction, resulting in redundant burdens and unfaithful results. More specifically, a query is commonly responsible for objects of different scales and positions, which is a challenge for the query itself, and will cause spatial resource competition among queries. To alleviate this issue, we propose Team DETR, which leverages query collaboration and position constraints to embrace objects of interest more precisely. We also dynamically cater to each query member’s prediction preference, offering the query better scale and spatial priors. In addition, the proposed Team DETR is flexible enough to be adapted to other existing DETR variants without increasing parameters and calculations. Extensive experiments on the COCO dataset showcase that Team DETR achieves remarkable gains, especially for small and large objects. Code is available at https://github.com/horrible-dong/TeamDETR.
Past Information Aggregation for Multi-Person Tracking

Multi-person tracking is often solved with the tracking-by-detection (TBD) paradigm. So-far tracked targets are matched to new detections on the basis of the current track states including motion or appearance information. If tracks are updated with inaccurate detections or unreliable appearance features, e.g., due to occlusion, the track states can become distorted leading to errors in the association. To mitigate this problem, we propose a simple and generic method for Past Information Aggregation (PIA) that utilizes more tracking information from the past in the association and can be applied within any TBD approach. We combine PIA with a new sophisticated distance measure fusing motion and appearance cues and a second matching stage which further improves the association accuracy. Our tracking framework is analyzed with extensive ablative experiments and state-of-the-art results are achieved on the MOT17 and MOT20 benchmarks.
Cross-Inferential Networks for Source-Free Unsupervised Domain Adaptation

One central challenge in source-free unsupervised domain adaptation (UDA) is the lack of an effective approach to evaluate the prediction results of the adapted network model in the target domain. To address this challenge, we propose to explore a new method called cross-inferential networks (CIN). Our main idea is that, when we adapt the network model to predict the sample labels from encoded features, we use these prediction results to construct new training samples with derived labels to learn a new examiner network that performs a different but compatible task in the target domain. Specifically, in this work, the base network model is performing image classification while the examiner network is tasked to perform relative ordering of triplets of samples whose training labels are carefully constructed from the prediction results of the base network model. Two similarity measures, cross-network correlation matrix similarity and attention consistency, are then developed to provide important guidance for the UDA process. Our experimental results on benchmark datasets demonstrate that our proposed CIN approach can significantly improve the performance of source-free UDA.
ST-MFNET Mini: Knowledge Distillation-Driven Frame Interpolation

Currently, one of the major challenges in deep learning-based video frame interpolation (VFI) is the large model size and high computational complexity associated with many high performance VFI approaches. In this paper, we present a distillation-based two-stage workflow for obtaining compressed VFI models which perform competitively compared to the state of the art, but with significantly reduced model size and complexity. Specifically, an optimization-based network pruning method is applied to a state of the art frame interpolation model, ST-MFNet, which suffers from large model size. The resulting network architecture achieves a 91% reduction in parameter numbers and a 35% increase in speed. The performance of the new network is further enhanced through a teacher-student knowledge distillation training process using a Laplacian distillation loss. The final low complexity model, ST-MFNet Mini, achieves a comparable performance to most existing high-complexity VFI methods, only outperformed by the original ST-MFNet. Our source code is available at https://github.com/crispianm/ST-MFNet-Mini
Efficient-HDRTV: Efficient SDR to HDR Conversion for HDR TV

The existing deep neural network (DNN) based SDR (Standard dynamic range) to HDR (High dynamic range) conversion methods outperform conventional methods, but they are either too large to implement on a device or with quantization artifacts generated on smooth regions on an image. We propose an efficient neural network for the SDR to HDR conversion, namely "Efficient-HDRTV". It consists of two efficient structures GIM (Global Inverse Mapping) and LIM (Local Inverse Mapping). The key features of GIM and LIM use the small series of a basis function with its coefficient function, which are implemented using small number of convolutions, logarithm and exponential functions. They are combined with other convolutional layers so that the entire network can be jointly trained for learning inverse tone, enhanced details and expanded color gamut from SDR to HDR. Thanks to the GIM and LIM, we can keep our network small with good performance. Our experimental results show that Efficient-HDRTV is much lighter but performs better than the state of the arts.
When Visible-to-Thermal Facial GAN Beats Conditional Diffusion

Thermal facial imagery offers valuable insight into physiological states such as inflammation and stress by detecting emitted radiation in the infrared spectrum, which is unseen in the visible spectra. Telemedicine applications could benefit from thermal imagery, but conventional computers are reliant on RGB cameras and lack thermal sensors. As a result, we propose the Visible-to-Thermal Facial GAN (VTF-GAN) that is specifically designed to generate high-resolution thermal faces by learning both the spatial and frequency domains of facial regions, across spectra. We compare VTF-GAN against several popular GAN baselines and the first conditional DDPM for VT face translation (VTF-Diff). Results show that VTF-GAN achieves high quality, crisp, and perceptually realistic thermal faces using a combined set of patch, temperature, perceptual, and Fourier Transform losses, compared to all baselines including diffusion.
Imposing Total Variation Prior Into Guided Filter

Guided filter is a popular filter thanks to its effectiveness in edge preserving and computation efficiency. On the other hand, total variation is a popular prior thanks to its mathematical properties. In this paper, we impose the total variation prior into guided filter, leading to a novel Total Variation Guided Filter (TVGF) that has both advantages from guided filter and total variation prior. First, TVGF has a closed-form expression and also a linear computation complexity. Our experiment confirms that TVGF is 1300+ times faster than traditional iterative solvers for total variation models. Second, thanks to the total variation, TVGF preserves edges better than the original guided filter and leads to much less artifacts. TVGF is also two times faster than the guided filter. We theoretically derive the TVGF and numerically confirm its effectiveness and efficiency. TVGF can be used in a large range of edge-aware applications, such as image smoothing, dehazing, depth estimation and optical flow estimation.
Interpretable Visual Question Answering Referring to Outside Knowledge

We present a novel multimodal interpretable VQA model that can answer the question more accurately and generate diverse explanations. Although researchers have proposed several methods that can generate human-readable and fine-grained natural language sentences to explain a model’s decision, these methods have focused solely on the information in the image. Ideally, the model should refer to various information inside and outside the image to correctly generate explanations, just as we use background knowledge daily. The proposed method incorporates information from outside knowledge and multiple image captions to increase the diversity of information available to the model. The contribution of this paper is to construct an interpretable visual question answering model using multimodal inputs to improve the rationality of generated results. Experimental results show that our model can outperform state-of-the-art methods regarding answer accuracy and explanation rationality.
Rethinking Long-Tailed Visual Recognition with Dynamic Probability Smoothing and Frequency Weighted Focusing

Deep learning models trained on long-tailed (LT) datasets often exhibit bias towards head classes with high frequency. This paper highlights the limitations of existing solutions that combine class- and instance-level re-weighting loss in a naive manner. Specifically, we demonstrate that such solutions result in overfitting the training set, significantly impacting the rare classes. To address this issue, we propose a novel loss function that dynamically reduces the influence of outliers and assigns class-dependent focusing parameters. We also introduce a new long-tailed dataset, ICText-LT, featuring various image qualities and greater realism than artificially sampled datasets. Our method has proven effective, outperforming existing methods through superior quantitative results on CIFAR-LT, Tiny ImageNet-LT, and our new ICText-LT datasets. The source code and new dataset are available at https://github.com/nwjun/FFDS-Loss.
Fine-Tuning Method of the GA-based Collision Avoidance System for AUVs

This article presents a fine-tuning methodology for an anti-collision system based on local path planning using Genetic Algorithm (GA). The anti-collision system has been implemented using the mathematical model of Autonomous Underwater Vehicle (AUV). Based on a map of the underwater environment with a high degree of complexity, depending on the input parameters of the simulation, the average number of collisions and the average deviation from the set trajectory were tested. The fine-tuning aimed to minimise the abovementioned parameters and determine the set of input parameters that would provide the best results. Fine-tuning was performed based on an exhaustive search method within specified limits. Compared to the average deviation value and the number of collisions calculated based on 45000 trials (all combinations of input parameters started 15 times) for each of the three selected sets, better results were obtained, which proves the purposefulness of this type of fine-tuning. Additionally, based on the determined impact of input parameter values change on simulations results, when fine-tuning the collision avoidance system, it is more effective to adjust the parameters related to the vehicle’s maneuvres first than the parameters related to the path planning method.