Reviewerly/exHarmony huggingface dataset:
Imbalance-Aware Adaptive Margin Loss for Fair Multi-Label Face Attribute Recognition

This paper proposes a novel metric learning method for fair face attribute recognition. Datasets with imbalanced sample distributions can spoil the fair discrimination ability of deep learning models. This problem becomes more complex in multi-label classification tasks due to the variety of imbalance levels for each label. To tackle this problem, we propose Imbalance-Aware Adaptive Margin (IAAM). IAAM utilizes prior information on data imbalance with new parameters that control the relative margins between classes for each label. Specifically, it adaptively tunes margins in the metric learning loss to give larger margins for minority classes and smaller margins for majority classes. The proposed method is a flexible reformulation of previous studies for multi-label classification tasks. We performed experiments on the Celeb-A and LFWA datasets and evaluated with fair classification metrics. Our findings showed that the IAAM led to excellent performance on fair face attribute recognition both in the weakly imbalanced labels and strongly imbalanced labels.
Fighting Over-Fitting with Quantization for Learning Deep Neural Networks on Noisy Labels

The rising performance of deep neural networks is often empirically attributed to an increase in the available computational power, which allows complex models to be trained upon large amounts of annotated data. However, increased model complexity leads to costly deployment of modern neural networks, while gathering such amounts of data requires huge costs to avoid label noise. In this work, we study the ability of compression methods to tackle both of these problems at once. We hypothesize that quantization-aware training, by restricting the expressivity of neural networks, behaves as a regularization. Thus, it may help fighting overfitting on noisy data while also allowing for the compression of the model at inference. We first validate this claim on a controlled test with manually introduced label noise. Furthermore, we also test the proposed method on Facial Action Unit detection, where labels are typically noisy due to the subtlety of the task. In all cases, our results suggests that quantization significantly improve the results compared with existing baselines, regularization as well as other compression methods.
LLA-Flow: A Lightweight Local Aggregation on Cost Volume for Optical Flow Estimation

Lack of texture often causes ambiguity in matching, and handling this issue is an important challenge in optical flow estimation. Some methods insert stacked transformer modules that allow the network to use global information of cost volume for estimation. But the global information aggregation often incurs serious memory and time costs during training and inference, which hinders model deployment. We draw inspiration from the traditional local region constraint and design the local similarity aggregation (LSA) and the shifted local similarity aggregation (SLSA). The aggregation for cost volume is implemented with lightweight modules that act on the feature maps. Experiments on the final pass of Sintel show the lower cost required for our approach while maintaining competitive performance.
Enabling High-Resolution Pose Estimation in Real Time Using Active Perception

Deep Learning (DL) models have enabled very accurate pose estimation. However, most of the existing approaches require images of relatively high resolution, since locating body parts and joints accurately is challenging, which increases the computational cost of these approaches. To overcome this limitation in this paper we propose an active perception method for high-resolution pose estimation that enables efficiently selecting the most appropriate image region for analysis and then employing a bottom-up pose estimator on the corresponding region. This allows for significantly improving the efficiency of pose estimation by selectively analyzing in high resolution only the parts of the image that contain humans. To ensure the computational efficiency of the proposed method we propose using low-resolution heat maps extracted using the same pose estimation model in order to guide the active perception process. The proposed method is model agnostic since it can be combined with any bottom-up pose estimation model in order to enable high-resolution analysis. We have experimentally evaluated the proposed method using a well-known pose estimation model, Lightweight OpenPose, demonstrating its effectiveness on three high-resolution variants of the COCO2017 dataset.
PL-UNeXt: Per-Stage Edge Detail and Line Feature Guided Segmentation for Power Line Detection

Power line detection is a critical inspection task for electricity companies and is also useful in avoiding drone obstacles. Accurately separating power lines from the surrounding area in the aerial image is still challenging due to the intricate background and low pixel ratio. In order to properly capture the guidance of the spatial edge detail prior and line features, we offer PL-UNeXt, a power line segmentation model with a booster training strategy. We design edge detail heads computing the loss in edge space to guide the lower-level detail learning and line feature heads generating auxiliary segmentation masks to supervise higher-level line feature learning. Benefited from this design, our model can reach 70.6 F1 score (+1.9%) on TTPLA and 68.41 mIoU (+5.2%) on VITL (without utilizing IR images), while preserving a real-time performance due to few inference parameters.
Unsupervised Deep Hashing With Deep Semantic Distillation

Many existing unsupervised hashing methods attempt to preserve as much semantic information as possible by reconstructing the input data. However, this approach can result in the hash code preserving a lot of redundant information. Besides, previous works usually adopt local structures to guide hashing learning, which will mislead hashing model due to a large amount of noise existing in the local structure. In this paper, we propose a novel Deep Semantic Distillation Hashing (DSDH) to solve the above problems. Specifically, to ensure that the hashing model focuses on preserving more discriminative information rather than background noise, we use random masked images as input for feature extraction. We then apply empirical Maximum Mean Discrepancy to match the output feature distribution with that of the original image. Additionally, to avoid misleading, we propose to constrain the consistency of the similarity structures of the two spaces from the perspective of global distribution, thus transferring the knowledge of the feature space to Hamming space. Experiments conducted on three benchmarks show the superiority of DSDH.
Machine-Attention-based Video Coding for Machines

Conventional video coding methods have been developed based on the human visual system (HVS). However, in recent years, video has occupied a huge portion of internet traffic, and the mount of video data for machine consumption has increased rapidly due to the progress of neural networks. This paper proposes a novel machine-attention-based video coding method for machines. Inspired by the saliency-driven research, we first extract attention regions, sensitively affecting the machine vision performance, from the object detection network. Subsequently, a maximum a posterior (MAP)-based bit allocation method is applied to assign more bits to the attention regions. Our proposed method helps to maintain high machine vision performance whereas reducing the bitrate. Experimental results show that our proposed method achieves up to 34.89% bjøntegaard delta (BD)-rate reduction for the video dataset and up to 44.70% BD-rate reduction for the image dataset compared to state-of-the-art video coding technology.
Feature Space Data Augmentation for Viewpoint-Robust Action Recognition in Videos

The ongoing research on human action recognition models is achieving very promising results, and the existing models reach very high performances. However, they still suffer from one major challenge: their performance decreases on viewpoints not seen in the training step of the model. In this paper, we introduce a new approach based on virtual viewpoint augmentation in the feature space to increase the robustness of the action recognition models to different camera viewpoints. This approach was evaluated on two action recognition datasets: DAHLIA and Toyota SmartHome. Our model shows promising results, with a significant performance increase on both datasets for viewpoints not seen during the training step.
Channel-separation-based Network for Object Detection under Foggy Conditions

Vision plays a key role in enhancing environmental awareness in several applications. However, under adverse weather conditions (particularly foggy conditions), it is difficult to locate objects from the captured low-quality images. Most existing methods attempt to restore high-quality images from the low-quality ones, which increases system complexity and results in the loss of the latent information of the images. In this study, a channel-separation-based detection network is proposed to preserve latent information. In particular, a fog filter is used to perform pruning during image processing to maintain the latent information of the images. By replacing the deep feature extraction layer with a plug-and-play block (MBConvBlock) and using a new CSPBottleNeck combined with CrossConv in the feature pyramid network, our model can overcome the disadvantages of convolutional neural network with fixed and global receptive fields and focus on more crucial object features. The model was trained jointly using an end-to-end method and hybrid data, thereby enhancing the generalization ability of the model network. The results indicate that our model achieves outstanding performance under both normal and real-world foggy conditions with an mAP of 74.40% and 42.10%, respectively.
Edge Synthesis Block: A Building Unit for Real-Time Single Image Super Resolution

This paper proposes a novel edge synthesis block as the basic building unit of model architectures for single image super resolution tasks. The edge synthesis block consists of two parallel branches of convolution layers designed to enhance the horizontal and vertical edges in an image. These building units can be stacked flexibly to build complex or light models depending on the desirable latency vs visual quality trade-off. The proposed building block has been tested in mobile phones for different image super resolution tasks such as pinch-zooming of gallery images, and image enhancement in a video call scenario. The edge-synthesis-block-based model architectures are not only very low complex in terms of multiply-accumulate (MAC) operations, but also are comparable to the state-of-the-art super resolution techniques in terms of visual quality.
Multiple Agent Path Planning for Autonomous Area Monitoring

This paper considers informative path planning for two collaborative autonomous agents tasked with monitoring an area against intruders. The area is decomposed into discrete cells that hold the intruder probability distribution and also form the a priori known map over which the agent moves. A computationally efficient intruder model is proposed and the optimal path is formulated in terms of detection probability. A control strategy based on the suboptimal but computationally feasible receding horizon optimization technique is described. The planning is tested in MATLAB on two experimental maps with 108 and 811 cells, respectively. It is concluded that the agents perform rationally and that two agents may almost half the search time compared to a single agent. Real-time performance on ordinary hardware with dedicated software is deemed feasible.
Object-Centric Video Prediction Via Decoupling of Object Dynamics and Interactions

We present a framework for object-centric video prediction, i.e., parsing a video sequence into objects, and modeling their dynamics and interactions in order to predict the future object states from which video frames are rendered. To facilitate the learning of meaningful spatio-temporal object representations and forecasting of their states, we propose two novel object-centric video prediction (OCVP) transformer modules, which decouple the processing of temporal dynamics and object interactions. We show how OCVP predictors outperform object-agnostic video prediction models on two different datasets. Furthermore, we observe that OCVP modules learn consistent and interpretable object representations. Animations and code to reproduce our results can be found in our project website <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> .
Residential Extraction Based on Weakly-Supervised Similarity-Aware Multi-Source Alignment Strategy with Limited SAR Data

Residential extraction based on deep learning approach is a significant task in Synthetic Aperture Radar (SAR) image processing. However, extremely limited SAR data brings great challenges to the data-driven method: 1) pixel-wise annotations are hard to obtain due to the expensive cost; 2) There is not any universal large-scale SAR image dataset with heterogeneous SAR images. In this paper, a novel residential extraction method based on similarity-aware multi-source alignment strategy is proposed to solve such problems. Firstly, we propose a weakly-supervised Multi-source Similarity-aware Extraction Network (MSENet) to preserve the context dependency of pixels and improve the integrity of the extraction. Then, to tackle with the lack of training samples, a multi-source knowledge alignment strategy is proposed to learn transferrable knowledge from heterogeneous SAR datasets. Finally, affinity-guided optimization is introduced to refine the coarse maps with clear boundaries. Comprehensive experiments demonstrate the efficiency of our method.
Pseudo Labels Refinement with Intra-Camera Similarity for Unsupervised Person Re-Identification

Unsupervised person re-identification (Re-ID) aims to retrieve person images across cameras without any identity labels. Most clustering-based methods roughly divide image features into clusters and neglect the feature distribution noise caused by domain shifts among different cameras, leading to inevitable performance degradation. To address this challenge, we propose a novel label refinement framework with clustering intra-camera similarity. Intra-camera feature distribution pays more attention to the appearance of pedestrians and labels are more reliable. We conduct intra-camera training to get local clusters in each camera, respectively, and refine inter-camera clusters with local results. We hence train the Re-ID model with refined reliable pseudo labels in a self-paced way. Extensive experiments demonstrate that the proposed method surpasses state-of-the-art performance. Code is available at https://github.com/leeBooMla/ICSR.
Adding Distance Information to Self-Supervised Learning for rich Representations

While contrastive learning has been the mainstream in self-supervised learning, negative sampling for learning is a challenging issue. To resolve this problem, recent studies instead used a pair of augmented images generated from each image as positive samples and trained models by maximizing the similarity between representation vectors. However, training models using positive samples may not be enough to let the models learn rich representations because these methods cannot consider the information from different images. To address the issue, we propose a learning method that uses clustered positive samples from different images to allow the network to learn better. As the first step, our method learns invariance representations from augmented images. Next, the resulting representation vectors are clustered to learn rich representations. According to our evaluation, it was found that our method achieved better performance than other models using a clustering method from scratch. Moreover, in downstream tasks, our model achieved a higher classification accuracy than Barlow Twins under the same experimental setting. Therefore, adding the distance information on the class centroid to the loss function of self-supervised learning can improve performance by exploiting rich representations in the data.
Adaptive and Robust Mmwave-Based 3D Human Mesh Estimation for Diverse Poses

This paper proposes a three-dimensional (3D) human mesh estimation framework with only a single commercial portable millimeter-wave device. Perceiving a 3D human mesh that includes poses and body shapes of a person with such a simple setting has remarkable potential for various applications such as daily activity monitoring and motion analysis for sports enhancement. Due to estimation difficulties, given a noisy input that includes signals reflected from the other person or objects in addition to the target person, most existing studies implicitly assume that the person stands almost vertically or that there is nothing else to be observed. Since such situations are unlikely to occur in real life, there is an urgent need for a more practical method. Therefore, we propose a framework that has the ability to extract only signals reflected from the target person and obtain local features that flexibly fit various poses that humans can take, including horizontal postures, such as lying. Our experiments suggest that the framework works effectively and that our method outperforms the baseline method.
Prnet: A Progressive Regression Network for No-Reference User-Generated-Content (UGC) Video Quality Assessment

Objectively perceptual quality assessment of UGC videos is a challenge due to many reasons. First, the sources of UGC-videos are not available, which makes the appropriate technique is the no-reference video quality assessment (NR-VQA). Second, subjective mean option scores (MOS) of all the UGC-datasets are not uniformly distributed. Third, authentic degradations occurred in the videos are not unique, therefore, not predicable. For example, an over- or underexposure video, brightness and contrast static information is critical for evaluation. Only employing verified priori statistic knowledge or generalized learning knowledge may not cover all possible distortions. To solve these problems, we introduce a framework—Progressive Regression Network (PRNet) in this paper. For the MOS problem, a PR module is proposed, which utilizes the coarse-to-fine strategy during the training process. It can turn sparse subjective human rating scores into integers with denser samples to address the problem. For the unpredictable distortions, a wide and deep feature extraction module is developed, which employs both low-level features generated from natural scene statistics (NSS) and high-level semantic features extracted by deep neural networks, to fuse memorizing priori knowledge and generalizing learning features. Experiments demonstrate that our PRNet achieves top performance in three main datasets.
Teacher-Student Network for Real-World Face Super-Resolution with Progressive Embedding of Edge Information

Traditional face super-resolution (FSR) methods trained on synthetic datasets usually have poor generalization ability for real-world face images. Recent work has utilized complex degradation models or training networks to simulate the real degradation process, but this limits the performance of these methods due to the domain differences that still exist between the generated low-resolution images and the real low-resolution images. Moreover, because of the existence of a domain gap, the semantic feature information of the target domain may be affected when synthetic data and real data are utilized to train super-resolution models simultaneously. In this study, a real-world face super-resolution teacher-student model is proposed, which considers the domain gap between real and synthetic data and progressively includes diverse edge information by using the recurrent network’s intermediate outputs. Extensive experiments demonstrate that our proposed approach surpasses state-of-the-art methods in obtaining high-quality face images for real-world FSR.
Spatially-Adaptive Learning-Based Image Compression with Hierarchical Multi-Scale Latent Spaces

Adaptive block partitioning is responsible for large gains in current image and video compression systems. This method is able to compress large stationary image areas with only a few symbols, while maintaining a high level of quality in more detailed areas. Current state-of-the-art neural-network-based image compression systems however use only one scale to transmit the latent space. In previous publications, we proposed RDONet, a scheme to transmit the latent space in multiple spatial resolutions. Following this principle, we extend a state-of-the-art compression network by a second hierarchical latent-space level to enable multi-scale processing. We extend the existing rate variability capabilities of RDONet by a gain unit. With that we are able to outperform an equivalent traditional autoencoder by 7% rate savings. Furthermore, we show that even though we add an additional latent space, the complexity only increases marginally and the decoding time can potentially even be decreased.
OCVOS: Object-Centric Representation for Video Object Segmentation

Semi-supervised video object segmentation (VOS) methods aim to segment target objects with the help of pixel-level annotations in the first frame. Many methods employ Transformer-based attention modules to propagate the given annotations in the first frame to the most similar patch or pixel in the following frames. Although they have shown impressive results, they can still be prone to errors in challenging scenes with multiple overlapping objects. To tackle this problem, we propose an object-centric VOS (OCVOS) method that exploits query-based Transformer decoder blocks. After aggregating target object information with typical matching-based approaches, the Transformer networks extract object-wise information by interacting with object queries. In this way, the proposed method considers not only global and contextual information but also object-centric representations. We validate its effectiveness in inducing object-wise information compared to existing methods on the DAVIS and YouTube-VOS benchmarks.
MSV-RGNN: Multiscale Voxel Graph Neural Network for 3D Object Detection

This paper proposes a two-stage 3D object detection framework, multiscale voxel graph neural network (MSV-RGNN) which aims to fully exploit multiple scale graph features by establishing global and local relationships between voxel features at different 3D convolutional neural network (CNN) layers. In contrast to conventional graph-based methods, our proposed multiscale-voxel-graph region-of-interest (RoI) pooling module constructs graphs across diverse voxel resolutions to obtain geometric structure information on voxel features. Initially, our multiscale-voxel-graph RoI pooling module sample voxel center points with voxel-wise feature vectors and 3D region proposals from backbone network. Subsequently, graphs are constructed at different scales and graph features are aggregated for second-stage refinement. The experimental results demonstrate the potential of using multiscale graphs across different voxel resolutions for 3D object detection, achieving decent experimental results with state-of-the-art methods.