Reviewerly/exHarmony huggingface dataset:
Deceiving Airborne Object Detectors Using Adversarial AI

Along with the dominant and security critical applications of Artificial Intelligence (AI), existence of its perils cannot be undermined. However, such threats become a blessing in disguise in scenarios where vulnerable assets are required to be hidden from autonomous systems including drones and UAVs. The hazards of Adversarial attacks on deep neural network (DNN) models are well known by research community. The intent of these attacks is to cause the DNN model to make inaccurate predictions or judgements. The aforementioned attacks signify a fundamental change in security-sensitive situations within practical contexts, wherein it becomes imperative to conceal vulnerable assets from self-governing detection mechanisms present on drones and unmanned aerial vehicles (UAVs). This paper presents a unique research in utilizing adversarial attacks to deceive autonomous detection systems onboard aerial reconnaissance, detection and surveillance systems. We provide a framework to deceive YOLOv3 object detection model through patch based attack by camouflaging a parked aircraft from overhead imaging. Our study successfully demonstrated patches when overlaid on minimal area of aircraft, was able to blind detection system with much greater accuracy. This work also aims to highlight the use of such methods in security related applications.
Assessing the Impact of Diversity on the Resilience of Deep Learning Ensembles: A Comparative Study on Model Architecture, Output, Activation, and Attribution

We investigate the relationship between different diversity metrics, accuracy, and resiliency to natural image corruptions of Deep Learning (DL) image classifier ensembles. We evaluate existing diversity dimensions such as model architecture, model prediction, and neuron activations, as well as a novel diversity dimension of input attribution.Using ResNet50 as a comparison baseline, we evaluate the resiliency of multiple individual DL model architectures against dataset distribution shifts corresponding to natural image corruptions. We compare ensembles created with diverse model architectures trained either independently or through a Neural Architecture Search technique and evaluate the correlation of prediction-based and attribution-based diversity to the final ensemble accuracy.Finally, we evaluate a set of diversity enforcement heuristics for training based on negative correlation learning (NCL) and compare how effective they are to achieve independent failure behavior.Our key observations are: 1) model architecture is more important for individual resiliency than model size or model accuracy but architecture diversity in an ensemble is typically not more resilient, 2) attribution-based diversity is less negatively correlated to the ensemble accuracy than prediction-based diversity, 3) a balanced loss function of individual and ensemble accuracy creates more resilient ensembles for image natural corruptions, 4) architecture diversity produces more diversity than NCL in all explored diversity metrics: predictions, attributions, and activations.
Lecture 8—Networked VR

Networked VR systems represent the systems that power distributed VE over various types of computer networks, including Local Area Networks (LAN) and Wide Area Networks (WAN), such as the Internet.
MSViT: Dynamic Mixed-scale Tokenization for Vision Transformers

The input tokens to Vision Transformers carry little semantic meaning as they are defined as regular equal-sized patches of the input image, regardless of its content. However, processing uniform background areas of an image should not necessitate as much compute as dense, cluttered areas. To address this issue, we propose a dynamic mixed-scale tokenization scheme for ViT, MSViT. Our method introduces a conditional gating mechanism that selects the optimal token scale for every image region, such that the number of tokens is dynamically determined per input. In addition, to enhance the conditional behavior of the gate during training, we introduce a novel generalization of the batch-shaping loss. We show that our gating module is able to learn meaningful semantics despite operating locally at the coarse patch-level. The proposed gating module is lightweight, agnostic to the choice of transformer backbone, and trained within a few epochs with little training overhead. Furthermore, in contrast to token pruning, MSViT does not lose information about the input, thus can be readily applied for dense tasks. We validate MSViT on the tasks of classification and segmentation where it leads to improved accuracy-complexity trade-off.
Uncertainty-Confidence Fused Pseudo-labeling for Graph Neural Networks

Graph Neural Networks (GNNs) have achieved promising performance for semi-supervised graph learning. However, the training of GNNs usually heavily relies on a large number of labeled nodes in a graph. When the labeled data are scarce, GNNs easily over-fit rto the few labeled samples, resulting in the degenerating performance. To address this issue, we propose a novel graph pseudo-labeling framework. The proposed framework combines both the predication confidence and approximate Bayesian uncertainty of GNNs, resulting in a metric for generating more reliable and balanced pseudo-labeled nodes in graph. Furthermore, an iterative re-training strategy is employed on the extended training label set (including original labeled and pseudo-labeled nodes) to train a more generalized GNNs. Extensive experiments on benchmark graph datasets demonstrate that the proposed pseudo-labeling framework can enhance node classification performance of two alternative GNNs models by a considerable margin, specifically when labeled data are scarce.
Fair Robust Active Learning by Joint Inconsistency

We introduce a new learning framework, Fair Robust Active Learning (FRAL), generalizing conventional active learning to fair and adversarial robust scenarios. This framework enables us to achieve fair-performance and fairrobustness with limited labeled data, which is essential for various annotation-expensive visual applications with safety-critical needs. However, existing fairness-aware data selection strategies face two challenges when applied to the FRAL framework: they are either ineffective under severe data imbalance or inefficient due to huge computations of adversarial training. To address these issues, we develop a novel Joint INconsistency (JIN) method that exploits prediction inconsistencies between benign and adversarial inputs and between standard and robust models. By leveraging these two types of easy-to-compute inconsistencies simultaneously, JIN can identify valuable samples that contribute more to fairness gains and class imbalance mitigation in both standard and adversarial robust settings. Extensive experiments on diverse datasets and sensitive groups demonstrate that our approach outperforms existing active data selection baselines, achieving fair-performance and fairrobustness under white-box PGD attacks.
LATrans-Unet: Improving CNN-Transformer with Location Adaptive for Medical Image Segmentation

Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) have been widely employed in medical image segmentation. While CNNs excel in local feature encoding, their ability to capture long-range dependencies is limited. In contrast, ViTs have strong global modeling capabilities. However, existing attention-based ViT models face difficulties in adaptively preserving accurate location information, rendering them unable to handle variations in important information within medical images. To inherit the merits of CNN and ViT while avoiding their respective limitations, we propose a novel framework called LATrans-Unet. By comprehensively enhancing the representation of information in both shallow and deep levels, LATrans-Unet maximizes the integration of location information and contextual details. In the shallow levels, based on a skip connection called SimAM-skip, we emphasize information boundaries and bridge the encoder-decoder semantic gap. Additionally, to capture organ shape and location variations in medical images, we propose Location-Adaptive Attention in the deep levels. It enables accurate segmentation by guiding the model to track changes globally and adaptively. Extensive experiments on multi-organ and cardiac segmentation tasks validate the superior performance of LATrans-Unet compared to previous state-of-the-art methods. The codes and trained models will be available soon.
An Effective Visible-Infrared Person Re-identification Network Based on Second-Order Attention and Mixed Intermediate Modality

Visible-infrared person re-identification (VI-ReID) is a challenging cross-modality pedestrian retrieval problem. Due to the significant cross-modality discrepancy, it is difficult to learn discriminative features. Attention-based methods have been widely utilized to extract discriminative features for VI-ReID. However, the existing methods are confined by first-order structures that just exploit simple and coarse information. The existing approach lacks the sufficient capability to learn both modality-irrelevant and modality-relevant features. In this paper, we extract the second-order information from mid-level features to complement the first-order cues. Specifically, we design a flexible second-order module, which considers the correlations between the common features and learns refined feature representations for pedestrian images. Additionally, the visible and infrared modality has a significant gap. Therefore, we propose a plug-and-play mixed intermediate modality module to generate intermediate modality representations to reduce the modality discrepancy between the visible and infrared features. Extensive experimental results on two challenging datasets SYSU-MM01 and RegDB demonstrate that our method considerably achieves competitive performance compared to the state-of-the-art methods.
Can Unstructured Pruning Reduce the Depth in Deep Neural Networks?

Pruning is a widely used technique for reducing the size of deep neural networks while maintaining their performance. However, such a technique, despite being able to massively compress deep models, is hardly able to remove entire layers from a model (even when structured): is this an addressable task? In this study, we introduce EGP, an innovative Entropy Guided Pruning algorithm aimed at reducing the size of deep neural networks while preserving their performance. The key focus of EGP is to prioritize pruning connections in layers with low entropy, ultimately leading to their complete removal. Through extensive experiments conducted on popular models like ResNet-18 and Swin-T, our findings demonstrate that EGP effectively compresses deep neural networks while maintaining competitive performance levels. Our results not only shed light on the underlying mechanism behind the advantages of unstructured pruning, but also pave the way for further investigations into the intricate relationship between entropy, pruning techniques, and deep learning performance. The EGP algorithm and its insights hold great promise for advancing the field of network compression and optimization.
DeCAB: Debiased Semi-supervised Learning for Imbalanced Open-Set Data

Semi-supervised learning (SSL) has received significant attention due to its ability to use limited labeled data and various unlabeled data to train models with high generalization performance. However, the assumption of a balanced class distribution in traditional SSL approaches limits a wide range of real applications, where the training data exhibits long-tailed distributions. As a consequence, the model is biased towards head classes and disregards tail classes, thereby leading to severe class-aware bias. Additionally, since the unlabeled data may contain out-of-distribution (OOD) samples without manual filtering, the model will be inclined to assign OOD samples to non-tail classes with high confidence, which further overwhelms the tail classes. To alleviate this class-aware bias, we propose an end-to-end semi-supervised method Debias Class-Aware Bias (DeCAB). DeCAB introduces positive-pair scores for contrastive learning instead of positive-negative pairs based on unreliable pseudo-labels, avoiding false negative pairs negatively impacts the feature space. At the same time, DeCAB utilizes class-aware thresholds to select more tail samples and selective sample reweighting for feature learning, preventing OOD samples from being misclassified as head classes and accelerating the convergence speed of the model. Experimental results demonstrate that DeCAB is robust in various semi-supervised benchmarks and achieves state-of-the-art performance. Our code is temporarily available at https://github.com/xlhuang132/decab .
Lecture 1—Introduction

During this lecture, we are going to first ponder about the nature of Virtual reality trying to determine a suitable definition of this concept. We will discuss some of the defining properties of Virtual Environments (VE) and Virtual Reality (VR), which set them apart from other similar media such as 3D video for example. Special attention will be devoted to concepts of immersion and interactivity, two very important terms, essential for understanding VR. Finally, we are going to discuss the general properties shared by all VR systems.
LightNet: Generative Model for Enhancement of Low-Light Images

In this work, we propose a generative model for enhancement of images captured in low-light conditions. Sensor constraints and inappropriate lighting conditions are accountable for degradations introduced in the image. The degradations limit the visibility of the scene and impedes vision in applications like detection, tracking and surveillance. Recently, deep learning algorithms have taken a leap for enhancement of images captured in low-light conditions. However, these algorithms fail to capture information on fine grained local structures and limit the performance. Towards this, we propose a generative model for enhancement of low-lit images to exploit both local and global information, and term it as LightNet. In proposed architecture LightNet, we include a hierarchical generator encompassing encoder-decoder module to capture global information and a patch discriminator to capture fine grained local information. Typically, the encoder-decoder module downsamples the low-lit image into distinct scales. Learning at distinct scales helps to capture both local and global features thereby suppressing the unwanted features (noise, blur). With this motivation, we downsample the captured low-lit image into 3 distinct scales. The decoder upsamples the encoded features at respective scales to generate an enhanced image. We demonstrate the results of proposed methodology on custom and benchmark datasets in comparison with SOTA methods using appropriate quantitative metrics.
Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritage

Cultural heritage applications and advanced machine learning models are creating a fruitful synergy to provide effective and accessible ways of interacting with artworks. Smart audio-guides, personalized art-related content and gamification approaches are just a few examples of how technology can be exploited to provide additional value to artists or exhibitions. Nonetheless, from a machine learning point of view, the amount of available artistic data is often not enough to train effective models. Off-the-shelf computer vision modules can still be exploited to some extent, yet a severe domain shift is present between art images and standard natural image datasets used to train such models. As a result, this can lead to degraded performance. This paper introduces a novel approach to address the challenges of limited annotated data and domain shifts in the cultural heritage domain. By leveraging generative vision-language models, we augment art datasets by generating diverse variations of artworks conditioned on their captions. This augmentation strategy enhances dataset diversity, bridging the gap between natural images and artworks, and improving the alignment of visual cues with knowledge from general-purpose datasets. The generated variations assist in training vision and language models with a deeper understanding of artistic characteristics and that are able to generate better captions with appropriate jargon.
Interactive Learning for Interpretable Visual Recognition via Semantic-Aware Self-Teaching Framework

Aimed at tracing back the decision-making process of deep neural networks on the fine-grained image recognition task, various interpretable methods have been proposed and obtained promising results. However, the existing methods still have limited interpretability in aligning the abstract semantic concepts with the concrete image regions, due to the lack of human guidance during the model training. Attempting to address this issue and inspired by the machine teaching techniques, we formulate the training process of interpretable methods as an interactive learning manner by concisely simulating the human learning mechanism. Specifically, we propose a semantic-aware self-teaching framework to progressively improve the given neural network through an interactive teacher-student learning protocol. After initialing from the well-trained parameters of the given model, the teacher model focuses on minimally providing informative image regions to train the student model to generate interpretable predictions (i.e., semantic image regions) as good feedback. These feedback can encourage the teacher model to further refine the alignment of semantic concepts and image regions. Besides, our proposed framework is compatible with most of the existing network architectures. Extensive and comprehensive comparisons with the existing state-of-the-art interpretable approaches on the public benchmarks demonstrate that our interactive learning manner showcases an improved interpretability, a higher classification accuracy, and a greater degree of generality.
Disentangled Attribute Features Vision Transformer for Pedestrian Attribute Recognition

Pedestrian attribute recognition, as a sub-task of multi-label classification, is influenced by the joint and coupled relationships among the numerous labels of the samples. This impact affects the model’s ability to judge labels and results in a loss of recognition accuracy for both overall and individual labels in unfamiliar scenes. In this paper, we propose a Disentangled Attribute Features Vision Transformer (DAF-ViT) model for pedestrian attribute recognition, aiming to enhance the discriminative ability of correlated features in pedestrian samples and suppress the discriminative ability of irrelevant features. The decoupling model consists of three parallel self-attention modules: a correlation self-attention module (CSA) for extracting correlated attribute features, a exclusive self-attention module (ESA) for extracting mutually exclusive attribute features, and a random sequence self-attention module (RSA) for extracting unrelated attribute features. The final predicted attributes are obtained by combining the results of these modules using a multi-level result fusion module. Compared to the baseline methods, our approach achieves a 2.8 $$\%$$ and 2.5 $$\%$$ increase in mA accuracy on the PETA and PA-100K datasets, respectively. This effectively improves the average accuracy of the model on various attributes.
Going Beyond Closed Sets: A Multimodal Perspective for Video Emotion Analysis

Emotion analysis plays a crucial role in understanding video content. Existing studies often approach it as a closed set classification task, which overlooks the important fact that the emotional experiences of humans are so complex and difficult to be adequately expressed in a limited number of categories. In this paper, we propose MM-VEMA, a novel MultiModal perspective for Video EMotion Analysis. We formulate the task as a crossmodal matching problem within a joint multimodal space of videos and emotional experiences (e.g. emotional words, phrases, sentences). By finding experiences that closely match each video in this space, we can derive the emotions evoked by the video in a more comprehensive manner. To construct this joint multimodal space, we introduce an efficient yet effective method that manipulates the multimodal space of a pre-trained vision-language model using a small set of emotional prompts. We conduct experiments and analyses to demonstrate the effectiveness of our methods. The results show that videos and emotional experiences are well aligned in the joint multimodal space. Our model also achieves state-of-the-art performance on three public datasets.
Saliency Driven Monocular Depth Estimation Based on Multi-scale Graph Convolutional Network

Monocular depth estimation is a fundamental and crucial task in computer vision that enables scene understanding from a single image. This paper proposes a novel approach for saliency-driven monocular depth estimation based on a multi-scale Graph Convolutional Network (GCN). Our method utilizes saliency information to guide the depth estimation process and employs a multi-scale GCN to capture local and global contextual cues. The proposed framework constructs a graph structure using RGB images to represent the relationships between image regions. We designed a multi-scale feature fusion module called DS Fusion, by applying GCN at multiple scales, our method effectively integrates depth features and saliency features to predict accurate depth maps. Extensive experiments conducted on KITTI and NYU datasets demonstrate the superior performance of our approach compared to state-of-the-art techniques. Additionally, we perform indepth analysis of the network architecture and discuss the impact of saliency cues on depth estimation accuracy. Our proposed method showcases the potential of combining saliency information and GCN in monocular depth estimation, contributing to the progress of scene understanding and depth perception from a single image.
Transformer-Based Multi-object Tracking in Unmanned Aerial Vehicles

Unmanned aerial vehicles (UAVs) possess a wide field of view, maneuverability, and autonomy in the field of multi-object tracking. By combining the high-altitude perspective and multidimensional perception capabilities of UAVs with artificial intelligence image processing and object recognition technologies, efficient and accurate multi-object tracking can be achieved. To enhance the processing capacity of relevant information, a transformer-based multi-object tracking model is proposed in UAV perspectives. The backbone network of detector is based on the transformer architecture to extract target features. In this structure, the network is further deepened to optimize feature fusion and improve the ability to capture small targets. To address the tracking challenges arising from dynamic variations between different targets, the tracker implements multi-object tracking in UAV scenarios based on the motion characteristics of targets and optimizes the tracking process accordingly. The model was evaluated on the Visdrone dataset, achieving a detection accuracy of 90.1% and a MOTA of 51.9%. Furthermore, the model demonstrated a remarkable speed of 20 frames per second. In summary, the proposed algorithm demonstrates excellent performance in target tracking from UAV perspectives and exhibits certain advantages over other algorithms.