Reviewerly/exHarmony huggingface dataset:
Adversarial Attacks Against Uncertainty Quantification

Machine-learning models can be fooled by adversarial examples, i.e., carefully-crafted input perturbations that force models to output wrong predictions. While uncertainty quantification has been recently proposed to detect adversarial inputs, under the assumption that such attacks exhibit a higher prediction uncertainty than pristine data, it has been shown that adaptive attacks specifically aimed at reducing also the uncertainty estimate can easily bypass this defense mechanism. In this work, we focus on a different adversarial scenario in which the attacker is still interested in manipulating the uncertainty estimate, but regardless of the correctness of the prediction; in particular, the goal is to undermine the use of machine-learning models when their outputs are consumed by a downstream module or by a human operator. Following such direction, we: (i) design a threat model for attacks targeting uncertainty quantification; (ii) devise different attack strategies on conceptually different UQ techniques spanning for both classification and semantic segmentation problems; (iii) conduct a first complete and extensive analysis to compare the differences between some of the most employed UQ approaches under attack. Our extensive experimental analysis shows that our attacks are more effective in manipulating uncertainty quantification measures than attacks aimed to also induce misclassifications.
Continual Learning with Deep Streaming Regularized Discriminant Analysis

Continual learning is increasingly sought after in real-world machine learning applications, as it enables learning in a more human-like manner. Conventional machine learning approaches fail to achieve this, as incrementally updating the model with non-identically distributed data leads to catastrophic forgetting, where existing representations are overwritten. Although traditional continual learning methods have mostly focused on batch learning, which involves learning from large collections of labeled data sequentially, this approach is not well-suited for real-world applications where we would like new data to be integrated directly. This necessitates a paradigm shift towards streaming learning. In this paper, we propose <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sup> a streaming version of regularized discriminant analysis as a solution to this challenge. We combine our algorithm with a convolutional neural network and demonstrate that it outperforms both batch learning and existing streaming learning algorithms on the ImageNet ILSVRC-2012 dataset.
Attending Generalizability in Course of Deep Fake Detection by Exploring Multi-task Learning

This work explores various ways of exploring multi-task learning (MTL) techniques aimed at classifying videos as original or manipulated in cross-manipulation scenario to attend generalizability in deep fake scenario. The dataset used in our evaluation is FaceForensics++, which features 1000 original videos manipulated by four different techniques, with a total of 5000 videos. We conduct extensive experiments on multi-task learning and contrastive techniques, which are well studied in literature for their generalization benefits. It can be concluded that the proposed detection model is quite generalized, i.e., accurately detects manipulation methods not encountered during training as compared to the state-of-the-art.
Actor-agnostic Multi-label Action Recognition with Multi-modal Query

Existing action recognition methods are typically actor-specific due to the intrinsic topological and apparent differences among the actors. This requires actor-specific pose estimation (e.g., humans vs. animals), leading to cumbersome model design complexity and high maintenance costs. Moreover, they often focus on learning the visual modality alone and single-label classification whilst neglecting other available information sources (e.g., class name text) and the concurrent occurrence of multiple actions. To overcome these limitations, we propose a new approach called ‘actor-agnostic multi-modal multi-label action recognition,’ which offers a unified solution for various types of actors, including humans and animals. We further formulate a novel Multi-modal Semantic Query Network (MSQNet) model in a transformer-based object detection framework (e.g., DETR), characterized by leveraging visual and textual modalities to represent the action classes better. The elimination of actor-specific model designs is a key advantage, as it removes the need for actor pose estimation altogether. Extensive experiments on five publicly available benchmarks show that our MSQNet consistently outperforms the prior arts of actor-specific alternatives on human and animal single- and multi-label action recognition tasks by up to 50%. Code is made available at https://github.com/mondalanindya/MSQNet.
LatentSwap3D: Semantic Edits on 3D Image GANs

3D GANs have the ability to generate latent codes for entire 3D volumes rather than only 2D images. These models offer desirable features like high-quality geometry and multi-view consistency, but, unlike their 2D counterparts, complex semantic image editing tasks for 3D GANs have only been partially explored. To address this problem, we propose LatentSwap3D, a semantic edit approach based on latent space discovery that can be used with any off-the-shelf 3D or 2D GAN model and on any dataset. LatentSwap3D relies on identifying the latent code dimensions corresponding to specific attributes by feature ranking using a random forest classifier. It then performs the edit by swapping the selected dimensions of the image being edited with the ones from an automatically selected reference image. Compared to other latent space control-based edit methods, which were mainly designed for 2D GANs, our method on 3D GANs provides remarkably consistent semantic edits in a disentangled manner and outperforms others both qualitatively and quantitatively. We show results on seven 3D GANs (π-GAN, GIRAFFE, StyleSDF, MVCGAN, EG3D, StyleNeRF, and VolumeGAN) and on five datasets (FFHQ, AFHQ, Cats, MetFaces, and CompCars).
Is there progress in activity progress prediction?

Activity progress prediction aims to estimate what percentage of an activity has been completed. Currently this is done with machine learning approaches, trained and evaluated on complicated and realistic video datasets. The videos in these datasets vary drastically in length and appearance. And some of the activities have unanticipated developments, making activity progression difficult to estimate. In this work, we examine the results obtained by existing progress prediction methods on these datasets. We find that current progress prediction methods seem not to extract useful visual information for the progress prediction task. Therefore, these methods fail to exceed simple frame-counting baselines. We design a precisely controlled dataset for activity progress prediction and on this synthetic dataset we show that the considered methods can make use of the visual information, when this directly relates to the progress prediction. We conclude that the progress prediction task is ill-posed on the currently used real-world datasets. Moreover, to fairly measure activity progression we advise to consider a, simple but effective, frame-counting baseline.
Flashback for Continual Learning

To strike a delicate balance between model stability and plasticity of continual learning, previous approaches have adopted strategies to guide model updates on new data to preserve old knowledge while implicitly absorbing new information through task objective function (e.g. classification loss). However, our goal is to achieve this balance more explicitly, proposing a bi-directional regularization that guides the model in preserving existing knowledge and actively absorbing new knowledge. To address this, we propose the Flashback Learning (FL) algorithm, a two-stage training approach that seamlessly integrates with diverse methods from different continual learning categories. FL creates two knowledge bases; one with high plasticity to control learning and one conservative to prevent forgetting, then it guides the model update using these two knowledge bases. FL significantly improves baseline methods on common image classification datasets such as CIFAR-10, CIFAR-100, and Tiny ImageNet in various settings.
PoseBias: On Dataset Bias and Task Difficulty - Is there an Optimal Camera Position for Facial Image Analysis?

Let's imagine you could choose the position of the camera for a particular face analysis task - where would you put it? In this work, we provide a first analysis based on synthetic training data to provide evidence that this choice is not trivial, not only dependent on the training data and different based on the task. We provide results for two major face analysis tasks, face recognition and landmark detection. For our experiments, we use a 3D Morphable Model as it provides us full control over pose, illumination, and identity to generate idealized training data. Whilst rendered images are not photorealistic we avoid any confounding factors and biases from other sources (e.g. pose bias in training data).Our results show that the optimal camera poses are near frontal but not exactly frontal and dependent on the task. By comparing the results obtained by pose-specific training set to a uniform training distribution without pose bias we show that the accuracy for both tasks not only depends on the bias in the training data but is actually dominated by the difficulty of the task depending on the particular pose.
Introspection of 2D Object Detection using Processed Neural Activation Patterns in Automated Driving Systems

While deep neural network (DNN) models have become extremely popular for object detection in automated driving systems (ADS), the dynamic and varied nature of the road traffic environment can still lead to model failures. To address this issue, researchers have recently explored introspection mechanisms, a.k.a, self-assessment, for monitoring the quality of perception in ADS. Subsequently, depending on the situation, these mechanisms can either hand over control to the human driver in SAE Level 3, or initiate a minimum risk maneuver in SAE Level 4 ADS. State-of-the-art introspection mechanisms for ADS train a neural network to learn the relationship between the raw neural activation patterns of the underlying DNN-based perception function per frame and the calculated mean average precision. In this paper, we show that the use of raw activation patterns may contain misleading information for introspecting 2D object detection in ADS. To this end, we investigate how to optimally pre-process these patterns for improving the error detection performance. We evaluate the developed mechanism with and without pre-processing of the raw neural activation patterns and compare its performance with a state-of-the-art algorithm highlighting that for the Berkeley Deep Drive (BDD) dataset, pre-processing reduced the ratio of missed errors by 14% and improved the overall detection performance by 3%.
Noise-in, Bias-out: Balanced and Real-time MoCap Solving

Real-time optical Motion Capture (MoCap) systems have not benefited from the advances in modern data-driven modeling. In this work we apply machine learning to solve noisy unstructured marker estimates in real-time and deliver robust marker-based MoCap even when using sparse affordable sensors. To achieve this we focus on a number of challenges related to model training, namely the sourcing of training data and their long-tailed distribution. Leveraging representation learning we design a technique for imbalanced regression that requires no additional data or labels and improves the performance of our model in rare and challenging poses. By relying on a unified representation, we show that training such a model is not bound to high-end MoCap training data acquisition, and exploit the advances in marker-less MoCap to acquire the necessary data. Finally, we take a step towards richer and affordable MoCap by adapting a body model-based inverse kinematics solution to account for measurement and inference uncertainty, further improving performance and robustness. Project page: moverseai.github.io/noise-tail.
Fast Object Detection in High-Resolution Videos

Despite the rapid evolution of video resolutions and progress on object detection algorithms, processing high resolution videos has had three main challenges so far. Firstly, it is non-trivial to use existing tracking algorithms to extend an object detection framework for efficient processing of high resolution videos. In theory, fully convolutional CNN architectures in most existing deep learning models allow any input resolution to be processed. However, in practice, inferencing on high resolution images decoded from a video incurs significant computational costs, making it impractical for real-time applications. Secondly, most tracking approaches typically require the entire frame to be decoded. Relatively little work has gone into object detection directly on compressed data, which include rich temporal cues that can be exploited to reduce the computational cost at inference time. Thirdly, most of these approaches require labeled data for training models, thereby limiting their adoption.We tackle all the three challenges in our framework by incorporating forward and backward motion cues from the compressed video to dramatically increase the processing speed of a pretrained baseline object detector, without any loss of accuracy. Our training is based on knowledge transfer from the baseline detector as a teacher network, thereby forgoing the need for any labeled data. Finally, the models are agnostic to teacher network architecture, and can be used to improve efficiency of any object detector. Our results show a speed gain of 3x to 20x compared to a frame-by-frame detector, depending upon input data resolution.
G2L: A High-Dimensional Geometric Approach for Automatic Generation of Highly Accurate Pseudo-labels

Transfer learning is a deep-learning technique that ameliorates the problem of learning when human-annotated labels are expensive and limited. In place of such labels, it uses instead the previously trained weights from a well-chosen source model as the initial weights for the training of a base model for a new target dataset. We demonstrate a novel general technique for automatically creating such source models. We generate pseudo-labels according to an efficient and extensible algorithm that is based on a classical result from the geometry of high dimensions, the Cayley-Menger determinant. This G2L ("geometry to label") method incrementally builds up pseudo-labels using a greedy computation of hypervolume content. We demonstrate that the method is tunable with respect to expected accuracy, which can be forecast by an information-theoretic measure of dataset similarity (divergence) between source and target. The results of 560 experiments show that this automatic technique generates base models that have similar or better transferability compared to a baseline of models trained on extensively human-annotated ImageNet1K labels, decreasing error in most divergent datasets tested.
Exploring Inlier and Outlier Specification for Improved Medical OOD Detection

We address the crucial task of developing well-calibrated out-of-distribution (OOD) detectors, in order to enable safe deployment of medical image classifiers. Calibration enables deep networks to protect against trivial decision rules and controls over-generalization, thereby supporting model reliability. Given the challenges involved in curating appropriate calibration datasets, synthetic augmentations have gained significant popularity for inlier/outlier specification. Despite the rapid progress in data augmentation techniques, our study reveals a remarkable finding: the synthesis space and augmentation type play a pivotal role in effectively calibrating OOD detectors. Using the popular energy-based OOD detection framework, we find that the optimal protocol is to synthesize latent-space inliers along with diverse pixel-space outliers. Through extensive empirical studies conducted on multiple medical imaging benchmarks, we consistently demonstrate the superiority of our approach, achieving substantial improvements of 15% - 35% in AUROC compared to the state-of-the-art across various open-set recognition settings.
Expressive Talking Head Video Encoding in StyleGAN2 Latent Space

While the recent advances in research on video reenactment have yielded promising results, the existing approaches fall short in capturing the fine, detailed, and expressive facial features (e.g., lip-pressing, mouth puckering, mouth gaping, and wrinkles), which are crucial in generating realistic animated face videos. To this end, we propose an end-to-end expressive face video encoding approach that facilitates data-efficient high-quality video resynthesis by optimizing low-dimensional edits of a single Identity-latent. The approach builds on StyleGAN2 image inversion and multi-stage non-linear latent space editing to generate videos that are nearly comparable to input videos. While existing StyleGAN latent-based editing techniques focus on simply generating plausible edits of static images, we automate the latent space editing to capture the fine expressive facial deformations in a sequence of frames using an encoding that resides in the Style-latent space (StyleSpace) of StyleGAN2. The encoding thus obtained could be super-imposed on a single Identity-latent to facilitate reenactment of high-resolution face videos at 1024 <sup xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">2</sup> . The proposed framework economically captures face identity, head-pose, and complex expressive facial motions at fine levels, and thereby bypasses training, person modeling, dependence on landmarks/keypoints, and low-resolution synthesis which tend to hamper most re-enactment approaches. The proposed method is designed with maximum data efficiency, where a single W+ latent and 35 parameters per frame enable high-fidelity video rendering. This pipeline can also be used for puppeteering (i.e., motion transfer). Project page: https://trevineoorloff.github.io/ExpressiveFaceVideoEncoding.io/.
A Lightweight Skeleton-Based 3D-CNN for Real-Time Fall Detection and Action Recognition

Implementing skeleton-based action recognition in real-world applications is a difficult task, because it involves multiple modules such as person detection and pose estimaton. In terms of context, skeleton-based approach has the strong advantage of robustness in understanding actual human actions. However, for most real-world videos in the standard benchmark datasets, human poses are not easy to detect, (i.e. only partially visible or occluded by other objects), and existing pose estimators mostly fail to detect the person during the falling motion. Thus, we propose a newly augmented human pose dataset to improve the accuracy of pose extraction. Furthermore, we propose a lightweight skeleton-based 3D-CNN action recognition network that shows significant improvement on accuracy and processing time over the baseline. Experimental results show that the proposed skeleton-based method shows high accuracy and efficiency in real world scenarios.
Estimation of Human Condition at Disaster Site Using Aerial Drone Images

Drones are being used to assess the situation in various disasters. In this study, we investigate a method to automatically estimate the damage status of people based on their actions in aerial drone images in order to understand disaster sites faster and save labor. We constructed a new dataset of aerial images of human actions in a hypothetical disaster that occurred in an urban area, and classified the human damage status using 3D ResNet. The results showed that the status with characteristic human actions could be classified with a recall rate of more than 80%, while other statuses with similar human actions could only be classified with a recall rate of about 50%. In addition, a cloud-based VR presentation application suggested the effectiveness of using drones to understand the disaster site and estimate the human condition.
Defense-Prefix for Preventing Typographic Attacks on CLIP

Vision-language pre-training models (VLPs) have exhibited revolutionary improvements in various vision-language tasks. In VLP, some adversarial attacks fool a model into false or absurd classifications. Previous studies addressed these attacks by fine-tuning the model or changing its architecture. However, these methods risk losing the original model's performance and are difficult to apply to downstream tasks. In particular, their applicability to other tasks has not been considered. In this study, we addressed the reduction of the impact of typographic attacks on CLIP without changing the model parameters. To achieve this, we expand the idea of "class-prefix learning" and introduce our simple yet effective method: Defense-Prefix (DP), which inserts the DP token before a class name to make words "robust" against typographic attacks. Our method can be easily applied to downstream tasks, such as object detection, because the proposed method is independent of the model parameters. Our method significantly improves the accuracy of classification tasks for typographic attack datasets, while maintaining the zero-shot capabilities of the model. In addition, we leverage our proposed method for object detection, demonstrating its high applicability and effectiveness. The codes and datasets are available at https://github.com/azuma164/Defense-Prefix.
Augmenting Features via Contrastive Learning-based Generative Model for Long-Tailed Classification

Thanks to the advances in deep learning-based computer vision, image classification has shown great achievements. However, it has faced a heavy class imbalance issue which is one of the characteristics of real-world datasets. The severe class imbalance makes the classifier easily biased toward majority classes and overfitting to minority classes. To address this issue, supplementing minority classes with artificially generated samples has proven effective. In addition, contrastive learning has been introduced to improve image classification performance recently. Motivated by recent works, we propose feature augmentation via a contrastive learning-based generative model for long-tailed classification. Specifically, features are augmented using the feature dictionary obtained by real samples and the generated convex weights, which are used for learning an image classification model. Here, the model for the feature augmentation is trained based on generative adversarial learning and contrastive learning in an end-to-end manner. The generative adversarial learning helps to generate real-like features, and the contrastive learning improves the feature’s discrimination power. Through extensive experiments with various long-tailed classification datasets, we verify the effectiveness of the proposed method.