GenAI4ELab/papercli-papers huggingface dataset:
Elusive Images: Beyond Coarse Analysis for Fine-Grained Recognition
Authors: Connor Anderson; Matt Gwilliam; Evelyn Gaskin; Ryan Farrell
Venue: WACV 2024 (main)

Abstract: While the community has seen many advances in recent years to address the challenging problem of Finegrained Visual Categorization (FGVC), progress seems to be slowing--new state-of-the-art methods often distinguish themselves by improving top-1 accuracy by mere tenths of a percent. However, across all of the now-standard FGVC datasets, there remain sizeable portions of the test data that none of the current state-of-the-art (SOTA) models can successfully predict. This paper provides a framework for identifying and studying the errors that current methods make across diverse fine-grained datasets. Three models of difficulty--Prediction Overlap, Prediction Rank and Pairwise Class Confusion--are employed to highlight the most challenging sets of images and classes. Extensive experiments apply a range of standard and SOTA methods, evaluating them on multiple FGVC domains and datasets. Insights acquired from coupling these difficulty paradigms with the careful analysis of experimental results suggest crucial areas for future FGVC research, focusing critically on the set of elusive images that none of the current models can correctly classify. Code is available at catalys1.github.io/elusive-images-fgvc.
Tracking Skiers From the Top to the Bottom
Authors: Matteo Dunnhofer; Luca Sordi; Niki Martinel; Christian Micheloni
Venue: WACV 2024 (main)

Abstract: Skiing is a popular winter sport discipline with a long history of competitive events. In this domain, computer vision has the potential to enhance the understanding of athletes' performance, but its application lags behind other sports due to limited studies and datasets. This paper makes a step forward in filling such gaps. A thorough investigation is performed on the task of skier tracking in a video capturing his/her complete performance. Obtaining continuous and accurate skier localization is preemptive for further higher-level performance analyses. To enable the study, the largest and most annotated dataset for computer vision in skiing, SkiTB, is introduced. Several visual object tracking algorithms, including both established methodologies and a newly introduced skier-optimized baseline algorithm, are tested using the dataset. The results provide valuable insights into the applicability of different tracking methods for vision-based skiing analysis. SkiTB, code, and results are available at https://machinelearning.uniud.it/datasets/skitb.
BPKD: Boundary Privileged Knowledge Distillation for Semantic Segmentation
Authors: Liyang Liu; Zihan Wang; Minh Hieu Phan; Bowen Zhang; Jinchao Ge; Yifan Liu
Venue: WACV 2024 (main)

Abstract: Current knowledge distillation approaches in semantic segmentation tend to adopt a holistic approach that treats all spatial locations equally. However, for dense prediction, students' predictions on edge regions are highly uncertain due to contextual information leakage, requiring higher spatial sensitivity knowledge than the body regions. To address this challenge, this paper proposes a novel approach called boundary-privileged knowledge distillation (BPKD). it distils the knowledge from the teacher model's body and edges separately to the compact student model. Specifically, we employ two distinct loss functions: (i) edge loss, which aims to distinguish between ambiguous classes at the pixel level in edge regions; (ii) body loss, which utilizes shape constraints and selectively attends to the inner-semantic regions. Our experiments demonstrate that the proposed BPKD method provides extensive refinements and aggregation for edge and body regions. Additionally, the method achieves state-of-the-art distillation performance for semantic segmentation on three popular benchmark datasets, highlighting its effectiveness and generalization ability. BPKD shows consistent improvements across a diverse array of lightweight segmentation structures, including both CNNs and transformers, underscoring its architecture-agnostic adaptability.
DREAM: Visual Decoding From Reversing Human Visual System
Authors: Weihao Xia; Raoul de Charette; Cengiz Oztireli; Jing-Hao Xue
Venue: WACV 2024 (main)

Abstract: In this work we present DREAM, an fMRI-to-image method for reconstructing viewed images from brain activities, grounded on fundamental knowledge of the human visual system. We craft reverse pathways that emulate the hierarchical and parallel nature of how humans perceive the visual world. These tailored pathways are specialized to decipher semantics, color, and depth cues from fMRI data, mirroring the forward pathways from visual stimuli to fMRI recordings. To do so, two components mimic the inverse processes within the human visual system: the Reverse Visual Association Cortex (R-VAC) which reverses pathways of this brain region, extracting semantics from fMRI data; the Reverse Parallel PKM (R-PKM) component simultaneously predicting color and depth from fMRI signals. The experiments indicate that our method outperforms the current state-of-the-art models in terms of the consistency of appearance, structure, and semantics. Code will be available at https://github.com/weihaox/DREAM.
Seeing Stars: Learned Star Localization for Narrow-Field Astrometry
Authors: Violet Felt; Justin Fletcher
Venue: WACV 2024 (main)

Abstract: Star localization in astronomical imagery is a computer vision task that underpins satellite tracking. Astronomical star extraction techniques often struggle to detect stars when applied to satellite tracking imagery due to the narrower fields of view and rate track observational modes of satellite tracking telescopes. We present a large dataset of real narrow-field rate-tracked imagery with ground truth stars, created using a combination of existing star detection techniques, an astrometric engine, and a star catalog. We train three state of the art object detection, instance segmentation, and line segment detection models on this dataset and evaluate them with object-wise, pixel-wise, and astrometric metrics. Our proposed approaches require no metadata; when paired with a lost-in-space astrometric engine, they find astrometric fits based solely on uncorrected image pixels. Experimental results on real data indicate the effectiveness of learned star detection: we report astrometric fit rates over double that of classical star detection algorithms, improved dim star recall, and comparable star localization residuals.
How Do Deepfakes Move? Motion Magnification for Deepfake Source Detection
Authors: Ilke Demir; Umur Aybars Çiftçi
Venue: WACV 2024 (main)

Abstract: With the proliferation of deep generative models, deepfakes are improving in quality and quantity everyday. However, there are subtle authenticity signals in pristine videos, not replicated by current generative models. We contrast the movement in deepfakes and authentic videos by motion magnification towards building a generalized deepfake source detector. The sub-muscular motion in faces has different interpretations per different generative models, which is reflected in their generative residue. Our approach exploits the difference between real motion and the amplified generative artifacts, by combining deep and traditional motion magnification, to detect whether a video is fake and its source generator if so. Evaluating our approach on two multi-source datasets, we obtain 97.77% and 94.03% for video source detection. Our approach performs at least 4.08% better than the prior deepfake source detector and other complex architectures. We also analyze magnification amount, phase extraction window, backbone network, sample counts, and sample lengths. Finally, we report our results on skin tones and genders to assess the model bias.
Separable Self and Mixed Attention Transformers for Efficient Object Tracking
Authors: Goutam Yelluru Gopal; Maria A. Amer
Venue: WACV 2024 (main)

Abstract: The deployment of transformers for visual object tracking has shown state-of-the-art results on several benchmarks. However, the transformer-based models are under-utilized for Siamese lightweight tracking due to the computational complexity of their attention blocks. This paper proposes an efficient self and mixed attention transformer-based architecture for lightweight tracking. The proposed backbone utilizes the separable mixed attention transformers to fuse the template and search regions during feature extraction to generate superior feature encoding. Our prediction head performs global contextual modeling of the encoded features by leveraging efficient self-attention blocks for robust target state estimation. With these contributions, the proposed lightweight tracker deploys a transformer-based backbone and head module concurrently for the first time. Our ablation study testifies to the effectiveness of the proposed combination of backbone and head modules. Simulations show that our Separable Self and Mixed Attention-based Tracker, SMAT, surpasses the performance of related lightweight trackers on GOT10k, TrackingNet, LaSOT, NfS30, UAV123, and AVisT datasets, while running at 37 fps on CPU, 158 fps on GPU, and having 3.8M parameters. For example, it significantly surpasses the closely related trackers E.T.Track and MixFormerV2-S on GOT10k-test by a margin of 7.9% and 5.8%, respectively, in the AO metric. The tracker code and model is available at https://github.com/goutamyg/SMAT
CLIPAG: Towards Generator-Free Text-to-Image Generation
Authors: Roy Ganz; Michael Elad
Venue: WACV 2024 (main)

Abstract: Perceptually Aligned Gradients (PAG) refer to an intriguing property observed in robust image classification models, wherein their input gradients align with human perception and pose semantic meanings. While this phenomenon has gained significant research attention, it was solely studied in the context of unimodal vision-only architectures. In this work, we extend the study of PAG to Vision-Language architectures, which form the foundations for diverse image-text tasks and applications. Through an adversarial robustification finetuning of CLIP, we demonstrate that robust Vision-Language models exhibit PAG in contrast to their vanilla counterparts. This work reveals the merits of CLIP with PAG (CLIPAG) in several vision-language generative tasks. Notably, we show that seamlessly integrating CLIPAG in a "plug-n-play" manner leads to substantial improvements in vision-language generative applications. Furthermore, leveraging its PAG property, CLIPAG enables text-to-image generation without any generative model, which typically requires huge generators.
Source-Guided Similarity Preservation for Online Person Re-Identification
Authors: Hamza Rami; Jhony H. Giraldo; Nicolas Winckler; Stéphane Lathuilière
Venue: WACV 2024 (main)

Abstract: Online Unsupervised Domain Adaptation (OUDA) for person Re-Identification (Re-ID) is the task of continuously adapting a model trained on a well-annotated source-domain dataset to a target domain observed as a data stream. In OUDA, person Re-ID models face two main challenges: catastrophic forgetting and domain shift. In this work, we propose a new Source-guided Similarity Preservation (S2P) framework to alleviate these two problems. Our framework is based on the extraction of a support set composed of source images that maximizes the similarity with the target data. This support set is used to identify feature similarities that must be preserved during the learning process. S2P can incorporate multiple existing UDA methods to mitigate catastrophic forgetting. Our experiments show that S2P outperforms previous state-of-the-art methods on multiple real-to-real and synthetic-to-real challenging OUDA benchmarks.
Uncertainty-Weighted Loss Functions for Improved Adversarial Attacks on Semantic Segmentation
Authors: Kira Maag; Asja Fischer
Venue: WACV 2024 (main)

Abstract: State-of-the-art deep neural networks have been shown to be extremely powerful in a variety of perceptual tasks like semantic segmentation. However, these networks are vulnerable to adversarial perturbations of the input which are imperceptible for humans but lead to incorrect predictions. Treating image segmentation as a sum of pixel-wise classifications, adversarial attacks developed for classification models were shown to be applicable to segmentation models as well. In this work, we present simple uncertainty-based weighting schemes for the loss functions of such attacks that (i) put higher weights on pixel classifications which can more easily perturbed and (ii) zero-out the pixel-wise losses corresponding to those pixels that are already confidently misclassified. The weighting schemes can be easily integrated into the loss function of a range of well-known adversarial attackers with minimal additional computational overhead, but lead to significant improved perturbation performance, as we demonstrate in our empirical analysis on several datasets and models.
Towards Realistic Generative 3D Face Models
Authors: Aashish Rai; Hiresh Gupta; Ayush Pandey; Francisco Vicente Carrasco; Shingo Jason Takagi; Amaury Aubel; Daeil Kim; Aayush Prakash; Fernando De la Torre
Venue: WACV 2024 (main)

Abstract: In recent years, there has been significant progress in 2D generative face models fueled by applications such as animation, synthetic data generation, and digital avatars. However, due to the absence of 3D information, these 2D models often struggle to accurately disentangle facial attributes like pose, expression, and illumination, limiting their editing capabilities. To address this limitation, this paper proposes a 3D controllable generative face model to produce high-quality albedo and precise 3D shapes by leveraging existing 2D generative models. By combining 2D face generative models with semantic face manipulation, this method enables editing of detailed 3D rendered faces. The proposed framework utilizes an alternating descent optimization approach over shape and albedo. Differentiable rendering is used to train high-quality shapes and albedo without 3D supervision. Moreover, this approach outperforms most state-of-the-art (SOTA) methods in the well-known NoW and REALY benchmarks for 3D face reconstruction. It also outperforms the SOTA reconstruction models in recovering rendered faces' identities across novel poses. Additionally, the paper demonstrates direct control of expressions in 3D faces by exploiting latent space leading to text-based editing of 3D faces.
Domain Generalization by Rejecting Extreme Augmentations
Authors: Masih Aminbeidokhti; Fidel A. Guerrero Peña; Heitor Rapela Medeiros; Thomas Dubail; Eric Granger; Marco Pedersoli
Venue: WACV 2024 (main)

Abstract: Data augmentation is one of the most powerful techniques for regularizing deep learning models and improving their recognition performance in a variety of tasks and domains. However, this holds for standard in-domain settings, in which the training and test data follow the same distribution. For the out-domain, in which the test data follows a different and unknown distribution, the best recipe for data augmentation is not clear. In this paper, we show that also for out-domain or domain generalization settings, data augmentation can bring a conspicuous and robust improvement in performance. For doing that, we propose a simple procedure: i) use uniform sampling on standard data augmentation transformations ii) increase transformations strength to adapt to the higher data variance expected when working out of domain iii) devise a new reward function to reject extreme transformations that can harm the training. With this simple formula, our data augmentation scheme achieves comparable or better results to state-of-the-art performance on most domain generalization datasets.
Towards Accurate Disease Segmentation in Plant Images: A Comprehensive Dataset Creation and Network Evaluation
Authors: Komuravelli Prashanth; Jaladi Sri Harsha; Sivapuram Arun Kumar; Jaladi Srilekha
Venue: WACV 2024 (main)

Abstract: Automated disease segmentation in plant images plays a crucial role in identifying and mitigating the impact of plant diseases on agricultural productivity. In this study, we address the problem of Northern Leaf Blight (NLB) disease segmentation in maize plants. We present a comprehensive dataset of 1000 plant images annotated with NLB disease regions. We employ the Mask R-CNN and Cascaded Mask R-CNN models with various backbone architectures to perform NLB disease segmentation. The experimental results demonstrate the effectiveness of the models in accurately delineating NLB disease regions. Specifically, the ResNet Strikes Back-50 backbone architecture achieves the highest mean average precision (mAP) score, indicating its ability to capture intricate details of NLB disease spots. Additionally, the cascaded approach enhances segmentation accuracy compared to the single-stage Mask R-CNN models. Our findings provide valuable insights into the performance of different backbone architectures and contribute to the development of automated NLB disease segmentation methods in plant images. The generated dataset and experimental results serve as a resource for further research in plant disease segmentation and management.
Deep Subdomain Alignment for Cross-Domain Image Classification
Authors: Yewei Zhao; Hu Han; Shiguang Shan; Xilin Chen
Venue: WACV 2024 (main)

Abstract: Unsupervised domain adaptation (UDA), which aims to transfer knowledge learned from a labeled source domain to an unlabeled target domain, is useful for various cross-domain image classification scenarios. A commonly used approach for UDA is to minimize the distribution differences between two domains, and subdomain alignment is found to be an effective method. However, most of the existing subdomain alignment methods are based on adversarial learning and focus on subdomain alignment procedures without considering the discriminability among individual subdomains, resulting in slow convergence and unsatisfactory adaptation results. To address these issues, we propose a novel deep subdomain alignment method for UDA in image classification, which consists of a Union Subdomain Contrastive Learning (USCL) module and a Multi-view Subdomain Alignment (MvSA) strategy. USCL can create discriminative and dispersed subdomains by bringing samples from the same subdomain closer while pushing away samples from different subdomains. MvSA makes use of labeled source domain data and easy target domain data to perform target-to-source and target-to-target alignment. Experimental results on three image classification datasets (Office-31, Office-Home, Visda-17) demonstrate that our proposed method is effective for UDA and achieves promising results in several cross-domain image classification tasks.
Classifying Cable Tendency With Semantic Segmentation by Utilizing Real and Simulated RGB Data
Authors: Pei-Chun Chien; Powei Liao; Eiji Fukuzawa; Jun Ohya
Venue: WACV 2024 (main)

Abstract: Cable tendency is the potential shape or characteristic that a cable may possess while being manipulated, of which some are considered erroneous and should be identified as a part of anomaly detection during an automatic manipulation. This research explores the ability of deep-learning models in learning the cable tendencies that, contrary to typical classification tasks of multi-object scenarios, is to differentiate the multiple states displayable by the same object -- in this case, cables. By training multiple models with different combinations of self-collected real-world data and self-generated simulation data, a comparative study is carried out to compare the performance of each approach. In conclusion, the effectiveness of detecting three abnormal states and shapes of cables, and using simulation data is certificated in experiments.
Visual Narratives: Large-Scale Hierarchical Classification of Art-Historical Images
Authors: Matthias Springstein; Stefanie Schneider; Javad Rahnama; Julian Stalter; Maximilian Kristen; Eric Müller-Budack; Ralph Ewerth
Venue: WACV 2024 (main)

Abstract: Iconography refers to the methodical study and interpretation of thematic content in the visual arts, distinguishing it, e.g., from purely formal or aesthetic considerations. In iconographic studies, Iconclass is a widely used taxonomy that encapsulates historical, biblical, and literary themes, among others. However, given the hierarchical nature and inherent complexity of such a taxonomy, it is highly desirable to use automated methods for (Iconclass-based) image classification. Previous studies either focused narrowly on certain subsets of narratives or failed to exploit Iconclass's hierarchical structure. In this paper, we propose a novel approach for Hierarchical Multi-label Classification (HMC) of iconographic concepts in images. We present three strategies, including Large Language Models (LLMs), for the generation of textual image descriptions using keywords extracted from Iconclass. These descriptions are utilized to pre-train a Vision-Language Model (VLM) based on a newly introduced data set of 477,569 images with more than 20,000 Iconclass concepts, far more than considered in previous studies. Furthermore, we present five approaches to multi-label classification, including a novel transformer decoder that leverages hierarchical information from the Iconclass taxonomy. Experimental results show the superiority of this approach over reasonable baselines.
Real-Time Weakly Supervised Video Anomaly Detection
Authors: Hamza Karim; Keval Doshi; Yasin Yilmaz
Venue: WACV 2024 (main)

Abstract: Weakly supervised video anomaly detection is an important problem in many real-world applications where during training there are some anomalous videos, in addition to nominal videos, without labelled frames to indicate when the anomaly happens. State-of-the-art methods in this domain typically focus on offline anomaly detection without any concern for real-time detection. Most of these methods rely on ad hoc feature aggregation techniques and the use of metric learning losses, which limit the ability of the models to detect anomalies in real-time. In line with the premise of deep neural networks, there also has been a growing interest in developing end-to-end approaches that can automatically learn effective features directly from the raw data. We propose the first real-time and end-to-end trained algorithm for weakly supervised video anomaly detection. Our training procedure builds upon recent action recognition literature and uses a trainable video model to learn visual features. This is in contrast to existing approaches which largely depend on pre-trained feature extractors. The proposed method significantly improves the anomaly detection speed and AUC performance compared to the existing methods. Specifically, on the UCF-Crime dataset, our method achieves 86.94% AUC with a decision period of 6.4 seconds while the competing methods achieve at most 85.92% AUC with a decision period of 273 seconds.