GenAI4ELab/papercli-papers huggingface dataset:
Heterogeneous Datasets for Unsupervised Image Anomaly Detection
Authors: Juan Lagos; Haider Ali; Adnan Faroque; Esa Rahtu
Venue: WACV 2025 (main)

Abstract: Unsupervised anomaly detection (AD) is a critical task in various domains from manufacturing to infrastructure monitoring. To advance this field we introduce two novel datasets: CARS-AD and ROADS-AD designed to challenge existing unsupervised AD methods with their diverse and heterogeneous image content. CARS-AD comprises real images of cars with various defects while ROADS-AD contains images of roads from multiple countries each presenting unique challenges in anomaly detection. These datasets provide ground truth pixel-wise masks and image-level ground truth labels enabling detailed evaluation and benchmarking of AD algorithms. We evaluate state-of-the-art unsupervised AD methods on both datasets using the AUROC metric to assess detection and localization performance. Our results reveal significant room for improvement underscoring the complexity of the datasets and the need for robust AD techniques. Notably Csflow and U-Flow demonstrate superior performance on the CARS-AD Dataset leveraging their ability to process multi-scale features effectively. Conversely Reverse Distillation excels in anomaly localization on the ROADS-AD Dataset showcasing the importance of nuanced approaches for diverse anomaly types. Our findings underscore the importance of addressing the challenges posed by heterogeneous datasets in unsupervised AD. We hope that the introduction of CARS-AD and ROADS-AD will inspire further research in this field driving the development of innovative AD methods capable of handling real-world anomalies with greater accuracy and reliability. CARS-AD Dataset and ROADS-AD Dataset are publicly available at https://github.com/juanb09111/heterogeneousAD.
A Parametric Approach to Adversarial Augmentation for Cross-Domain Iris Presentation Attack Detection
Authors: Debasmita Pal; Redwan Sony; Arun Ross
Venue: WACV 2025 (main)

Abstract: Iris-based biometric systems are vulnerable to presentation attacks (PAs) where adversaries present physical artifacts (e.g. printed iris images textured contact lenses) to defeat the system. This has led to the development of various presentation attack detection (PAD) algorithms which typically perform well in intra-domain settings. However they often struggle to generalize effectively in cross-domain scenarios where training and testing employ different sensors PA instruments and datasets. In this work we use adversarial training samples of both bonafide irides and PAs to improve the cross-domain performance of a PAD classifier. The novelty of our approach lies in leveraging transformation parameters from classical data augmentation schemes (e.g. translation rotation) to generate adversarial samples. We achieve this through a convolutional autoencoder ADV-GEN that inputs original training samples along with a set of geometric and photometric transformations. The transformation parameters act as regularization variables guiding ADV-GEN to generate adversarial samples in a constrained search space. Experiments conducted on the LivDet-Iris 2017 database comprising four datasets and the LivDet-Iris 2020 dataset demonstrate the efficacy of our proposed method. The code is available at https://github.com/iPRoBe-lab/ADV-GEN-IrisPAD.
OPTIMUS: Observing Persistent Transformations in Multi-Temporal Unlabeled Satellite-Data
Authors: Raymond Yu; Paul Han; Piper Wolters; Favyen Bastani
Venue: WACV 2025 (main)

Abstract: In the face of pressing environmental issues in the 21st century monitoring surface changes on Earth is more important than ever. Large-scale remote sensing such as satellite imagery is an important tool for this task. However using supervised methods to detect changes is difficult because of the lack of satellite data annotated with change labels especially for rare categories of change. Annotation proves challenging due to the sparse occurrence of changes in satellite images. Even within a vast collection of images only a small fraction may exhibit persistent changes of interest. To address this challenge we introduce OPTIMUS a self-supervised learning method based on an intuitive principle: if a model can recover information about the relative order of images in the time series then that implies that there are long-lasting changes in the images. OPTIMUS demonstrates this principle by using change point detection methods on model outputs in a time series. We demonstrate that OPTIMUS can directly detect interesting changes in satellite images achieving an improvement in AUROC score from 56.3% to 87.6% at distinguishing changed time series from unchanged ones compared to baselines. Our code and dataset are available at https://huggingface.co/datasets/optimus-change/optimus-dataset/.
Copy or Not? Reference-Based Face Image Restoration with Fine Details
Authors: Min Jin Chong; Dejia Xu; Yi Zhang; Zhangyang Wang; David Forsyth; Gurunandan Krishnan; Yicheng Wu; Jian Wang
Venue: WACV 2025 (main)

Abstract: Reference-guided face restoration can have better identity preservation than non-reference-based methods. However existing methods can (a) easily produce artifacts possibly attributable to inefficient facial priors and (b) do not well preserve fine-grained facial details crucial for identity such as freckles tattoos and scars. In this work we propose solutions for these problems. (1) We incorporate a stronger facial prior generative facial prior (GFP) for reference-based face image restoration. (2) We identify an ambiguity and point out that traditional loss prevents the network from heavily copying facial features from the reference. To address this we set a new goal and come up with a new loss to realize the new goal. More specifically when the ground truth and reference are different (e.g. differences in wrinkles makeup facial hair etc.) which one should the output look like? As a simple example ground truth does not have a mole while reference has one. Traditional loss chose the ground truth which seems natural but then the network also learns to ignore reference's facial features; during testing the network often hesitates. Our new goal is to copy features from the reference as much as possible while maintaining semantic consistency with the degraded input. We propose to use spatial minimum loss and cycle consistency loss to realize the new goal and make the network copy features without hesitation. Using only a single reference image our proposed method is able to restore highly degraded images while accurately capturing fine-grained facial details. To our knowledge we are the first face restoration framework that is able to restore faces at this granularity. Code and data are available at https://github.com/RefineFIR/RefineFIR.
PVT: An Implicit Surface Reconstruction Framework via Point Voxel Geometric-Aware Transformer
Authors: Chuanmao Fan; Chenxi Zhao; Ye Duan
Venue: WACV 2025 (main)

Abstract: 3D surface reconstruction from unorganized point clouds is a fundamental task in visual computing with numerous applications in areas such as robotics virtual reality augmented reality and animation. To date many deep learning-based surface reconstruction methods have been proposed demonstrating great performance on many benchmark datasets. Among these neural implicit field learning-based methods have gained popularity for their capability of representing complex structures in a continuous implicit distance field. Existing neural implicit field learning methods either utilize voxelized point cloud then feed them to a deep network or directly take points as input. In this paper we propose an implicit surface reconstruction framework based on point voxel geometric-aware transformer PVT to seamlessly integrate point-based convolution with voxel-based convolution using bidirectional transformers. Experiments show that the proposed PVT framework can better encode local geometry details and provide a significant performance boost over existing state-of-the-art methods.
SyncDiff: Diffusion-Based Talking Head Synthesis with Bottlenecked Temporal Visual Prior for Improved Synchronization
Authors: Xulin Fan; Heting Gao; Ziyi Chen; Peng Chang; Mei Han; Mark Hasegawa-Johnson
Venue: WACV 2025 (main)

Abstract: Talking head synthesis also known as speech-to-lip synthesis reconstructs the facial motions that align with the given audio tracks. The synthesized videos are evaluated on mainly two aspects lip-speech synchronization and image fidelity. Recent studies demonstrate that GAN-based and diffusion-based models achieve state-of-the-art (SOTA) performance on this task with diffusion-based models achieving superior image fidelity but experiencing lower synchronization compared to their GAN-based counterparts. To this end we propose SyncDiff a simple yet effective approach to improve diffusion-based models using a temporal pose frame with information bottleneck and facial-informative audio features extracted from AVHuBERT as conditioning input into the diffusion process. We evaluate SyncDiff on two canonical talking head datasets LRS2 and LRS3 for direct comparison with other SOTA models. Experiments on LRS2/LRS3 datasets show that SyncDiff achieves a synchronization score 27.7%/62.3% relatively higher than previous diffusion-based methods while preserving their high-fidelity characteristics.
AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuning
Authors: Muhammad Awais; Ali Husain Salem Abdulla Alharthi; Amandeep Kumar; Hisham Cholakkal; Rao Muhammad Anwer
Venue: WACV 2025 (main)

Abstract: Significant progress has been made in advancing large multimodal conversational models (LMMs) capitalizing on vast repositories of image-text data available online. Despite this progress these models often encounter substantial domain gaps hindering their ability to engage in complex conversations across new domains. Recent efforts have aimed to mitigate this issue albeit relying on domain-specific image-text data to curate instruction-tuning data. However many domains such as agriculture lack such vision-language data. In this work we propose an approach to construct instruction-tuning data that harnesses vision-only data for the agriculture domain. We utilize diverse agricultural datasets spanning multiple domains curate class-specific information and employ large language models (LLMs) to construct an expert-tuning set resulting in a 70k AgroInstruct. Subsequently we expert-tuned and created AgroGPT an efficient LMM that can hold complex agriculture-related conversations and provide useful insights. We also develop AgroEvals for evaluation and compare AgroGPT's performance with large open and closed-source models. AgroGPT excels at identifying fine-grained agricultural concepts can act as an agriculture expert and provides helpful information for multimodal agriculture questions. The code datasets and models are available at https://github.com/awaisrauf/agroGPT.
Can Location Embeddings Enhance Super-Resolution of Satellite Imagery?
Authors: Daniel Panangian; Ksenia Bittner
Venue: WACV 2025 (main)

Abstract: Publicly available satellite imagery such as Sentinel-2 often lacks the spatial resolution required for accurate analysis of remote sensing tasks including urban planning and disaster response. Current super-resolution techniques are typically trained on limited datasets leading to poor generalization across diverse geographic regions. In this work we propose a novel super-resolution framework that enhances generalization by incorporating geographic context through location embeddings. Our framework employs GANs and incorporates techniques from diffusion models to enhance image quality. Furthermore we address tiling artifacts by integrating information from neighboring images enabling the generation of seamless high-resolution outputs. We demonstrate the effectiveness of our method on the building segmentation task showing significant improvements over state-of-the-art methods and highlighting its potential for real-world applications.
TFM^2: Training-Free Mask Matching for Open-Vocabulary Semantic Segmentation
Authors: Yaoxin Zhuo; Zachary Bessinger; Lichen Wang; Naji Khosravan; Baoxin Li; Sing Bing Kang
Venue: WACV 2025 (main)

Abstract: The potential of Open-Vocabulary Semantic Segmentation (OVSS) in few-shot scenarios is not fully explored due to the complexity of extending few-shot concepts to semantic segmentation tasks. To address this challenge we propose Training-Free Mask Matching (TFM^2) an efficient mask-based adapter method that enhances OVSS models for the few-shot open vocabulary semantic segmentation task. TFM^2 is a key-value cache that explicitly designed for image masks. We introduce three modules to construct and refine the mask cache subsequently enhancing the OVSS mask classification performance. Comprehensive experiments demonstrate that TFM^2 improves the performance of state-of-the-art OVSS methods by a margin of 1% to 5% across different settings. Moreover TFM^2 is not limited to any specific methods or backbones. This work underscores the importance and potential of few-shot data in OVSS and presents a significant step toward leveraging this potential.
Visual Robustness Benchmark for Visual Question Answering (VQA)
Authors: Farhan Ishmam; Ishmam Tashdeed; Talukder Asir Saadat; Hamjajul Ashmafee; Abu Raihan Mostofa Kamal; Azam Hossain
Venue: WACV 2025 (main)

Abstract: Can Visual Question Answering (VQA) systems maintain their performance when deployed in the real world? Or are they susceptible to realistic corruption effects e.g. image blur which can be detrimental in sensitive applications such as medical VQA? While linguistic robustness has been thoroughly explored within the VQA literature there has yet to be any significant work on visual robustness. In this work we present the first large-scale benchmark to evaluate the visual robustness of VQA models including multimodal large language models and zero-shot evaluation and assess the strength of the realistic corruption effects. Additionally we have designed several robustness evaluation metrics that quantify an aspect of robustness. These sub-metrics can be aggregated into a unified metric and tailored to fit a variety of use cases. The experiments reveal important insights into the relationships between model size accuracy and robustness against the visual corruptions. Our benchmark highlights the need for a balanced approach in model development that considers model performance without compromising robustness.
NCAdapt: Dynamic Adaptation with Domain-Specific Neural Cellular Automata for Continual Hippocampus Segmentation
Authors: Amin Ranem; John Orlando Kalkhof; Anirban Mukhopadhyay
Venue: WACV 2025 (main)

Abstract: Continual learning (CL) in medical imaging presents a unique challenge requiring models to adapt to new domains while retaining previously acquired knowledge. We introduce NCAdapt a Neural Cellular Automata (NCA) based method designed to address this challenge. NCAdapt features a domain-specific multi-head structure integrating adaptable convolutional layers into the NCA backbone for each new domain encountered. After initial training the NCA backbone is frozen and only the newly added adaptable convolutional layers consisting of 384 parameters are trained along with domain-specific NCA convolutions. We evaluate NCAdapt on hippocampus segmentation tasks benchmarking its performance against Lifelong nnU-Net and U-Net models with state-of-the-art (SOTA) CL methods. Our lightweight approach achieves SOTA performance underscoring its effectiveness in addressing CL challenges in medical imaging. Upon acceptance we will make our code base publicly accessible to support reproducibility and foster further advancements in medical CL.
Inverse Problems with Diffusion Models: A MAP Estimation Perspective
Authors: Sai Bharath Chandra Gutha; Ricardo Vinuesa; Hossein Azizpour
Venue: WACV 2025 (main)

Abstract: Inverse problems have many applications in science and engineering. In Computer vision several image restoration tasks such as inpainting deblurring and super-resolution can be formally modeled as inverse problems. Recently methods have been developed for solving inverse problems that only leverage a pre-trained unconditional diffusion model and do not require additional task-specific training. In such methods however the inherent intractability of determining the conditional score function during the reverse diffusion process poses a real challenge leaving the methods to settle with an approximation instead which affects their performance in practice. Here we propose a MAP estimation framework to model the reverse conditional generation process of a continuous time diffusion model as an optimization process of the underlying MAP objective whose gradient term is tractable. In theory the proposed framework can be applied to solve general inverse problems using gradient-based optimization methods. However given the highly non-convex nature of the loss objective finding a perfect gradient-based optimization algorithm can be quite challenging nevertheless our framework offers several potential research directions. We use our proposed formulation to develop empirically effective algorithms for image restoration. We validate our proposed algorithms with extensive experiments over multiple datasets across several restoration tasks.
Temporally Grounding Instructional Diagrams in Unconstrained Videos
Authors: Jiahao Zhang; Frederic Z. Zhang; Cristian Rodriguez; Yizhak Ben-Shabat; Anoop Cherian; Stephen Gould
Venue: WACV 2025 (main)

Abstract: We study the challenging problem of simultaneously localizing a sequence of queries in the form of instructional diagrams in a video. This requires understanding not only the individual queries but also their interrelationships. However most existing methods focus on grounding one query at a time ignoring the inherent structures among queries such as the general mutual exclusiveness and the temporal order. Consequently the predicted timespans of different step diagrams may overlap considerably or violate the temporal order thus harming the accuracy. In this paper we tackle this issue by simultaneously grounding a sequence of step diagrams. Specifically we propose composite queries constructed by exhaustively pairing up the visual content features of the step diagrams and a fixed number of learnable positional embeddings. Our insight is that self-attention among composite queries carrying different content features suppress each other to reduce timespan overlaps in predictions while the cross-attention corrects the temporal misalignment via content and position joint guidance. We demonstrate the effectiveness of our approach on the IAW dataset for grounding step diagrams and the YouCook2 benchmark for grounding natural language queries significantly outperforming existing methods while simultaneously grounding multiple queries.
A Data Perspective on Enhanced Identity Preservation for Diffusion Personalization
Authors: Xingzhe He; Zhiwen Cao; Nick Kolkin; Lantao Yu; Kun Wan; Helge Rhodin; Ratheesh Kalarot
Venue: WACV 2025 (main)

Abstract: Large text-to-image models have revolutionized the ability to generate imagery using natural language. However particularly unique or personal visual concepts such as pets and furniture will not be captured by the original model. This has led to interest in how to personalize a text-to-image model. Despite significant progress this task remains a formidable challenge particularly in preserving the subject's identity. Most researchers attempt to address this issue by modifying model architectures. These methods are capable of keeping the subject structure and color but fail to preserve identity details. Towards this issue our approach takes a data-centric perspective. We introduce a novel regularization dataset generation strategy on both the text and image level. This strategy enables the model to preserve fine details of the desired subjects such as text and logos. Our method is architecture-agnostic and can be flexibly applied on various text-to-image models. We show on established benchmarks that our data-centric approach forms the new state of the art in terms of identity preservation and text alignment.
Multimodal Interpretable Depression Analysis using Visual Physiological Audio and Textual Data
Authors: Puneet Kumar; Shreshtha Misra; Zhuhong Shao; Bin Zhu; Balasubramanian Raman; Xiaobai Li
Venue: WACV 2025 (main)

Abstract: Motivated by depression's significant impact on global health this work proposes MultiDepNet a novel multimodal interpretable depression detection system integrating visual physiological audio and textual data. Through dedicated feature extraction methods (MTCNN for video TS-CAN for physiological ResNet-18 for audio and RoBERTa for text modalities) and a strategic fusion of modality-specific networks including CNN-RNN Transformer MLP and ResNet-18 it achieves significant advancements in depression detection. Its performance evaluated across four benchmark datasets (AVEC 2013 AVEC 2014 DAIC and E-DAIC) demonstrates average MAE of 5.64 RMSE of 7.15 accuracy of 74.19 precision of 0.7373 recall of 0.7378 and F1 of 0.7376. It also implements a MultiViz-based interpretability mechanism that computes each modality's contribution to the model's performance. The results reveal the visual modality to be the most significant contributing 37.88% towards depression detection.
Seeing Eye to AI: Comparing Human Gaze and Model Attention in Video Memorability
Authors: Prajneya Kumar; Eshika Khandelwal; Makarand Tapaswi; Vishnu Sreekumar
Venue: WACV 2025 (main)

Abstract: Understanding what makes a video memorable has important applications in advertising and education technology. Towards this goal we investigate spatio-temporal attention mechanisms underlying video memorability. Different from previous works that fuse multiple features we adopt a simple CNN+Transformer architecture that enables analysis of spatio-temporal attention while matching state-of-the-art (SoTA) performance on video memorability prediction. We compare model attention against human gaze fixations collected through a small-scale eye-tracking study where humans perform the video memory task. We uncover the following insights: (i) Quantitative saliency metrics show that our model trained only to predict a memorability score exhibits similar spatial attention patterns to human gaze especially for more memorable videos. (ii) The model assigns greater importance to initial frames in a video mimicking human attention patterns. (iii) Panoptic segmentation reveals that both (model and humans) assign a greater share of attention to things and less attention to stuff as compared to their occurrence probability.
Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency
Authors: Maor Dikter; Tsachi Blau; Chaim Baskin
Venue: WACV 2025 (main)

Abstract: Concept bottleneck models (CBMs) have emerged as critical tools in domains where interpretability is paramount. These models rely on predefined textual descriptions referred to as concepts to inform their decision-making process and offer more accurate reasoning. As a result the selection of concepts used in the model is of utmost significance. This study proposes Conceptual Learning via Embedding Approximations for Reinforcing Interpretability and Transparency abbreviated as CLEAR a framework for constructing a CBM for image classification. Using score matching and Langevin sampling we approximate the embedding of concepts within the latent space of a vision-language model (VLM) by learning the scores associated with the joint distribution of images and concepts. A concept selection process is then employed to optimize the similarity between the learned embeddings and the predefined ones. The derived bottleneck offers insights into the CBM's decision-making process enabling more comprehensive interpretations. Our approach was evaluated through extensive experiments and achieved state-of-the-art performance on various benchmarks. The code for our experiments is available at https://github.com/clearProject/CLEAR/tree/main.