GenAI4ELab/papercli-papers huggingface dataset:
Can Adversarial Examples Be Parsed to Reveal Victim Model Information?
Authors: Yuguang Yao; Jiancheng Liu; Yifan Gong; Xiaoming Liu; Yanzhi Wang; Xue Lin; Sijia Liu
Venue: WACV 2025 (main)

Abstract: Numerous adversarial attack methods have been developed to generate imperceptible image perturbations that cause erroneous predictions in state-of-the-art machine learning (ML) models particularly deep neural networks (DNNs). Despite extensive research on adversarial examples limited efforts have been made to explore the hidden characteristics carried by these perturbations. In this study we investigate the feasibility of deducing information about the victim model (VM)--specifically characteristics such as architecture type kernel size activation function and weight sparsity--from adversarial examples. We approach this problem as a supervised learning task where we aim to attribute categories of VM characteristics to individual adversarial examples. To facilitate this we have assembled a dataset of adversarial attacks spanning seven types generated from 135 victim models systematically varied across five architecture types three kernel size configurations three activation functions and three levels of weight sparsity. We demonstrate that a supervised model parsing network (MPN) can effectively extract concealed details of the VM from adversarial examples. We also validate the practicality of this approach by evaluating the effects of various factors on parsing performance such as different input formats and generalization to out-of-distribution cases. Furthermore we highlight the connection between model parsing and attack transferability by showing how the MPN can uncover VM attributes in transfer attacks.
Aerial Mirage: Unmasking Hallucinations in Large Vision Language Models
Authors: Debolena Basak; Soham Bhatt; Sahith Kanduri; Maunendra Sankar Desarkar
Venue: WACV 2025 (main)

Abstract: Drones excel at capturing aerial-view images especially in human unreachable areas. Automatically interpreting and describing these images enables decision-making easier without the need to review the images extensively. Traditional image captioning models struggle with aerial imagery due to diverse orientations perspectives and unclear objects. Integrating the capabilities of Large Vision Language Models (LVLMs) with drone images can improve description utility benefiting strategic missions like surveillance search and rescue etc. However the lack of image-caption datasets for drone imagery poses a significant challenge for training and evaluating drone image captioning. To address this gap we contribute the first Aerial-view Image Captioning dataset (AeroCaps) containing four captions per image. Another major hurdle for the task is the hallucinatory nature of LVLMs. To this end we perform the first extensive analysis of hallucinations on aerial imagery by two SOTA LVLMs - LLaVA and InstructBLIP on our proposed dataset and VisDrone. We explore the reasons behind such hallucinations. We release the LVLM-generated image captions along with our hallucination-labelled annotations as the Labelled Illusion Dataset (LID) for further research. Additionally we review how effective advanced LLMs like GPT-4 are in evaluating the degree of hallucinations made by other LVLMs like LLaVA.
Global-Guided Focal Neural Radiance Field for Large-Scale Scene Rendering
Authors: Mingqi Shao; Feng Xiong; Hang Zhang; Shuang Yang; Mu Xu; Wei Bian; Xueqian Wang
Venue: WACV 2025 (main)

Abstract: Neural radiance fields (NeRF) have recently been applied to render large-scale scenes. However their limited model capacity typically results in blurred rendering results. Existing large-scale NeRFs primarily address this limitation by partitioning the scene into blocks which are subsequently handled by separate sub-NeRFs. These sub-NeRFs trained from scratch and processed independently lead to inconsistencies in geometry and appearance across the scene. Consequently the rendering quality fails to exhibit significant improvement despite the expansion of model capacity. In this work we present global-guided focal neural radiance field (GF-NeRF) that achieves high-fidelity rendering of large-scale scenes. Our proposed GF-NeRF utilizes a two-stage (Global and Focal) architecture and a global-guided training strategy. The global stage obtains a continuous representation of the entire scene while the focal stage decomposes the scene into multiple blocks and further processes them with distinct sub-encoders. Leveraging this two-stage architecture sub-encoders only need fine-tuning based on the global encoder thus reducing training complexity in the focal stage while maintaining scene-wide consistency. Spatial information and error information from the global stage also benefit the sub-encoders to focus on crucial areas and effectively capture more details of large-scale scenes. Notably our approach does not rely on any prior knowledge about the target scene attributing GF-NeRF adaptable to various large-scale scene types including street-view and aerial-view scenes. We demonstrate that our method achieves high-fidelity natural rendering results on various types of large-scale datasets. Our project page: https://shaomq2187.github.io/GF-NeRF/
SAND: Enhancing Open-Set Neuron Descriptions through Spatial Awareness
Authors: Anvita Agarwal Srinivas; Tuomas Oikarinen; Divyansh Srivastava; Wei-Hung Weng; Tsui-Wei Weng
Venue: WACV 2025 (main)

Abstract: We propose Spatially-Aware open-set Network Dissection (SAND) a technique to identify and label the learned representation of the neurons of deep vision networks. Contrary to earlier open-vocabulary neuron explanation methods we also leverage a neuron's spatial pattern of activation to guide our predictions towards more accurate and relevant concepts while avoiding being misled by confounding visual information. We highlight important regions for a neuron through image masking which has the advantage of being able to block out irrelevant concepts from an image handling irregularly shaped activation regions and revealing the visual concepts that a neuron learns in order to identify objects. We use CLIP to connect highly activating image regions with descriptive concepts and measure the quality of our results through human evaluation. Further since such manual evaluation can be highly time consuming costly and unscalable we also propose an automated approach which uses image generation to get quantitative feedback on the generated concepts. Finally as an application of our interpretability method we demonstrate how it can be tuned to the medical domain. Our code is available at https://github.com/Trustworthy-ML-Lab/SAND.
When Visual State Space Model Meets Backdoor Attacks
Authors: Sankalp Nagaonkar; Achyut Mani Tripathi; Ashish Mishra
Venue: WACV 2025 (main)

Abstract: The recently proposed Visual State Space Model (VMamba) operating on the principle of state space mechanisms (SSM) processes images as a sequence of patches and outperforms Vision Transformers (ViT) in several computer vision tasks. Given their substantial design differences from CNNs and ViT it is crucial to investigate their vulnerability to backdoor attacks and the impact of various advanced backdoor attacks on their robustness. Backdoor attacks involve embedding a specific trigger into a small subset of training images which remains dormant until activated later. While the model performs well on clean test images an attacker can manipulate its decisions by presenting the trigger in one of the test images. This work examines state-of-the-art (SOTA) vision architectures (ResNet ViT MLP-mixer VMamba) focusing on their susceptibility to backdoor attacks and the effect of different backdoor attacks on their robustness. The well-known Visual State Space Model (VMamba) is the least susceptible to backdoor attacks among these architectures. To address this in this paper we propose two novel QR decomposition-based backdoor attacks that are visually imperceptible and achieve a high attack success rate (ASR) against the VMamba. We also present a qualitative analysis of the proposed backdoor attacks explaining the reasons behind their success or failure against the VMamba model. Experiments and results conducted on two popular image datasets (CIFAR-10 and ImageNet-1K) demonstrate that the proposed backdoor attacks exceed the performance of SOTA backdoor attacks and effectively fool the recently proposed VMamba model.
ARF-Plus: Controlling Perceptual Factors in Artistic Radiance Fields for 3D Scene Stylization
Authors: Wenzhao Li; Tianhao Wu; Fangcheng Zhong; A. Cengiz Oztireli
Venue: WACV 2025 (main)

Abstract: The radiance fields style transfer is an emerging field that has recently gained popularity as a means of 3D scene stylization thanks to the outstanding performance of neural radiance fields in 3D reconstruction and view synthesis. We highlight a research gap in radiance fields style transfer the need for sufficient perceptual controllability motivated by the existing concept in the 2D image style transfer. In this paper we present ARF-Plus a unique 3D neural style transfer framework offering manageable control over perceptual factors to systematically explore the perceptual controllability in 3D scene stylization. Four distinct types of controls - color preservation control (style pattern) scale control spatial (selective stylization area) control and depth enhancement control - come with our proposed novel loss functions and strategies seamlessly integrated into this framework. This unlocks a realm of limitless possibilities allowing customized modifications of stylization effects and flexible merging of the strengths of different styles ultimately enabling the creation of novel and eye-catching stylistic effects on 3D scenes.
Token Turing Machines are Efficient Vision Models
Authors: Purvish Jajal; Nick Eliopoulous; Benjamin Shiue-Hal Chou; George K. Thiravathukal; James C. Davis; Yung-Hsiang Lu
Venue: WACV 2025 (main)

Abstract: We propose Vision Token Turing Machines (ViTTM) an efficient low-latency memory-augmented Vision Transformer (ViT). Our approach builds on Neural Turing Machines (NTM) and Token Turing Machines (TTM) which were applied to NLP and sequential visual understanding tasks. ViTTMs are designed for non-sequential computer vision tasks such as image classification and segmentation. Our model creates two sets of tokens: process tokens and memory tokens; process tokens pass through encoder blocks and read-write from memory tokens at each encoder block in the network allowing them to store and retrieve information from memory. By ensuring that there are fewer process tokens we are able to reduce the inference time of the network while maintaining its accuracy. On ImageNet-1K the state-of-the-art ViT-B has median latency of 529.5ms and 81.0% accuracy while our ViTTM-B is 56% faster (234.1ms) with 2.4x fewer FLOPs with an accuracy of 82.9%. On ADE20K semantic segmentation ViT-B achieves 45.65mIoU at 13.8 frame-per-second (FPS) whereas our ViTTM-B model acheives a 45.17 mIoU with 26.8 FPS (+94%).
ReBotNet: Fast Real-Time Video Enhancement
Authors: Jeya Maria Jose Valanarasu; Rahul Garg; Andeep Toor; Xin Tong; Weijuan Xi; Andreas Lugmayr; Vishal M. Patel; Anne Menini
Venue: WACV 2025 (main)

Abstract: Most video restoration networks are slow have high computational load and can't be used for real-time video enhancement. In this work we design an efficient and fast framework to perform real-time video enhancement for practical use-cases like live video calls and video streams. Our proposed method called Recurrent Bottleneck Mixer Network (ReBotNet) employs a dual-branch framework. The first branch learns spatio-temporal features by tokenizing the input frames along the spatial and temporal dimensions using a ConvNext-based encoder and processing these abstract tokens using a bottleneck mixer. To further improve temporal consistency the second branch employs a mixer directly on tokens extracted from individual frames. A common decoder then merges the features form the two branches to predict the enhanced frame. In addition we use a recurrent training approach where the last frame's prediction is leveraged to efficiently enhance the current frame while improving temporal consistency. To evaluate our method we curate two new datasets that emulate real-world video call and streaming scenarios and show extensive results on multiple datasets where ReBotNet outperforms existing approaches with lower computations reduced memory requirements and faster inference time. Code: https://github.com/jeya-mariajose/rebot-net.
UniTMGE: Uniform Text-Motion Generation and Editing Model via Diffusion
Authors: Ruoyu Wang; Yangfan He; Tengjiao Sun; Xiang Li; Tianyu Shi
Venue: WACV 2025 (main)

Abstract: Current methods have shown promising results in applying diffusion models to motion generation given text input. However these methods are limited to unimodal inputs and outputs restricted to motion generation alone and lacking multimodal control capabilities. To address these issues we introduce TMMGE a text-motion multimodal generation and editing framework based on diffusion. TMMGE overcomes single-modality limitations enabling exceptional performance and strong generalization across multiple tasks like text-driven motion generation motion captioning motion completion and multi-modal motion editing. TMMGE comprises three components: UTMV for mapping text and motion into a shared latent space using contrastive learning a controllable diffusion model customized for the UTMV space and MCRE for unifying multimodal conditions into CLIP representations enabling precise multimodal control and flexible motion editing through simple linear operations. We conducted both closed-world experiments and open-world experiments using the Motion-X dataset with detailed text descriptions with results demonstrating our model's effectiveness and generalizability across multiple tasks.
Uncertainty-Aware Online Extrinsic Calibration: A Conformal Prediction Approach
Authors: Mathieu Cocheteux; Julien Moreau; Franck Davoine
Venue: WACV 2025 (main)

Abstract: Accurate sensor calibration is crucial for autonomous systems yet its uncertainty quantification remains underexplored. We present the first approach to integrate uncertainty awareness into online extrinsic calibration combining Monte Carlo Dropout with Conformal Prediction to generate prediction intervals with a guaranteed level of coverage. Our method proposes a framework to enhance existing calibration models with uncertainty quantification compatible with various network architectures. Validated on KITTI (RGB Camera-LiDAR) and DSEC (Event Camera-LiDAR) datasets we demonstrate effectiveness across different visual sensor types measuring performance with adapted metrics to evaluate the efficiency and reliability of the intervals. By providing calibration parameters with quantifiable confidence measures we offer insights into the reliability of calibration estimates which can greatly improve the robustness of sensor fusion in dynamic environments and usefully serve the Computer Vision community.
Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentation
Authors: Sina Hajimiri; Ismail Ben Ayed; Jose Dolz
Venue: WACV 2025 (main)

Abstract: Despite the significant progress in deep learning for dense visual recognition problems such as semantic segmentation traditional methods are constrained by fixed class sets. Meanwhile vision-language foundation models such as CLIP have showcased remarkable effectiveness in numerous zero-shot image-level tasks owing to their robust generalizability. Recently a body of work has investigated utilizing these models in open-vocabulary semantic segmentation (OVSS). However existing approaches often rely on impractical supervised pre-training or access to additional pre-trained networks. In this work we propose a strong baseline for training-free OVSS termed Neighbour-Aware CLIP (NACLIP) representing a straightforward adaptation of CLIP tailored for this scenario. Our method enforces localization of patches in the self-attention of CLIP's vision transformer which despite being crucial for dense prediction tasks has been overlooked in the OVSS literature. By incorporating design choices favouring segmentation our approach significantly improves performance without requiring additional data auxiliary pre-trained networks or extensive hyperparameter tuning making it highly practical for real-world applications. Experiments are performed on 8 popular semantic segmentation benchmarks yielding state-of-the-art performance on most scenarios. Our code is publicly available at https://github.com/sinahmr/NACLIP.
Facial Expression Recognition with Controlled Privacy Preservation and Feature Compensation
Authors: Feng Xu; David Ahmedt-Aristizabal; Lars Petersson; Dadong Wang; Xun Li
Venue: WACV 2025 (main)

Abstract: Facial expression recognition (FER) systems raise significant privacy concerns due to the potential exposure of sensitive identity information. This paper presents a study on removing identity information while preserving FER capabilities. Drawing on the observation that low-frequency components predominantly contain identity information and high-frequency components capture expression we propose a novel two-stream framework that applies privacy enhancement to each component separately. We introduce a controlled privacy enhancement mechanism to optimize performance and a feature compensator to enhance task-relevant features without compromising privacy. Furthermore we propose a novel privacy-utility trade-off providing a quantifiable measure of privacy preservation efficacy in closed-set FER tasks. Extensive experiments on the benchmark CREMA-D dataset demonstrate that our framework achieves 78.84% recognition accuracy with a privacy (facial identity) leakage ratio of only 2.01% highlighting its potential for secure and reliable video-based FER applications.
Learning to Identify Seen Unseen and Unknown in the Open World: A Practical Setting for Zero-Shot Learning
Authors: Sethupathy Parameswaran; Yuan Fang; Chandan Gautam; Savitha Ramasamy; Xiaoli Li
Venue: WACV 2025 (main)

Abstract: As vision-language models advance addressing the Zero-Shot Learning (ZSL) problem in the open world becomes increasingly crucial. Specifically a robust model must handle three types of samples during inference: seen classes with visual and semantic information provided in training unseen classes with only the semantic information in training and unknown samples with no prior information from training. Existing methods either handle seen and unseen classes together (ZSL) or seen and unknown classes (known as Open-Set Recognition OSR). However none addresses the simultaneous handling of all three which we term Open-Set Zero-Shot Learning (OZSL). To address this problem we propose a two-stage approach for OZSL that recognizes seen unseen and unknown samples. The first stage classifies samples as either seen or not while the second stage distinguishes unseen from unknown. Furthermore we introduce a cross-stage knowledge transfer mechanism that leverages semantic relationships between seen and unseen classes to enhance learning in the second stage. Extensive experiments demonstrate the efficacy of the proposed approach compared to naively combining existing ZSL and OSR methods. The code is available at https://github.com/smufang/OZSL.
Multi-Aperture Transformers for 3D (MAT3D) Segmentation of Clinical and Microscopic Images
Authors: Muhammad Sohaib; Siyavash Shabani; Sahar A. Mohammed; Garrett Winkelmaier; Bahram Parvin
Venue: WACV 2025 (main)

Abstract: 3D segmentation of biological structures is critical in biomedical imaging offering significant insights into structures and functions. This paper introduces a novel segmentation of biological images that couples Multi-Aperture representation with Transformers for 3D (MAT3D) segmentation. Our method integrates the global context-awareness of Transformer networks with the local feature extraction capabilities of Convolutional Neural Networks (CNNs) providing a comprehensive solution for accurately delineating complex biological structures. First we evaluated the performance of the proposed technique on two public clinical datasets of ACDC and Synapse multi-organ segmentation rendering superior Dice scores of 93.34+-0.05 and 89.73+-0.04 respectively with fewer parameters compared to the published literature. Next we assessed the performance of our technique on an organoid dataset comprising four breast cancer subtypes. The proposed method achieved a Dice 95.12+-0.02 and a PQ score of 97.01+-0.01 respectively. MAT3D also significantly reduces the parameters to 40 million. The code is available on https://github.com/sohaibcs1/MAT3D.
Harmonizing Attention: Training-Free Texture-Aware Geometry Transfer
Authors: Eito Ikuta; Yohan Lee; Akihiro Iohara; Yu Saito; Toshiyuki Tanaka
Venue: WACV 2025 (main)

Abstract: Creating images where surface patterns of one object - such as cracks holes or grooves - are precisely transferred onto objects made of different materials remains a challenging task in computer graphics. For example recreating the exact pattern of wood grain cracks on a metallic surface while maintaining the realistic metallic texture requires sophisticated technical solutions. In this study we introduce Harmonizing Attention a new method that can automatically extract these surface patterns from photographs and recreate them with different materials while preserving natural-looking textures. Our approach achieves this through a novel attention mechanism that can process multiple reference images simultaneously without requiring additional training. This makes the method both practical and efficient for real-world applications opening up new possibilities in augmented reality image editing and beyond.
Towards Generalized Face Anti-Spoofing from a Frequency Shortcut View
Authors: Junyi Cao; Chao Ma
Venue: WACV 2025 (main)

Abstract: The generalization capability of a face anti-spoofing (FAS) model is critical to its practicality in the real world. Recent studies have theoretically and empirically uncovered that neural networks tend to exploit easy-to-learn frequency sets for decisions. These simplicity-biased representations depending on what best simplifies the training objective may hamper generalization. This paper thus focuses on mitigating the frequency shortcut learning of prior FAS models for improved generalization. Specifically we introduce a frequency-aware autoencoder to retain more frequency details in intermediate features via reconstruction facilitating comprehensive judgment of FAS. Based on the encoder output we propose a dynamic frequency masking mechanism to select and suppress the probable shortcut bands during training enabling broader horizons on under-explored frequencies. Moreover we employ a style inhibited modulation to weaken stylized information in frequency space to reduce the reliance on spurious style features. Experiment results on generalized FAS benchmarks verify the superiority of our framework over existing methods. Our code has been integrated into this project: https://github.com/VISION-SJTU/UniDefense.