GenAI4ELab/papercli-papers huggingface dataset:
Beyond Grids: Exploring Elastic Input Sampling for Vision Transformers
Authors: Adam Pardyl; Grzegorz Kurzejamski; Jan Olszewski; Tomasz Trzcinski; Bartosz Zielinski
Venue: WACV 2025 (main)

Abstract: Vision transformers have excelled in various computer vision tasks but mostly rely on rigid input sampling using a fixed-size grid of patches. It limits their applicability in real-world problems such as active visual exploration where patches have various scales and positions. Our paper addresses this limitation by formalizing the concept of input elasticity for vision transformers and introducing an evaluation protocol for measuring this elasticity. Moreover we propose modifications to the transformer architecture and training regime which increase its elasticity. Through extensive experimentation we spotlight opportunities and challenges associated with such architecture.
Pruning One More Token is Enough: Leveraging Latency-Workload Non-Linearities for Vision Transformers on the Edge
Authors: Nicholas John Eliopoulos; Purvish Jajal; James C. Davis; Gaowen Liu; George K. Thiruvathukal; Yung-Hsiang Lu
Venue: WACV 2025 (main)

Abstract: This paper investigates how to efficiently deploy vision transformers on edge devices for small workloads. Recent methods reduce the latency of transformer neural networks by removing or merging tokens with small accuracy degradation. However these methods are not designed with edge device deployment in mind: they do not leverage information about the latency-workload trends to improve efficiency. We address this shortcoming in our work. First we identify factors that affect ViT latency-workload relationships. Second we determine token pruning schedule by leveraging non-linear latency-workload relationships. Third we demonstrate a training-free token pruning method utilizing this schedule. We show other methods may increase latency by 2-30% while we reduce latency by 9-26%. For similar latency (within 5.2% or 7ms) across devices we achieve 78.6%-84.5% ImageNet1K classification accuracy while the state-of-the-art Token Merging achieves 45.8%-85.4%.
Reframing Image Difference Captioning with BLIP2IDC and Synthetic Augmentation
Authors: Gautier Evennou; Antoine Chaffin; Vivien Chappelier; Ewa Kijak
Venue: WACV 2025 (main)

Abstract: The rise of the generative models quality during the past years enabled the generation of edited variations of images at an important scale. To counter the harmful effects of such technology the Image Difference Captioning (IDC) task aims to describe the differences between two images. While this task is successfully handled for simple 3D rendered images it struggles on real-world images. The reason is twofold: the training data-scarcity and the difficulty to capture fine-grained differences between complex images. To address those issues we propose in this paper a simple yet effective framework to both adapt existing image captioning models to the IDC task and augment IDC datasets. We introduce BLIP2IDC an adaptation of BLIP2 to the IDC task at low computational cost and show it outperforms two-streams approaches by a significant margin on real-world IDC datasets. We also propose to use synthetic augmentation to improve the performance of IDC models in an agnostic fashion. We show that our synthetic augmentation strategy provides high quality data leading to a challenging new dataset well-suited for IDC named Syned. The code weights and dataset are available here https://github.com/gautierevn/BLIP2IDC.
EfficientMorph: Parameter-Efficient Transformer-Based Architecture for 3D Image Registration
Authors: Abu Zahid Bin Aziz; Mokshagna Sai Teja Karanam; Tushar Kataria; Shireen Y. Elhabian
Venue: WACV 2025 (main)

Abstract: Transformers have emerged as the state-of-the-art architecture in medical image registration outperforming convolutional neural networks (CNNs) by addressing their limited receptive fields and overcoming gradient instability in deeper models. Despite their success transformer-based models require substantial resources for training including data memory and computational power which may restrict their applicability for end users with limited resources. In particular existing transformer-based 3D image registration architectures face two critical gaps that challenge their efficiency and effectiveness. Firstly although window-based attention mechanisms reduce the quadratic complexity of full attention by focusing on local regions they often struggle to effectively integrate both local and global information. Secondly the granularity of tokenization a crucial factor in registration accuracy presents a performance trade-off: smaller voxel-size tokens enhance detail capture but come with increased computational complexity higher memory usage and a greater risk of overfitting. We present EFFICIENTMORPH a transformerbased architecture for unsupervised 3D image registration that balances local and global attention in 3D volumes through a plane-based attention mechanism and employs a Hi-Res tokenization strategy with merging operations thus capturing finer details without compromising computational efficiency. Notably EFFICIENTMORPH sets a new benchmark for performance on the OASIS dataset with  16-27x fewer parameters. https://github.com/MedVICLab/Efficient_Morph_Registration
PrevPredMap: Exploring Temporal Modeling with Previous Predictions for Online Vectorized HD Map Construction
Authors: Nan Peng; Xun Zhou; Mingming Wang; Xiaojun Yang; Songming Chen; Guisong Chen
Venue: WACV 2025 (main)

Abstract: Temporal information is crucial for detecting occluded instances. Existing temporal representations have progressed from BEV or PV (perspective view) features to more compact query features. Compared to these aforementioned features predictions offer the highest level of abstraction providing explicit information. In the context of online vectorized HD map construction this unique characteristic of predictions is potentially advantageous for long-term temporal modeling and the integration of map priors. This paper introduces PrevPredMap a pioneering temporal modeling framework that leverages previous predictions for constructing online vectorized HD maps. We have meticulously crafted two essential modules for PrevPredMap: the previous-predictions-based query generator and the dynamic-position-query decoder. Specifically the previous-predictions-based query generator is designed to separately encode different types of information from previous predictions which are then effectively utilized by the dynamic-position-query decoder to generate current predictions. Furthermore we have developed a dual-mode strategy to ensure PrevPredMap's robust performance across both single-frame and temporal modes. Extensive experiments demonstrate that PrevPredMap achieves state-of-the-art performance on the nuScenes and Argoverse2 datasets. Code is available at https://github.com/pnnnnnnn/PrevPredMap.
GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts
Authors: Zoltán Á. Milacski; Koichiro Niinuma; Ryosuke Kawamura; Fernando de la Torre; László A. Jeni
Venue: WACV 2025 (main)

Abstract: The connection between our 3D surroundings and the descriptive language that characterizes them would be well-suited for localizing and generating human motion in context but for one problem. The complexity introduced by multiple modalities makes capturing this connection challenging with a fixed set of descriptors. Specifically closed vocabulary scene encoders which require learning text-scene associations from scratch have been favored in the literature often resulting in inaccurate motion grounding. In this paper we propose a method that integrates an open vocabulary scene encoder into the architecture establishing a robust connection between text and scene. Our two-step approach starts with pretraining the scene encoder through knowledge distillation from an existing open vocabulary semantic image segmentation model ensuring a shared text-scene feature space. Subsequently the scene encoder is fine-tuned for conditional motion generation incorporating two novel regularization losses that regress the category and size of the goal object. Our methodology achieves up to a 30% reduction in the goal object distance metric compared to the prior state-of-the-art baseline model on the HUMANISE dataset. This improvement is demonstrated through evaluations conducted using three implementations of our framework a perceptual study and an open vocabulary experiment. Additionally our method is designed to accommodate future 2D open vocabulary segmentation methods for distillation in a plug-and-play manner.
Unified Framework for Open-World Compositional Zero-Shot Learning
Authors: Hirunima Jayasekara; Khoi Pham; Nirat Saini; Abhinav Shrivastava
Venue: WACV 2025 (main)

Abstract: Open-World Compositional Zero-Shot Learning (OW-CZSL) addresses the challenge of recognizing novel compositions of known primitives and entities. Even though prior works utilize language knowledge for recognition such approaches exhibit limited interactions between language-image modalities. Our approach primarily focuses on enhancing the inter-modality interactions through fostering richer interactions between image and textual data. Additionally we introduce a novel module aimed at alleviating the computational burden associated with exhaustive exploration of all possible compositions during the inference stage. While previous methods exclusively learn compositions jointly or independently we introduce an advanced hybrid procedure that leverages both learning mechanisms to generate final predictions. Our proposed model achieves state-of-the-art in OW-CZSL in three datasets while surpassing Large Vision Language Models (LLVM) in two datasets. Our code is available at https://github.com/hirunima/OWCZSL
An Image is Worth Multiple Words: Multi-Attribute Inversion for Constrained Text-to-Image Synthesis
Authors: Aishwarya Agarwal; Srikrishna Karanam; Tripti Shukla; Balaji Vasan Srinivasan
Venue: WACV 2025 (main)

Abstract: We consider the problem of constraining diffusion model outputs with a user-supplied reference image. Our key objective is to extract multiple attributes (e.g. color object layout style) from this single reference image and then generate new samples and novel compositions with them. We first perform an extensive attribute distribution analysis that leads to the discovery of an extended conditioning space consisting of multiple textual conditions. These textual conditions vary both per-layer of the U-Net as well as per-timestep of the denoising process. We observe that although the extended conditioning space provides greater control over different attributes of the generated image often a subset of these attributes are captured in the same set of U-Net layers and/or across same denoising timesteps. For instance color and style are captured across same U-Net layers whereas layout and color are captured across same timestep stages. Existing works in multi-attribute constrained image generation proposed extending textual inversion by learning per-layer or per-timestep tokens but they suffer from attribute entanglement for the reasons described above. To address the aforementioned gap we introduce our second contribution where we design a new multi-attribute textual inversion algorithm MATTE with associated disentanglement-enhancing regularization losses that operates jointly across both layer and timestep dimensions and explicitly leads to four disentangled tokens (color style layout and object). We conduct extensive qualitative and quantitative evaluations to demonstrate the effectiveness of the proposed approach.
Learning to Visually Connect Actions and their Effects
Authors: Paritosh Parmar; Eric Peh; Basura Fernando
Venue: WACV 2025 (main)

Abstract: We introduce the novel concept of visually Connecting Actions and Their Effects (CATE) in video understanding. CATE can have applications in areas like task planning and learning from demonstration. We identify and explore two different aspects of the concept of CATE: Action Selection (AS) and Effect-Affinity Assessment (EAA) where video understanding models connect actions and effects at semantic and fine-grained levels respectively. We design various baseline models for AS and EAA. Despite the intuitive nature of the task we observe that models struggle and humans outperform them by a large margin. Our experiments show that in solving AS and EAA models learn intuitive properties like object tracking and encoding pose-related features without explicit supervision. We demonstrate that CATE can be an effective self-supervised task for learning video representations from unlabeled videos. The study aims to showcase the fundamental nature and versatility of CATE with the hope of inspiring advanced formulations and models.
SwinIA: Self-Supervised Blind-Spot Image Denoising without Convolutions
Authors: Mikhail Papkov; Pavel Chizhov; Leopold Parts
Venue: WACV 2025 (main)

Abstract: Self-supervised image denoising implies restoring the signal from a noisy image without access to the ground truth. State-of-the-art solutions for this task rely on predicting masked pixels with a fully-convolutional neural network. This most often requires multiple forward passes information about the noise model or intricate regularization functions. In this paper we propose a Swin Transformer-based Image Autoencoder (SwinIA) the first fully-transformer architecture for self-supervised denoising. The flexibility of the attention mechanism helps to fulfill the blind-spot property that convolutional counterparts normally approximate. SwinIA can be trained end-to-end with a simple mean squared error loss without masking and does not require any prior knowledge about clean data or noise distribution. Simple to use SwinIA establishes the state of the art on several common benchmarks.
NarrAD: Automatic Generation of Audio Descriptions for Movies with Rich Narrative Context
Authors: Jaehyeong Park; Junchel Ye; Seungkook Lee; Hyun W. Ka; Dongsu Han
Venue: WACV 2025 (main)

Abstract: Audio Description (AD) is a narration designed to enhance accessibility for visually impaired individuals by conveying the key visual elements of a video. Thus automating AD generation for long-form videos such as movies and dramas provides high social value but is a challenging task. First AD must reflect the narrative context of the entire movie including the storyline names of characters and places and the cultural setting. Second to avoid disrupting the immersive experience of the movie AD must not overlap with the characters' dialogues requiring the delivery of numerous visual elements in concise sentences. This paper presents NarrAD a training-free AD generation framework that satisfies both of the requirements by leveraging rich narrative context in movie scripts and curating information across narration slots. Experiments on the MAD dataset demonstrate that our approach outperforms prior works in both captioning and LLM-based metrics. In the user study with 600 subjects NarrAD achieves the highest user experience and movie comprehension. NarrAD's AD samples are available at https://bit.ly/4aSwOTr.
Anchored Diffusion for Video Face Reenactment
Authors: Idan Kligvasser; Regev Cohen; George Leifman; Ehud Rivlin; Michael Elad
Venue: WACV 2025 (main)

Abstract: Video generation has drawn significant interest recently pushing the development of large-scale models capable of producing realistic videos with coherent motion. Due to memory constraints these models typically generate short video segments that are then combined into long videos. The merging process poses a significant challenge as it requires ensuring smooth transitions and overall consistency. In this paper we introduce Anchored Diffusion a novel method for synthesizing relatively long and seamless videos. We extend Diffusion Transformers (DiTs) to incorporate temporal information creating our sequence-DiT (sDiT) model for generating short video segments. Unlike previous works we train our model on video sequences with random non-uniform temporal spacing and incorporate temporal information via external guidance increasing flexibility and allowing it to capture both short and long-term relationships. Furthermore during inference we leverage the transformer architecture to modify the diffusion process generating a batch of non-uniform sequences anchored to a common frame ensuring consistency regardless of temporal distance. To demonstrate our method we focus on face reenactment a task of transforming the action from the driving video to the source face. Through comprehensive experiments we show our approach outperforms current techniques in producing longer consistent high-quality videos while offering editing capabilities.
Blind Image Deblurring with FFT-ReLU Sparsity Prior
Authors: Abdul Mohaimen Al Radi; Prothito Shovon Majumder; Md. Mosaddek Khan
Venue: WACV 2025 (main)

Abstract: Blind image deblurring is the process of recovering a sharp image from a blurred one without prior knowledge about the blur kernel. It is a small data problem since the key challenge lies in estimating the unknown degrees of blur from a single image or limited data instead of learning from large datasets. The solution depends heavily on developing algorithms that effectively model the image degradation process. We introduce a method that leverages a prior which targets the blur kernel to achieve effective deblurring across a wide range of image types. In our extensive empirical analysis our algorithm achieves results that are competitive with the state-of-the-art blind image deblurring algorithms and it offers up to two times faster inference making it a highly efficient solution.
MIP-GAF: A MLLM-Annotated Benchmark for Most Important Person Localization and Group Context Understanding
Authors: S. Madan; S. Ghosh; L. R. Sookha; M.A. Ganaie; R. Subramanian; A. Dhall; T. Gedeon
Venue: WACV 2025 (main)

Abstract: Estimating the Most Important Person (MIP) in any social event setup is a challenging problem mainly due to contextual complexity and scarcity of labeled data. Moreover the causality aspects of MIP estimation are quite subjective and diverse. To this end we aim to address the problem by annotating a large-scale 'in-the-wild' dataset for identifying human perceptions about the 'Most Important Person (MIP)' in an image. The paper provides a thorough description of our proposed Multimodal Large Language Model (MLLM) based data annotation strategy and a thorough data quality analysis. Further we perform a comprehensive benchmarking of the proposed dataset utilizing state-of-the-art MIP localization methods indicating a significant drop in performance compared to existing datasets. The performance drop shows that the existing MIP localization algorithms must be more robust with respect to 'in-the-wild' situations. We believe the proposed dataset will play a vital role in building the next-generation social situation understanding methods. The dataset and associated code will be made available for research purposes.
Difficulty Diversity and Plausibility: Dynamic Data-Free Quantization
Authors: Cheeun Hong; Sungyong Baik; Junghun Oh; Kyoung Mu Lee
Venue: WACV 2025 (main)

Abstract: Without access to the original training data data-free quantization (DFQ) aims to recover the performance loss induced by quantization. Most previous works have focused on using an original network to extract the train data information which is instilled into surrogate synthesized images. However existing DFQ methods do not take into account important aspects of quantization: the extent of a computational-cost-and-accuracy trade-off varies for each image depending on its task difficulty. To handle such varying trade-offs several efforts have been made to dynamically allocate bit-widths for each image. Such dynamic quantization however remains challenging and unexplored in the data-free domain because synthesized images of previous works fail to possess properties in natural test images that are crucial for learning the appropriate dynamic allocation policy: difficulty its diversity and its plausibility. By contrast we propose a data-free quantization framework that is dynamic-friendly by modeling varying extents of task difficulties with plausibility. We generate plausibly difficult images with soft labels whose probabilities are allocated to a group of similar classes. Images with diverse and plausible difficulties enable us to train the framework to dynamically handle the varying trade-offs. Consequently our framework achieves better accuracy-complexity Pareto front than existing data-free quantization approaches.
Uncertainty and Energy Based Loss Guided Semi-Supervised Semantic Segmentation
Authors: Rini Smita Thakur; Vinod K Kurmi
Venue: WACV 2025 (main)

Abstract: Semi-supervised (SS) semantic segmentation exploits both labeled and unlabeled images to overcome tedious and costly pixel-level annotation problems. Pseudolabel supervision is one of the core approaches of training networks with both pseudo labels and ground-truth labels. This work uses aleatoric or data uncertainty and energy based modeling in intersection-union pseudo supervised network. The aleatoric uncertainty is modeling the inherent noise variations of the data in a network with two predictive branches. The per-pixel variance parameter obtained from the network gives a quantitative idea about the data uncertainty. Moreover energy-based loss realizes the potential of generative modeling on the downstream SS segmentation task. The aleatoric and energy loss are applied in conjunction with pseudo-intersection labels pseudo-union labels and ground-truth on the respective network branch. The comparative analysis with state-of-the-art methods has shown improvement in performance metrics. The code is availaible at https://visdomlab.github.io/DUEB/.
BIV-Priv-Seg: Locating Private Content in Images Taken by People with Visual Impairments
Authors: Yu-Yun Tseng; Tanusree Sharma; Lotus Zhang; Abigale Stangl; Leah Findlater; Yang Wang; Danna Gurari
Venue: WACV 2025 (main)

Abstract: Individuals who are blind or have low vision (BLV) are at a heightened risk of sharing private information if they share photographs they have taken. To facilitate developing technologies that can help them preserve privacy we introduce BIV-Priv-Seg the first localization dataset originating from people with visual impairments that shows private content. It contains 1028 images with segmentation annotations for 16 private object categories. We first characterize BIV-Priv-Seg and then evaluate modern models' performance for locating private content in the dataset. We find modern models struggle most with locating private objects that are not salient small and lack text as well as recognizing when private content is absent from an image. We facilitate future extensions by sharing our new dataset with the evaluation server at https://vizwiz.org/tasks-and-datasets/object-localization/.