GenAI4ELab/papercli-papers huggingface dataset:
TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigm
Authors: Bingqing Zhang; Zhuo Cao; Heming Du; Xin Yu; Xue Li; Jiajun Liu; Sen Wang
Venue: WACV 2025 (main)

Abstract: Text-Video Retrieval (TVR) methods typically match query-candidate pairs by aligning text and video features in coarse-grained fine-grained or combined (coarse-to-fine) manners. However these frameworks predominantly employ a one(query)-to-one(candidate) alignment paradigm which struggles to discern nuanced differences among candidates leading to frequent mismatches. Inspired by Comparative Judgement in human cognitive science where decisions are made by directly comparing items rather than evaluating them independently we propose TokenBinder. This innovative two-stage TVR framework introduces a novel one-to-many coarse-to-fine alignment paradigm imitating the human cognitive process of identifying specific items within a large collection. Our method employs a Focused-view Fusion Network with a sophisticated cross-attention mechanism dynamically aligning and comparing features across multiple videos to capture finer nuances and contextual variations. Extensive experiments on six benchmark datasets confirm that TokenBinder substantially outperforms existing state-of-the-art methods. These results demonstrate its robustness and the effectiveness of its fine-grained alignment in bridging intra- and inter-modality information gaps in TVR tasks.
@BENCH: Benchmarking Vision-Language Models for Human-Centered Assistive Technology
Authors: Xin Jiang; Junwei Zheng; Ruiping Liu; Jiahang Li; Jiaming Zhang; Sven Matthiesen; Rainer Stiefelhagen
Venue: WACV 2025 (main)

Abstract: As Vision-Language Models (VLMs) advance human-centered Assistive Technologies (ATs) for helping People with Visual Impairments (PVIs) are evolving into generalists capable of performing multiple tasks simultaneously. However benchmarking VLMs for ATs remains under-explored. To bridge this gap we first create a novel AT benchmark (@BENCH). Guided by a pre-design user study with PVIs our benchmark includes the five most crucial vision-language tasks: Panoptic Segmentation Depth Estimation Optical Character Recognition (OCR) Image Captioning and Visual Question Answering (VQA). Besides we propose a novel AT model (@MODEL) that addresses all tasks simultaneously and can be expanded to more assistive functions for helping PVIs. Our framework exhibits outstanding performance across tasks by integrating multi-modal information and it offers PVIs a more comprehensive assistance. Extensive experiments prove the effectiveness and generalizability of our framework.
VioPose: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inference
Authors: Seong Jong Yoo; Snehesh Shrestha; Irina Muresanu; Cornelia Fermuller
Venue: WACV 2025 (main)

Abstract: Musicians delicately control their bodies to generate music. Sometimes their motions are too subtle to be captured by the human eye. To analyze how they move to produce the music we need to estimate precise 4D human pose (3D pose over time). However current state-of-the-art (SoTA) visual pose estimation algorithms struggle to produce accurate monocular 4D poses because of occlusions partial views and human-object interactions. They are limited by the viewing angle pixel density and sampling rate of the cameras and fail to estimate fast and subtle movements such as in the musical effect of vibrato. We leverage the direct causal relationship between the music produced and the human motions creating them to address these challenges. We propose VioPose: a novel multimodal network that hierarchically estimates dynamics. High-level features are cascaded to low-level features and integrated into Bayesian updates. Our architecture is shown to produce accurate pose sequences facilitating precise motion analysis and outperforms SoTA. As part of this work we collected the largest and the most diverse calibrated violin-playing dataset including video sound and 3D motion capture poses. Code and dataset can be found in our project page https://sj-yoo.info/viopose/.
Event-Guided Video Transformer for End-to-End 3D Human Pose Estimation
Authors: Bo Lang; Mooi Choo Chuah
Venue: WACV 2025 (main)

Abstract: 3D human pose estimation (3D HPE) is an important computer vision task with various practical applications. However 3D pose estimation for multi-person from a monocular video (3DMPPE) is particularly challenging. Recent transformer-based approaches focus on capturing the spatial-temporal information from sequential 2D poses which unfortunately loses the visual feature relevant for 3D pose estimation. In this paper we propose an end-to-end framework called Event Guided Video Transformer (EVT) which predicts 3D poses directly from video frames by learning spatial-temporal contextual information from visual features effectively. In addition our design is the first that incorporates event features to help guide 3D pose estimation. EVT first decouples persons into different instance-aware feature maps from video frames. These features containing specific clues of body structure information are then fed together with event features into an attention based Event-Aware Embedding Module. Next the fused features for each instance are then fed into an intra-human relation extraction module and subsequently to a temporal transformer to extract inter-frame relationship. Finally the extracted features are fed into a decoder for 3D pose estimation. Experiments using three widely used 3D pose estimation benchmarks show that our proposed EVT achieves better performance than state-of-the-art models.
GlobalDoc: A Cross-Modal Vision-Language Framework for Real-World Document Image Retrieval and Classification
Authors: Souhail Bakkali; Sanket Biswas; Zuheng Ming; Mickaël Coustaty; Marçal Rusiñol; Oriol Ramos Terrades; Josep Lladós
Venue: WACV 2025 (main)

Abstract: Visual document understanding (VDU) has rapidly advanced with the development of powerful multi-modal language models. However these models typically require extensive document pre-training data to learn intermediate representations and often suffer a significant performance drop in real-world online industrial settings. A primary issue is their heavy reliance on OCR engines to extract local positional information within document pages which limits the models' ability to capture global information and hinders their generalizability flexibility and robustness. In this paper we introduce GlobalDoc a cross-modal transformer-based architecture pre-trained in a self-supervised manner using three novel pretext objective tasks. GlobalDoc improves the learning of richer semantic concepts by unifying language and visual representations resulting in more transferable models. For proper evaluation we also propose two novel document-level downstream VDU tasks Few-Shot Document Image Classification (DIC) and Content-based Document Image Retrieval (DIR) designed to simulate industrial scenarios more closely. Extensive experimentation has been conducted to demonstrate GlobalDoc's effectiveness in practical settings.
TRH2TQA: Table Recognition with Hierarchical Relationships to Table Question-Answering on Business Table Images
Authors: Pongsakorn Jirachanchaisiri; Nam Tuan Ly; Atsuhiro Takasu
Venue: WACV 2025 (main)

Abstract: Despite advancements in visual question answering challenges persist with documents like financial reports often structured in complicated tabular structures with complex numerical computations. An alternative approach the pipeline-driven methodology includes table recognition (TR) and table question-answering (TQA). Recent advancements in TR support this approach with better accuracy and interpretability. However real-world tables usually represent hierarchical tables. They pose additional challenges due to merged cells and indents necessitating a specific approach for hierarchical relationship extraction. In this paper we propose TRH2TQA (Table Recognition with Hierarchical Relationships to Table Question-Answering) for business table images. It consists of three modules on table images with question-answer pairs. First the TR module extracts structure and textual content from table images into HTML format. Second post-structure extraction is applied to identify header and hierarchical relationships using predicted column span and bounding box. Finally this information is combined with natural language questions in the TQA module to generate the answer through the decoder. In extensive experiments TRH2TQA outperforms in question-answering performance on the VQAonBD 2023 dataset.
Beta Sampling is All You Need: Efficient Image Generation Strategy for Diffusion Models using Stepwise Spectral Analysis
Authors: Haeil Lee; Hansang Lee; Seoyeon Gye; Junmo Kim
Venue: WACV 2025 (main)

Abstract: Generative diffusion models have emerged as a powerful tool for high-quality image synthesis yet their iterative nature demands significant computational resources. This paper proposes an efficient time step sampling method based on an image spectral analysis of the diffusion process aimed at optimizing the denoising process. Instead of the traditional uniform distribution-based time step sampling we introduce a Beta distribution-like sampling technique that prioritizes critical steps in the early and late stages of the process. Our hypothesis is that certain steps exhibit significant changes in image content while others contribute minimally. We validated our approach using Fourier transforms to measure frequency response changes at each step revealing substantial low-frequency changes early on and high-frequency adjustments later. Experiments with ADM and Stable Diffusion demonstrated that our Beta Sampling method consistently outperforms uniform sampling achieving better FID and IS scores and offers competitive efficiency relative to state-of-the-art methods like AutoDiffusion. This work provides a practical framework for enhancing diffusion model efficiency by focusing computational resources on the most impactful steps with potential for further optimization and broader application.
GAUDA: Generative Adaptive Uncertainty-Guided Diffusion-Based Augmentation for Surgical Segmentation
Authors: Yannik Frisch; Christina Bornberg; Moritz Fuchs; Anirban Mukhopadhyay
Venue: WACV 2025 (main)

Abstract: Augmentation by generative modelling yields a promising alternative to the accumulation of surgical data where ethical organisational and regulatory aspects must be considered. Yet the joint synthesis of (image mask) pairs for segmentation a major application in surgery is rather unexplored. We propose to learn semantically comprehensive yet compact latent representations of the (image mask) space which we jointly model with a Latent Diffusion Model. We show that our approach can effectively synthesise unseen high-quality paired segmentation data of remarkable semantic coherence. Generative augmentation is typically applied pre-training by synthesising a fixed number of additional training samples to improve downstream task models. To enhance this approach we further propose Generative Adaptive Uncertainty-guided Diffusion-based Augmentation (GAUDA) leveraging the epistemic uncertainty of a Bayesian downstream model for targeted online synthesis. We condition the generative model on classes with high estimated uncertainty during training to produce additional unseen samples for these classes. By adaptively utilising the generative model online we can minimise the number of additional training samples and centre them around the currently most uncertain parts of the data distribution. GAUDA effectively improves downstream segmentation results over comparable methods by an average absolute IoU of 1.6% on CaDISv2 and 1.5% on CholecSeg8k two prominent surgical datasets for semantic segmentation.
Multi-Task Learning of Classification and Generation for Set-Structured Data
Authors: Fumioki Sato; Hideaki Hayashi; Hajime Nagahara
Venue: WACV 2025 (main)

Abstract: In this study we propose a multi-task learning model of classification and generation for set-structured data. The proposed model learns data generation and classification in a single neural network by integrating a classification layer into a variational autoencoder while maintaining permutation invariance and equivariance nature which are characteristics of set-structured data. The proposed model allows for semi-supervised learning in set-structured data classification and can also be applied to confidence calibration using the input data distribution estimated by the generative model. In the experiments we evaluated the performance of the proposed model in a semi-supervised classification task on set-structured datasets and compared it with a baseline model consisting only of a classifier. The results demonstrated that simultaneous learning of the classification and generation effectively improves the classification accuracy and confidence reliability for set-structured data even with a limited number of labeled data.
TaCOS: Task-Specific Camera Optimization with Simulation
Authors: Chengyang Yan; Donald G. Dansereau
Venue: WACV 2025 (main)

Abstract: The performance of perception tasks is heavily influenced by imaging systems. However designing cameras with high task performance is costly requiring extensive camera knowledge and experimentation with physical hardware. Additionally cameras and perception tasks are mostly designed in isolation whereas recent methods that jointly design cameras and tasks have shown improved performance. Therefore we present a novel end-to-end optimization approach that co-designs cameras with specific vision tasks. This method combines derivative-free and gradient-based optimizers to support both continuous and discrete camera parameters within manufacturing constraints. We leverage recent computer graphics techniques and physical camera characteristics to simulate the cameras in virtual environments making the design process cost-effective. We validate our simulations against physical cameras and provide a procedurally generated virtual environment. Our experiments demonstrate that our method designs cameras that outperform common off-the-shelf options and more efficiently compared to the state-of-the-art approach requiring only 2 minutes to design a camera on an example experiment compared with 67 minutes for the competing method. Designed to support the development of cameras under manufacturing constraints multiple cameras and unconventional cameras we believe this approach can advance the fully automated design of cameras. Code is available on our project page at https://roboticimaging.org/Projects/TaCOS/.
SAM-Mamba: Mamba Guided SAM Architecture for Generalized Zero-Shot Polyp Segmentation
Authors: Tapas Kumar Dutta; Snehashis Majhi; Deepak Ranjan Nayak; Debesh Jha
Venue: WACV 2025 (main)

Abstract: Polyp segmentation in colonoscopy images is crucial for detecting colorectal cancer but is challenging due to variations in the structure color and size of polyps as well as the lack of clear boundaries with surrounding tissues. Traditional segmentation models based on Convolutional Neural Networks (CNNs) struggle to capture detailed patterns and global context limiting their performance. Vision Transformer (ViT)-based models address some of these issues but have difficulties in capturing local context and lack strong zero-shot generalization. To this end we propose the Mamba-guided Segment Anything Model (SAM-Mamba) for efficient polyp segmentation. Our approach introduces a Mamba-Prior module in the encoder to bridge the gap between the general pre-trained representation of SAM and polyp-relevant trivial clues. It injects salient cues of polyp images into the SAM image encoder as a domain prior while capturing global dependencies at various scales leading to more accurate segmentation results. Extensive experiments on five benchmark datasets show that SAM-Mamba outperforms traditional CNN ViT and Adapter-based models in both quantitative and qualitative measures. Additionally SAM-Mamba demonstrates excellent adaptability to unseen datasets making it highly suitable for real-time clinical use.
KDC-MAE: Knowledge Distilled Contrastive Mask Auto-Encoder
Authors: Maheswar Bora; Saurabh Atreya; Aritra Mukherjee; Abhijit Das
Venue: WACV 2025 (main)

Abstract: In this work we attempted to extend the thought and showcase a way forward for the Self-supervised Learning (SSL) learning paradigm by combining contrastive learning self-distillation (knowledge distillation) and masked data modelling the three major SSL frameworks to learn a joint and coordinated representation. The proposed technique of SSL learns by the collaborative power of different learning objectives of SSL. Hence to jointly learn the different SSL objectives we proposed a new SSL architecture KDC-MAE a complementary masking strategy to learn the modular correspondence and a weighted way to combine them coordinately. Experimental results conclude that the contrastive masking correspondence along with the KD learning objective has lent a hand to performing better learning for multiple modalities over multiple tasks.
Corgi: Cached Memory Guided Video Generation
Authors: Xindi Wu; Uriel Singer; Zhaojiang Lin; Andrea Madotto; Xide Xia; Yifan Xu; Paul Crook; Xin Luna Dong; Seungwhan Moon
Venue: WACV 2025 (main)

Abstract: Text-to-Video generation has achieved remarkable progress with the rise of diffusion models. In this work we introduce Cached Memory-Guided Video Generation (Corgi) aiming to generate multi-scene videos with arbitrary number of video clips conditioned on input images and instruction prompts. This is a challenging task as traditional T2V methods often struggle to maintain the quality of longer videos due to the difficulties in preserving visual context from earlier scenes. We address this by introducing a cached memory mechanism that stores the key frames. Our multi-scene video generation process is explicitly conditioned on the cached memories to avoid forgetting the visual appearance of target subjects. Corgi shows significant improvement in multi-scene video generation compared to the prior art with up to 59.2% in long-term consistency and 7.6% in diversity.
I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting
Authors: Nicola Fanelli; Gennaro Vessio; Giovanna Castellano
Venue: WACV 2025 (main)

Abstract: Inpainting focuses on filling missing or corrupted regions of an image to blend seamlessly with its surrounding content and style. While conditional diffusion models have proven effective for text-guided inpainting we introduce the novel task of multi-mask inpainting where multiple regions are simultaneously inpainted using distinct prompts. Furthermore we design a fine-tuning procedure for multimodal LLMs such as LLaVA to generate multi-mask prompts automatically using corrupted images as inputs. These models can generate helpful and detailed prompt suggestions for filling the masked regions. The generated prompts are then fed to Stable Diffusion which is fine-tuned for the multi-mask inpainting problem using rectified cross-attention enforcing prompts onto their designated regions for filling. Experiments on digitized paintings from WikiArt and the Densely Captioned Images dataset demonstrate that our pipeline delivers creative and accurate inpainting results. Our code data and trained models are available at https://cilabuniba.github.io/i-dream-my-painting.
CATALOG: A Camera Trap Language-Guided Contrastive Learning Model
Authors: Julian D. Santamaria; Claudia Isaza; Jhony H. Giraldo
Venue: WACV 2025 (main)

Abstract: Foundation Models (FMs) have been successful in various computer vision tasks like image classification object detection and image segmentation. However these tasks remain challenging when these models are tested on datasets with different distributions from the training dataset a problem known as domain shift. This is especially problematic for recognizing animal species in camera-trap images where we have variability in factors like lighting camoulage and occlusions. In this paper we propose the Camera Trap Language-guided Contrastive Learning (CATALOG) model to address these issues. Our approach combines multiple FMs to extract visual and textual features from camera-trap data and uses a contrastive loss function to train the model. We evaluate CATALOG on two benchmark datasets and show that it outperforms previous state-of-the-art methods in camera-trap image recognition especially when the training and testing data have different animal species or come from different geographical areas. Our approach demonstrates the potential of using FMs in combination with multi-modal fusion and contrastive learning for addressing domain shifts in camera-trap image recognition. The code of CATALOG is publicly available at https://github.com/Julian075/CATALOG.
Patch Ranking: Token Pruning as Ranking Prediction for Efficient CLIP
Authors: Cheng-En Wu; Jinhong Lin; Yu Hen Hu; Pedro Morgado
Venue: WACV 2025 (main)

Abstract: Contrastive image-text pre-trained models such as CLIP have shown remarkable adaptability to downstream tasks. However they face challenges due to the high computational requirements of the Vision Transformer (ViT) backbone. Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP and identifying the optimal subset of tokens for maximum performance. To address this we propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. To compensate for any performance degradation resulting from token pruning we incorporate learnable visual tokens that aid in restoring and potentially enhancing the model's performance. Our work presents a comprehensive and systematic investigation of pruning tokens within the ViT backbone of CLIP models. Through our framework we successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3% across seven datasets. Our study lays the groundwork for building more computationally efficient multimodal models without sacrificing their performance addressing a key challenge in the application of advanced vision-language models.
Refining Text-to-Image Generation: Towards Accurate Training-Free Glyph-Enhanced Image Generation
Authors: Sanyam Lakhanpal; Shivang Chopra; Vinija Jain; Aman Chadha; Man Luo
Venue: WACV 2025 (main)

Abstract: Over the past few years Text-to-Image (T2I) generation approaches based on diffusion models have gained significant attention. However vanilla diffusion models often suffer from spelling inaccuracies in the text displayed within the generated images. The capability to generate visual text is crucial offering both academic interest and a wide range of practical applications. To produce accurate visual text images state-of-the-art techniques adopt a glyph-controlled image generation approach consisting of a text layout generator followed by an image generator that is conditioned on the generated text layout. Nevertheless our study reveals that these models still face three primary challenges prompting us to develop a testbed to facilitate future research. We introduce a benchmark LenCom-Eval specifically designed for testing models capability in generating images with Lengthy and Complex visual text. Subsequently we introduce a training-free framework to enhance the two-stage generation approaches. We examine the effectiveness of our approach on both LenCom-Eval and MARIO-Eval benchmarks and demonstrate notable improvements across a range of evaluation metrics including CLIPScore OCR precision recall F1 score accuracy and edit distance scores. For instance our proposed framework improves the backbone model TextDiffuser by more than 23% and 13.5% in terms of OCR word F1 on LenCom-Eval and MARIO-Eval respectively. Our work makes a unique contribution to the field by focusing on generating images with long and rare text sequences a niche previously unexplored by existing literature.