GenAI4ELab/papercli-papers huggingface dataset:
Transferable-Guided Attention is All You Need for Video Domain Adaptation
Authors: André Sacilotti; Samuel Felipe dos Santos; Nicu Sebe; Jurandy Almeida
Venue: WACV 2025 (main)

Abstract: Unsupervised domain adaptation (UDA) in videos is a challenging task that remains not well explored compared to image-based UDA techniques. Although vision transformers (ViT) achieve state-of-the-art performance in many computer vision tasks their use in video UDA has been little explored. Our key idea is to use transformer layers as a feature encoder and incorporate spatial and temporal transferability relationships into the attention mechanism. A Transferable-guided Attention (TransferAttn) framework is then developed to exploit the capacity of the transformer to adapt cross-domain knowledge across different backbones. To improve the transferability of ViT we introduce a novel and effective module named Domain Transferable-guided Attention Block (DTAB). DTAB compels ViT to focus on the spatio-temporal transferability relationship among video frames by changing the self-attention mechanism to a transferability attention mechanism. Extensive experiments were conducted on UCF-HMDB Kinetics-Gameplay and Kinetics-NEC Drone datasets with different backbones like ResNet101 I3D and STAM to verify the effectiveness of TransferAttn compared with state-of-the-art approaches. Also we demonstrate that DTAB yields performance gains when applied to other state-of-the-art transformer-based UDA methods from both video and image domains. Our code is available at https://github.com/Andre-Sacilotti/transferattn-project-code.
An Encoder-Agnostic Weakly Supervised Method for Describing Textures
Authors: Shangbo Mao; Deepu Rajan
Venue: WACV 2025 (main)

Abstract: Recent advances in Large Language Models (LLMs) have enabled the semantic description of textures in natural language aiming to capture them in richer detail. However most methods are confined to either depending on supervised training with pairs of images and manually annotated visual attributes that most texture datasets lack or using Vision-Language Models (VLMs) such as CLIP. In this paper we develop an encoder-agnostic Weakly supervised Texture Description Generator (WTDG) that employs a novel Scaled Ranked Kullback-Leibler divergence (SR-KL) loss between image and text modalities. Within the SR-KL loss formulation we leverage category information which is always available as ground-truths for all benchmark texture recognition datasets. We further extend our proposed WTDG to assist in texture recognition by using its generated texture descriptions. Thus we develop a multimodal framework called Tex^2 which is adept at simultaneous generation of texture description and recognition. Our approach exhibits promising performance in describing and recognizing textures on benchmark datasets.
Deciphering the Complaint Aspects: Towards an Aspect-Based Complaint Identification Model with Video Complaint Dataset in Finance
Authors: Sarmistha Das; Basha Mujavarsheik; R E Zera Lyngkhoi; Sriparna Saha; Alka Maurya
Venue: WACV 2025 (main)

Abstract: In today's competitive marketing landscape effective complaint management is crucial for customer service and business success. Video complaints integrating text and image content offer invaluable insights by addressing customer grievances and delineating product benefits and drawbacks. However comprehending nuanced complaint aspects within vast daily multimodal financial data remains a formidable challenge. Addressing this gap we have curated a proprietary multimodal video complaint dataset comprising 433 publicly accessible instances. Each instance is meticulously annotated at the utterance level encompassing five distinct categories of financial aspects and their associated complaint labels. To support this endeavour we introduce Solution 3.0 a model designed for multimodal aspect-based complaint identification task. Solution 3.0 is tailored to perform three key tasks: 1) handling multimodal features (audio and video) 2) facilitating multilabel aspect classification and 3) conducting multitasking for aspect classifications and complaint identification parallelly. Solution 3.0 utilizes a CLIP-based dual frozen encoder with an integrated image segment encoder for global feature fusion enhanced by contextual attention (ISEC) to improve accuracy and efficiency. Our proposed framework surpasses current multimodal baselines exhibiting superior performance across nearly all metrics by opening new ways to strengthen appropriate customer care initiatives and effectively assisting individuals in resolving their problems.
DiffuPT: Class Imbalance Mitigation for Glaucoma Detection via Diffusion Based Generation and Model Pretraining
Authors: Youssof Nawar; Nouran Soliman; Moustafa Wassel; Mohamed ElHabebe; Noha Adly; Marwan Torki; Ahmed Elmassry; Islam Ahmed
Venue: WACV 2025 (main)

Abstract: Glaucoma is a progressive optic neuropathy characterized by structural damage to the optic nerve head and functional changes in the visual field. Detecting glaucoma early is crucial to preventing loss of eyesight. However medical datasets often suffer from class imbalances making detection more difficult for deep-learning algorithms. We use a generative-based framework to enhance glaucoma diagnosis specifically addressing class imbalance through synthetic data generation. In addition we collected the largest national dataset for glaucoma detection to support our study. The imbalance between normal and glaucomatous cases leads to performance degradation of classifier models. By combining our proposed framework leveraging diffusion models with a pretraining approach we created a more robust classifier training process. This training process results in a better-performing classifier. The proposed approach shows promising results in improving the harmonic mean "sensitivity and specificity" and AUC for the roc for the glaucoma classifier. We report an improvement in the harmonic mean metric from 89.09% to 92.59% on the test set of our national dataset. We examine our method against other methods to overcome imbalance through extensive experiments. We report similar improvements on the AIROGS dataset. This study highlights that diffusion-based generation can be of great importance in tackling class imbalances in medical datasets to improve diagnostic performance.
DiHuR: Diffusion-Guided Generalizable Human Reconstruction
Authors: Jinnan Chen; Chen Li; Gim Hee Lee
Venue: WACV 2025 (main)

Abstract: We introduce DiHuR a novel Diffusion-guided model for generalizable Human 3D Reconstruction and view synthesis from sparse minimally overlapping images. While existing generalizable human radiance fields excel at novel view synthesis they often struggle with comprehensive 3D reconstruction. Similarly directly optimizing implicit Signed Distance Function (SDF) fields from sparse-view images typically yields poor results due to limited overlap. To enhance 3D reconstruction quality we propose using learnable tokens associated with SMPL vertices to aggregate sparse view features and then to guide SDF prediction. These tokens learn a generalizable prior across different identities in training datasets leveraging the consistent projection of SMPL vertices onto similar semantic areas across various human identities. This consistency enables effective knowledge transfer to unseen identities during inference. Recognizing SMPL's limitations in capturing clothing details we incorporate a diffusion model as an additional prior to fill in missing information particularly for complex clothing geometries. Our method integrates two key priors in a coherent manner: the prior from generalizable feed-forward models and the 2D diffusion prior and it requires only multi-view image training without 3D supervision. DiHuR demonstrates superior performance in both within-dataset and cross-dataset generalization settings as validated on THuman ZJU-MoCap and HuMMan datasets compared to existing methods.
CUNSB-RFIE: Context-Aware Unpaired Neural Schrodinger Bridge in Retinal Fundus Image Enhancement
Authors: Xuanzhao Dong; Vamsi Krishna Vasa; Wenhui Zhu; Peijie Qiu; Xiwen Chen; Yi Su; Yujian Xiong; Zhangsihao Yang; Yanxi Chen; Yalin Wang
Venue: WACV 2025 (main)

Abstract: Retinal fundus photography is significant in diagnosing and monitoring retinal diseases. However systemic imperfections and operator/patient-related factors can hinder the acquisition of high-quality retinal images. Previous efforts in retinal image enhancement primarily relied on GANs which are limited by the trade-off between training stability and output diversity. In contrast the Schrodinger Bridge (SB) offers a more stable solution by utilizing Optimal Transport (OT) theory to model a stochastic differential equation (SDE) between two arbitrary distributions. This allows SB to effectively transform low-quality retinal images into their high-quality counterparts. In this work we leverage the SB framework to propose an image-to-image translation pipeline for retinal image enhancement. Additionally previous methods often fail to capture fine structural details such as blood vessels. To address this we enhance our pipeline by introducing Dynamic Snake Convolution whose tortuous receptive field can better preserve tubular structures. We name the resulting retinal fundus image enhancement framework the Context-aware Unpaired Neural Schrodinger Bridge (CUNSB-RFIE). To the best of our knowledge this is the first endeavor to use the SB approach for retinal image enhancement. Experimental results on a large-scale dataset demonstrate the advantage of the proposed method compared to several state-of-the-art supervised and unsupervised methods in terms of image quality and performance on downstream tasks. The code is available at https://github.com/Retinal-Research/CUNSB-RFIE.
Self-Supervised Incremental Learning of Object Representations from Arbitrary Image Sets
Authors: George Leotescu; Alin-Ionut Popa; Diana-Nicoleta N Grigore; Daniel Voinea; Pietro Perona
Venue: WACV 2025 (main)

Abstract: Computing a comprehensive and robust visual representation of an arbitrary object or category of objects is a complex problem. The difficulty increases when one starts from a set of uncalibrated images obtained from different sources. We propose a self-supervised approach Multi-Image Latent Embedding (MILE) which computes a single representation from such an image set. MILE operates incrementally considering one image at a time while processing various depictions of the class through a shared gated cross-attention mechanism. The representations are progressively refined as more available images are incorporated without requiring additional training. Our experiments on Amazon Berkeley Objects (ABO) and iNaturalist demonstrate the effectiveness in two tasks: object or category-specific image retrieval and unsupervised context-conditioned object segmentation. Moreover the proposed multi-image input setup opens new frontiers for the task of object retrieval. Our studies indicate that our models can capture descriptive representations that better encapsulate the intrinsic characteristics of the objects. Our code is available at https://github.com/amazon-science/mile.
ECF-YOLOv7-Tiny: Improving Feature Fusion and the Receptive Field for Lightweight Object Detectors
Authors: Dan-Sebastian Bacea; Florin Oniga
Venue: WACV 2025 (main)

Abstract: In this work we aim to increase the efficiency and the detection performance of lightweight object detectors with focus on feature fusion and receptive field of the models. For improved feature fusion we introduce the Convolutional Squeeze-and-Excitation (CSE) module which requires only minimal additional computation. For improving the receptive field and feature extraction capabilities in a resource effective manner we introduce the Cross-Stage Partial Context Augmentation Module (CSP-CAM). Furthermore for improving real-time performance we apply two model scaling techniques with minimal impact on the detection performance. We prove the effectiveness of the proposed modules by inserting them into YOLOv7-tiny and YOLOv9-t. We build a new network architecture ECFYOLOv7-tiny which we train on the MS COCO dataset and evaluate the inference speed on NVIDIA Jetson Nano. ECFYOLOv7-tiny achieves 37.8% mAP@[0.5:0.95] on the test set while reaching 9.3 FPS on Jetson Nano outperforming the other state-of-the-art lightweight object detectors on the 416 input image resolution. Code and models are released at: https://github.com/dbacea/ecf-yolov7-tiny.
Breaking the Frame: Visual Place Recognition by Overlap Prediction
Authors: Tong Wei; Philipp Lindenberger; Jirí Matas; Daniel Barath
Venue: WACV 2025 (main)

Abstract: Visual place recognition methods struggle with occlusion and partial visual overlaps. We propose a novel visual place recognition approach based on overlap prediction called VOP shifting from traditional reliance on global image similarities and local features to image overlap prediction. VOP proceeds co-visible image sections by obtaining patch-level embeddings using a Vision Transformer backbone and establishing patch-to-patch correspondences without requiring expensive feature detection and matching. Our approach uses a voting mechanism to assess overlap scores for potential database images. It provides a nuanced image retrieval metric in challenging scenarios. Experimental results show that VOP leads to more accurate relative pose estimation and localization results on the retrieved image pairs than state-of-the-art baselines on a number of large-scale real-world indoor and outdoor benchmarks. The code is available at https://github.com/weitong8591/vop.git.
Paladin: Understanding Video Intentions in Political Advertisement Videos
Authors: Hong Liu; Yuta Nakashima; Noboru Babaguchi
Venue: WACV 2025 (main)

Abstract: In this paper we introduce a novel task for video understanding that focuses on detecting editing intentions in political advertisement videos. Political advertisement videos are edited with some intentions (e.g. "associating some candidates with negative emotions") of making people unthinkingly believe the messages in the videos potentially ending up with some irrational bias. Detecting such intentions is thus the primary step toward fairer decision-making based on the messages themselves. To this end we classify such editing intentions into 10 categories (referred to as communication techniques) in consultation with a professional editor as well as based on communication techniques presented in the natural language processing community and build a dataset of 12526 political advertisement videos each of which is annotated with several communication technique segments. We also explore the capability of existing video understanding models in detecting editing intentions over the dataset which identifies new dimensions of challenges to be addressed.
DarSwin-Unet: Distortion Aware Architecture
Authors: Akshaya Athwale; Ichrak Shili; Émile Bergeron; Ola Ahmad; Jean-Francois Lalonde
Venue: WACV 2025 (main)

Abstract: Wide angle fisheye images are becoming increasingly common for perception tasks in applications such as robotics security and mobility (e.g. drones avionics). However current models often either ignore the distortions in wide angle images or are not suitable to perform pixel-level tasks. In this paper we present an encoder-decoder model based on a radial transformer architecture that adapts to distortions in wide angle lenses by leveraging the physical characteristics defined by the radial distortion profile. In contrast to the original model which only performs classification tasks we introduce a U-Net architecture DarSwin-Unet designed for pixel level tasks. Furthermore we propose a novel strategy that minimizes sparsity when sampling the image for creating its input tokens. Our approach enhances the model capability to handle pixel-level tasks in wide angle fisheye images making it more effective for real-world applications. Compared to other baselines DarSwin-Unet achieves the best results across different datasets with significant gains when trained on bounded levels of distortions (very low low medium and high) and tested on all including out-of-distribution distortions. We demonstrate its performance on depth estimation and show through extensive experiments that DarSwin-Unet can perform zero-shot adaptation to unseen distortions of different wide angle lenses. The code and models are publicly available at https://lvsn.github.io/darswin-unet/.
Transientangelo: Few-Viewpoint Surface Reconstruction using Single-Photon Lidar
Authors: Weihan Luo; Anagh Malik; David B Lindell
Venue: WACV 2025 (main)

Abstract: We consider the problem of few-viewpoint 3D surface reconstruction using raw measurements from a lidar system. Lidar captures 3D scene geometry by emitting pulses of light to a target and recording the speed-of-light time delay of the reflected light. However conventional lidar systems do not output the raw captured waveforms of backscattered light; instead they preprocess these data into a 3D point cloud. Since this procedure typically does not accurately model the noise statistics of the system exploit spatial priors or incorporate information about downstream tasks it ultimately discards useful information that is encoded in raw measurements of backscattered light. Here we propose to leverage raw measurements captured with a single-photon lidar system from multiple viewpoints to optimize a neural surface representation of a scene. The measurements consist of time-resolved photon count histograms or transients which capture information about backscattered light at picosecond time scales. Additionally we develop new regularization strategies that improve robustness to photon noise enabling accurate surface reconstruction with as few as 10 photons per pixel. Our method outperforms other techniques for few-viewpoint 3D reconstruction based on depth maps point clouds or conventional lidar as demonstrated in simulation and with captured data.
Shape-Biased Texture Agnostic Representations for Improved Textureless and Metallic Object Detection and 6D Pose Estimation
Authors: Peter Hönig; Stefan Thalhammer; Jean-Baptiste Weibel; Matthias Hirschmanner; Markus Vincze
Venue: WACV 2025 (main)

Abstract: Recent advances in machine learning have greatly benefited object detection and 6D pose estimation. However textureless and metallic objects still pose a significant challenge due to few visual cues and the texture bias of CNNs. To address this issue we propose a strategy for inducing a shape bias to CNN training. In particular by randomizing textures applied to object surfaces during data rendering we create training data without consistent textural cues. This methodology allows for seamless integration into existing data rendering engines and results in negligible computational overhead for data rendering and network training. Our findings demonstrate that the shape bias we induce via randomized texturing improves over existing approaches using style transfer. We evaluate with five detectors and two pose estimators. For three object detectors and for pose estimation in general estimation accuracy improves for textureless and metallic objects. Additionally we show that our approach increases the pose estimation accuracy in the presence of image noise and strong illumination changes. Code available at https://github.com/hoenigpeter/randomized_texturing.
Multi-Modal Large Language Models are Effective Vision Learners
Authors: Li Sun; Chaitanya Ahuja; Peng Chen; Matt D'Zmura; Kayhan Batmanghelich; Philip Bontrager
Venue: WACV 2025 (main)

Abstract: Large language models (LLMs) pre-trained on vast amounts of text have shown remarkable abilities in understanding general knowledge and commonsense. Therefore it's desirable to leverage pre-trained LLM to help solve computer vision tasks. Previous works on multi-modal LLM mainly focus on the generation capability. In this work we propose LLM-augmented visual representation learning (LMVR). Our approach involves initially using a vision encoder to extract features which are then projected into the word embedding space of the LLM. The LLM then generates responses based on the visual representation and a text prompt. Finally we aggregate sequence-level features from the hidden layers of the LLM to obtain image-level representations. We conduct extensive experiments on multiple datasets and have the following findings: (a) LMVR outperforms traditional vision encoder on various downstream tasks and effectively learns the correspondence between words and image regions; (b) LMVR improves the generalizability compared to using a vision encoder alone as evidenced by its superior resistance to domain shift; (c) LMVR improves the robustness of models to corrupted and perturbed visual data. Our findings demonstrate LLM-augmented visual representation learning is effective as it learns object-level concepts and commonsense knowledge.
GeoGuide: Geometric Guidance of Diffusion Models
Authors: Mateusz Poleski; Jacek Tabor; Przemyslaw Spurek
Venue: WACV 2025 (main)

Abstract: Diffusion models are currently one of the most effective tools in image generation. This is in particular due to the fact that contrary to GANs during training they can be easily conditioned. However given a pretrained diffusion guiding it to obtain desired result is typically a more delicate task. A typical technique based on the probabilistic derivation lies in adding the rescaled gradient of the classifier during backward propagation. In this paper we switch the perspective from probabilistic to metric. By studying the distance of the trajectory of the diffusion model from the data manifold we introduce a new guideance model GeoGuide. GeoGuide is not only easy to apply as it is based on classifier gradient normalization but it outperforms the probabilistic approach both with respect to FID and the quality of generated images.
ATM: Enhanced Alignment for Text-to-Motion Generation
Authors: Ke Han; Yueming Lyu; Weichen Yu; Nicu Sebe
Venue: WACV 2026 (main)

Abstract: Existing text-to-motion (T2M) generation methods primarily rely on regression-based objectives, such as minimizing positional errors. However, they lack effective semantic supervision and correction mechanisms, often leading to substantial misalignment between text and motion. To address this, we propose Aligned Text-to-Motion (ATM), a semantics-aware generation framework that automatically identifies and corrects text-motion misalignment. ATM incorporates two key components: (1) Inter-motion alignment, which detects semantic contradictions across motions and applies adaptive corrections based on the degree of semantic discrepancy, flexibly handing diverse misalignments and ensuring global text-motion consistency; (2) Intra-motion alignment, which refines locally missing or inaccurate motion semantics in an unsupervised manner by inferring semantic proxies, effectively addressing the absence of localized textual annotations. ATM is model-agnostic and can be seamlessly integrated into various T2M methods as a plug-and-play module. Extensive experiments on HumanML3D and KIT demonstrate that ATM consistently improves both generation quality and text-motion alignment. Code is available at https://github.com/ke-han-aca/ATM.git.