GenAI4ELab/papercli-papers huggingface dataset:
Focusing on What to Decode and What to Train: SOV Decoding with Specific Target Guided DeNoising and Vision Language Advisor
Authors: Junwen Chen; Yingcheng Wang; Keiji Yanai
Venue: WACV 2025 (main)

Abstract: Recent transformer-based methods achieve notable gains in the Human-object Interaction Detection (HOID) task by leveraging the detection of DETR and the prior knowledge of Vision-Language Model (VLM). However these methods suffer from extended training times and complex optimization due to the entanglement of object detection and HOI recognition during the decoding process. Especially the query embeddings used to predict both labels and boxes suffer from ambiguous representations and the gap between the prediction of HOI labels and verb labels is not considered. To address these challenges we introduce SOV-STG-VLA with three key components: Subject-Object-Verb (SOV) decoding Specific Target Guided (STG) denoising and a Vision-Language Advisor (VLA). Our SOV decoders disentangle object detection and verb recognition with a novel interaction region representation. The STG denoising strategy learns label embeddings with ground-truth information to guide the training and inference. Our SOV-STG achieves a fast convergence speed and high accuracy and builds a foundation for the VLA to incorporate the prior knowledge of the VLM. We introduce a vision advisor decoder to fuse both the interaction region information and the VLM's vision knowledge and a Verb-HOI prediction bridge to promote interaction representation learning. Our VLA notably improves our SOV-STG and achieves SOTA performance with one-sixth of training epochs compared to recent SOTA. Code and models are available at https://github.com/cjw2021/SOV-STG-VLA.
Efficient Video Object Segmentation via Modulated Cross-Attention Memory
Authors: Abdelrahman Shaker; Syed Talal Wasim; Martin Danelljan; Salman Khan; Ming-Hsuan Yang; Fahad Shahbaz Khan
Venue: WACV 2025 (main)

Abstract: Recently transformer-based approaches have shown promising results for semi-supervised video object segmentation. However these approaches typically struggle on long videos due to increased GPU memory demands as they frequently expand the memory bank every few frames. We propose a transformer-based approach named MAVOS that introduces an optimized and dynamic long-term modulated cross-attention (MCA) memory to model temporal smoothness without requiring frequent memory expansion. The proposed MCA effectively encodes both local and global features at various levels of granularity while efficiently maintaining consistent speed regardless of the video length. Extensive experiments on multiple benchmarks LVOS Long-Time Video and DAVIS 2017 demonstrate the effectiveness of our proposed contributions leading to real-time inference and markedly reduced memory demands without any degradation in segmentation accuracy on long videos. Compared to the best existing transformer-based approach our MAVOS increases the speed by 7.6x while significantly reducing the GPU memory by 87% with comparable segmentation performance on short and long video datasets. Notably on the LVOS dataset our MAVOS achieves a J&F score of 63.3% while operating at 37 frames per second (FPS) on a single V100 GPU. Our code and models will be publicly released at https://github.com/Amshaker/MAVOS.
OpenCity3D: What do Vision-Language Models Know About Urban Environments?
Authors: Valentin Bieri; Marco Zamboni; Nicolas Samuel Blumer; Qingxuan Chen; Francis Engelmann
Venue: WACV 2025 (main)

Abstract: The rise of 2D vision-language models (VLMs) has enabled new possibilities for language-driven 3D scene understanding tasks. Existing works focus on indoor scenes or autonomous driving scenarios and typically validate against a pre-defined set of semantic object classes. In this work we analyze the capabilities of vision-language models for large-scale urban 3D scene understanding and propose new applications of VLMs that directly operate on aerial 3D reconstructions of cities. In particular we address higher-level 3D scene understanding tasks such as population density building age property prices crime rate and noise pollution. Our analysis reveals surprising zero-shot and few-shot performance of VLMs in urban environments.
NPL-MVPS: Neural Point-Light Multi-View Photometric Stereo
Authors: Fotios Logothetis; Ignas Budvytis; Roberto Cipolla
Venue: WACV 2025 (main)

Abstract: In this work we present a novel multi-view photometric stereo (MVPS) method. Like many works in 3D reconstruction we are leveraging neural shape representations and learnt renderers. However our work differs from the state-of-the-art multi-view PS methods such as PS-NeRF or Supernormal in that we explicitly leverage per-pixel intensity renderings rather than relying mainly on estimated normals. We model point light attenuation and explicitly raytrace cast shadows in order to best approximate the incoming radiance for each point. The estimated incoming radiance is used as input to a fully neural material renderer that uses minimal prior assumptions and it is jointly optimised with the surface. Estimated normals and segmentation maps are also incorporated in order to maximise the surface accuracy. Our method is among the first (along with Supernormal) to outperform the classical MVPS approach proposed by the DiLiGenT-MV benchmark and achieves average 0.2mm Chamfer distance for objects imaged at approx 1.5m distance away with approximate 400x400 resolution. Moreover our method shows high robustness to the sparse MVPS setup (6 views 6 lights) greatly outperforming the SOTA competitor (0.38mm vs 0.61mm) illustrating the importance of neural rendering in multi-view photometric stereo.
RAW-Diffusion: RGB-Guided Diffusion Models for High-Fidelity RAW Image Generation
Authors: Christoph Reinders; Radu Berdan; Beril Besbinar; Junji Otsuka; Daisuke Iso
Venue: WACV 2025 (main)

Abstract: Current deep learning approaches in computer vision primarily focus on RGB data sacrificing information. In contrast RAW images offer richer representation which is crucial for precise recognition particularly in challenging conditions like low-light environments. The resultant demand for comprehensive RAW image datasets contrasts with the labor-intensive process of creating specific datasets for individual sensors. To address this we propose a novel diffusion-based method for generating RAW images guided by RGB images. Our approach integrates an RGB-guidance module for feature extraction from RGB inputs then incorporates these features into the reverse diffusion process with RGB-guided residual blocks across various resolutions. This approach yields high-fidelity RAW images enabling the creation of camera-specific RAW datasets. Our RGB2RAW experiments on four DSLR datasets demonstrate state-of-the-art performance. Moreover RAW-Diffusion demonstrates exceptional data efficiency achieving remarkable performance with as few as 25 training samples or even fewer. We extend our method to create BDD100K-RAW and Cityscapes-RAW datasets revealing its effectiveness for object detection in RAW imagery significantly reducing the amount of required RAW images. The code is available at https:// github.com/SonyResearch/RAW-Diffusion.
Uncertainty-Guided Metric Learning without Labels
Authors: Dhanunjaya Varma Devalraju; C Chandra Sekhar
Venue: WACV 2025 (main)

Abstract: Unsupervised metric learning aims to learn the discriminative representations by grouping similar examples in the absence of labels. Many unsupervised metric learning algorithms combine clustering-based pseudo-label generation with embedding fine-tuning. However pseudo-labels can be unreliable and noisy. This could affect metric learning and degrade the quality of the learned representations. In this work we propose an approach to reduce the negative effect of label noise on learning discriminative embeddings by using context and prediction uncertainty. In particular we refine the pseudo-labels by aggregating information from neighbors. We propose a function to weigh the pairs leveraging their prediction confidence and uncertainty. We modify the metric learning loss function to incorporate this weight. Experimental results demonstrate the effectiveness of our proposed method on standard datasets for metric learning.
TaxaBind: A Unified Embedding Space for Ecological Applications
Authors: Srikumar Sastry; Subash Khanal; Aayush Dhakal; Adeel Ahmad; Nathan Jacobs
Venue: WACV 2025 (main)

Abstract: We present TaxaBind a unified embedding space for characterizing any species of interest. TaxaBind is a multimodal embedding space across six modalities: ground-level images of species geographic location satellite image text audio and environmental features useful for solving ecological problems. To learn this joint embedding space we leverage ground-level images of species as a binding modality. We propose multimodal patching a technique for effectively distilling the knowledge from various modalities into the binding modality. We construct two large datasets for pretraining: iSatNat with species images and satellite images and iSoundNat with species images and audio. Additionally we introduce TaxaBench-8k a diverse multimodal dataset with six paired modalities for evaluating deep learning models on ecological tasks. Experiments with TaxaBind demonstrate its strong zero-shot and emergent capabilities on a range of tasks including species classification cross-model retrieval and audio classification. The datasets and models are made available at https://github.com/mvrl/TaxaBind.
MetaVIn: Meteorological and Visual Integration for Atmospheric Turbulence Strength Estimation
Authors: Ripon Kumar Saha; Scott McCloskey; Suren Jayasuriya
Venue: WACV 2025 (main)

Abstract: Long-range image understanding is a challenging task for computer vision due to the presence of atmospheric turbulence. Turbulence can degrade image quality (blur and geometric distortion) due to the medium's spatio-temporal varying index of refraction bending light rays. The strength of atmospheric turbulence is quantified by the refractive index structure parameter Cn2 and estimating it is important both as an indicator of image degradation and is useful for downstream tasks including video restoration and estimating true shape and range/depth. However traditional methods for estimating Cn2 involve expensive and complex optical equipment limiting their practicality. In this paper we propose MetaVIn: a Meteorological and Visual Integration system to predict atmospheric turbulence strength. Our method leverages image quality metrics to capture sharpness and blur combined with meteorological information within a Kolmogorov Arnold Network (KAN). We demonstrate that this approach provides a more accurate and generalizable estimation of Cn2 outperforming previous state-of-the-art methods in both blind image quality assessment and passive video-based turbulence strength estimation on a large dataset of 35364 image samples with accompanying ground truth scintillometer measurements for Cn2. Our method enables better prediction and mitigation of atmospheric image degradation while being useful in applications such as shape and range estimation enhancing the practical utility of our approach.
A Versatile and Differentiable Hand-Object Interaction Representation
Authors: Théo Morales; Omid Taheri; Gerard Lacey
Venue: WACV 2025 (main)

Abstract: Synthesizing accurate hands-object interactions (HOI) is critical for applications in Computer Vision Augmented Reality (AR) and Mixed Reality (MR). Despite recent advances the accuracy of reconstructed or generated HOI leaves room for refinement. Some techniques have improved the accuracy of dense correspondences by shifting focus from generating explicit contacts to using rich HOI fields. Still they lack full differentiability or continuity and are tailored to specific tasks. In contrast we present a Coarse Hand-Object Interaction Representation (CHOIR) a novel versatile and fully differentiable field for HOI modelling. CHOIR leverages discrete unsigned distances for continuous shape and pose encoding alongside multivariate Gaussian distributions to represent dense contact maps with few parameters. To demonstrate the versatility of CHOIR we design JointDiffusion a diffusion model to learn a grasp distribution conditioned on noisy hand-object interactions or only object geometries for both refinement and synthesis applications. We demonstrate JointDiffusion's improvements over the SOTA in both applications: it increases the contact F1 score by 5% for refinement and decreases the sim. displacement by 46% for synthesis. Our experiments show that JointDiffusion with CHOIR yields superior contact accuracy and physical realism compared to SOTA methods designed for specific tasks. Project page: https://theomorales.com/CHOIR
Data Augmentation for Image Classification using Generative AI
Authors: Fazle Rahat; M Shifat Hossain; Md Rubel Ahmed; Sumit Kumar Jha; Rickard Ewetz
Venue: WACV 2025 (main)

Abstract: Scaling laws dictate that the performance of AI models is proportional to the amount of available data. Data augmentation is a promising solution to expanding the dataset size. Traditional approaches focused on augmentation using rotation translation and resizing. Recent approaches use generative AI models to improve dataset diversity. However the generative methods struggle with issues such as subject corruption and the introduction of irrelevant artifacts. In this paper we propose the Automated Generative Data Augmentation (AGA). The framework combines the utility of large language models (LLMs) diffusion models and segmentation models to augment data. AGA preserves foreground authenticity while ensuring background diversity. Specific contributions include: i) segment and superclass based object extraction ii) prompt diversity with combinatorial complexity using prompt decomposition and iii) affine subject manipulation. We evaluate AGA against state-of-the-art (SOTA) techniques on three representative datasets ImageNet CUB and iWildCam. The experimental evaluation demonstrates an accuracy improvement of 15.6% and 23.5% for in and out-of-distribution data compared to baseline models respectively. There is also 64.3% improvement in SIC score compared to the baselines.
EchoDFKD: Data-Free Knowledge Distillation for Cardiac Ultrasound Segmentation using Synthetic Data
Authors: Grégoire Petit; Nathan Palluau; Axel Bauer; Clemens Dlaska
Venue: WACV 2025 (main)

Abstract: The application of machine learning to medical ultrasound videos of the heart i.e. echocardiography has recently gained traction with the availability of large public datasets. Traditional supervised tasks such as ejection fraction regression are now making way for approaches focusing more on the latent structure of data distributions as well as generative methods. We propose a model trained exclusively by knowledge distillation either on real or synthetical data involving retrieving masks suggested by a teacher model. We achieve state-of-the-art (SOTA) values on the task of identifying end-diastolic and end-systolic frames. By training the model only on synthetic data it reaches segmentation capabilities close to the performance when trained on real data with a significantly reduced number of weights. A comparison with the 5 main existing methods shows that our method outperforms the others in most cases. We also present a new evaluation method that does not require human annotation and instead relies on a large auxiliary model. We show that this method produces scores consistent with those obtained from human annotations. Relying on the integrated knowledge from a vast amount of records this method overcomes certain inherent limitations of human annotator labeling. Code: https://github.com/GregoirePetit/EchoDFKD
Latency Robust Cooperative Perception using Asynchronous Feature Fusion
Authors: Junjie Wang; Tomas Nordström
Venue: WACV 2025 (main)

Abstract: Recent advancements in cooperative perception have showcased substantial improvements compared to single-agent perception. Nonetheless the inherent latency present in such systems can dramatically impair their effectiveness. In this paper we propose a Latency Robust Cooperative Perception framework named LRCP to compensate for the effect of temporal asynchrony. The intuition of LRCP is to directly fuse asynchronous bird's-eye view (BEV) features instead of estimating aligned features. To achieve this we first propose a novel flow prediction module that uses cached past BEV features to predict the flow with a non-discrete time delay at the BEV feature level. Then the predicted flow is employed to guide the spatial sampling location of interests. Our approach substantially enhances the robustness of temporal asynchronous cooperative perception. Specifically we achieved robust performance across a range of latencies up to 500 ms with a performance degradation of only 1 percent point for AP@0.5 metric and 4 percent points for AP@0.7 metric at 500 ms on two public datasets (V2X-Sim and Dair-V2X). Code to reproduce our results is available at https://github. com/JesseWong333/LRCP.
DSTR: Dual Scenes Transformer for Cross-Modal Fusion in 3D Object Detection
Authors: Haojie Cai; Dongfu Yin; Fei Richard Yu; Siting Xiong
Venue: WACV 2025 (main)

Abstract: Increasing attention has been garnered by LiDAR points and multi-view images fusion based on Transformer to supplement another modality in 3D object detection. However most current methods perform data fusion based on the entire scene which entails substantial redundant background information and lacks fine-grained local details of the foreground objects to be detected. Furthermore global scene fusion results in coarse fusion granularity and the excessive redundancy leads to slow convergence and reduced accuracy. In this work a novel Dual Scenes Transformer pipeline (DSTR) which comprises a Global-Scene Integration (GSI) module Local-Scene Integration (LSI) module and Dual Scenes Fusion (DSF) module is presented to tackle the above challenge. Concretely features from point clouds and images are utilized for gathering the global scene information in GSI. The insufficiency issues of global scene fusion are addressed by extracting local instance features for both modalities in LSI supplementing GSI in a more fine-grained way. Furthermore DSF is proposed to aggregate the local scene to the global scene which fully explores dual-modal information. Experiments on the nuScenes dataset show that our DSTR has state-of-the-art (SOTA) performance in certain 3D object detection benchmark categories on validation and test sets.
SAM-DA: Decoder Adapter for Efficient Medical Domain Adaptation
Authors: Javier Gamazo Tejero; Moritz J Schmid; Pablo Márquez Neila; Martin Zinkernagel; Sebastian Wolf; Raphael Sznitman
Venue: WACV 2025 (main)

Abstract: This paper addresses the domain adaptation challenge for semantic segmentation in medical imaging. Despite the impressive performance of recent foundational segmentation models like SAM on natural images they struggle with medical domain images. Beyond this recent approaches that perform end-to-end fine-tuning of models are simply not computationally tractable. To address this we propose a novel SAM adapter approach that minimizes the number of trainable parameters while achieving comparable performances to full fine-tuning. The proposed SAM adapter is strategically placed in the mask decoder offering excellent and broad generalization capabilities and improved segmentation across both fully supervised and test-time domain adaptation tasks. Extensive validation on four datasets showcases the adapter's efficacy outperforming existing methods while training less than 1% of SAM's total parameters.
Social EgoMesh Estimation
Authors: Luca Scofano; Alessio Sampieri; Edoardo De Matteis; Indro Spinelli; Fabio Galasso
Venue: WACV 2025 (main)

Abstract: Accurately estimating the 3D pose of the camera wearer in egocentric video sequences is crucial to modeling human behavior in virtual and augmented reality applications. The task presents unique challenges due to the limited visibility of the user's body caused by the front-facing camera mounted on their head. Recent research has explored the utilization of the scene and ego-motion but it has overlooked humans' interactive nature. We propose a novel framework for Social Egocentric Estimation of body MEshes (SEE-ME). Our approach is the first to estimate the wearer's mesh using only a latent probabilistic diffusion model which we condition on the scene and for the first time on the social wearer-interactee interactions. Our in depth study sheds light on when social interaction matters most for ego-mesh estimation: it quantifies the impact of interpersonal distance and gaze direction. Overall SEE-ME surpasses the current best technique reducing the pose estimation error (MPJPE) by 53%. The code is available at https://github.com/L-Scofano/SEEME/tree/main.
Localized Gaussian Splatting Editing with Contextual Awareness
Authors: Hanyuan Xiao; Yingshu Chen; Huajian Huang; Haolin Xiong; Jing Yang; Pratusha Prasad; Yajie Zhao
Venue: WACV 2025 (main)

Abstract: Recent advancements in text-guided 3D object generation using diffusion priors struggle with illumination inconsistencies when applied to scene editing tasks like object replacement or insertion. To address this we propose an illumination-aware 3D scene editing pipeline for 3D Gaussian Splatting (3DGS). Our method leverages state-of-the-art 2D diffusion inpainting to handle global illumination context effectively. Specifically we identify representative anchor views that capture scene-wide illumination inpaint them using 2D diffusion models and integrate the results into a coarse-to-fine 3DGS optimization process. In the fine step we introduce Depth-guided Inpainting Score Distillation Sampling (DI-SDS) to refine geometry and texture details capitalizing on the diversity of 2D priors. Our approach achieves locally precise edits with globally consistent illumination demonstrating robustness in real scenes with highlights and shadows. Comparisons show superior results over state-of-the-art text-to-3D editing methods. Project page: https://corneliushsiao.github.io/GSLE.html.
S3PT: Scene Semantics and Structure Guided Clustering to Boost Self-Supervised Pre-Training for Autonomous Driving
Authors: Maciej K. Wozniak; Hariprasath Govindarajan; Marvin Klingner; Camille Maurice; B Ravi Kiran; Senthil Yogamani
Venue: WACV 2025 (main)

Abstract: Recent self-supervised clustering-based pre-training techniques like DINO and CriBo have shown impressive results for downstream detection and segmentation tasks. However real-world applications such as autonomous driving face challenges with imbalanced object class and size distributions and complex scene geometries. In this paper we propose S3PT a novel scene semantics and structure guided clustering to provide more scene-consistent objectives for self-supervised training. Specifically our contributions are threefold: First we incorporate semantic distribution consistent clustering to encourage better representation of rare classes such as motorcycles or animals. Second we introduce object diversity consistent spatial clustering to handle imbalanced and diverse object sizes ranging from large background areas to small objects such as pedestrians and traffic signs. Third we propose a depth-guided spatial clustering to regularize learning based on geometric information of the scene thus further refining region separation on the feature level. Our learned representations significantly improve performance in downstream semantic segmentation and 3D object detection tasks on the nuScenes nuImages and Cityscapes datasets and show promising domain translation properties.