GenAI4ELab/papercli-papers huggingface dataset:
ARD-VAE: A Statistical Formulation to Find the Relevant Latent Dimensions of Variational Autoencoders
Authors: Surojit Saha; Sarang Joshi; Ross Whitaker
Venue: WACV 2025 (main)

Abstract: The variational autoencoder (VAE) is a popular deep latent-variable model (DLVM) due to its simple yet effective formulation for modeling the data distribution. Moreover optimizing the VAE objective function is more manageable than other DLVMs. The bottleneck dimension of the VAE is a crucial design choice and it has strong ramifications for the model's performance such as finding the hidden explanatory factors of a dataset using the representations learned by the VAE. However the size of the latent dimension of the VAE is often treated as a hyperparameter estimated empirically through trial and error. To this end we propose a statistical formulation to discover the relevant latent factors required for modeling a dataset. In this work we use a hierarchical prior in the latent space that estimates the variance of the latent axes using the encoded data which identifies the relevant latent dimensions. For this we replace the fixed prior in the VAE objective function with a hierarchical prior keeping the remainder of the formulation unchanged. We call the proposed method the automatic relevancy detection in the variational autoencoder (ARD-VAE). We demonstrate the efficacy of the ARD-VAE on multiple benchmark datasets in finding the relevant latent dimensions and their effect on different evaluation metrics such as FID score and disentanglement analysis.
MatSpectNet: Material Segmentation Network with Domain-Aware and Physically-Constrained Hyperspectral Reconstruction
Authors: Yuwen Heng; Yihong Wu; Srinandan Dasmahapatra; Hansung Kim
Venue: WACV 2025 (main)

Abstract: Achieving accurate material segmentation for 3-channel RGB images is challenging due to the considerable variation in material appearance. Hyperspectral images which are sets of spectral measurements sampled at multiple wavelengths theoretically offer distinct information for material identification as variations in the intensity of electromagnetic radiation reflected by a surface depend on the material composition of a scene. However existing hyperspectral datasets are impoverished in terms of the number of images and material categories for the dense material segmentation task and collecting and annotating hyperspectral images with a spectral camera is prohibitively expensive. To address this we propose a novel model the MatSpectNet to segment materials with recovered hyperspectral images from RGB images. The network leverages the principles of colour perception in modern cameras to constrain the reconstructed hyperspectral images and employs the domain adaptation method to generalise the hyperspectral reconstruction capability from a spectral recovery dataset to material segmentation datasets. The reconstructed hyperspectral images are further filtered using learnt response curves and enhanced with human perception. The performance of MatSpectNet is evaluated on the LMD dataset as well as the OpenSurfaces dataset. Our experiments demonstrate that MatSpectNet attains a 1.60% increase in average pixel accuracy and a 3.42% improvement in mean class accuracy compared with the most recent publication. Additional experiments and the project code are published on https://github.com/heng-yuwen/MatSpectNet.
ALSTER: A Local Spatio-Temporal Expert for Online 3D Semantic Reconstruction
Authors: Silvan Weder; Francis Engelmann; Johannes L. Schönberger; Akihito Seki; Marc Pollefeys; Martin R. Oswald
Venue: WACV 2025 (main)

Abstract: We propose an online 3D semantic segmentation method that incrementally reconstructs a 3D semantic map from a stream of RGB-D frames. Unlike offline methods ours is directly applicable to scenarios with real-time constraints such as robotics or mixed reality. To overcome the inherent challenges of online methods we make two main contributions. First to effectively extract information from the input RGB-D video stream we jointly estimate geometry and semantic labels per frame in 3D. A key focus of our approach is to reason about semantic entities both in the 2D input and the local 3D domain to leverage differences in spatial context and network architectures. Our method predicts 2D features using an off-the-shelf segmentation network. The extracted 2D features are refined by a lightweight 3D network to enable reasoning about the local 3D structure. Second to efficiently deal with an infinite stream of input RGBD frames a subsequent network serves as a temporal expert predicting the incremental scene updates by leveraging 2D 3D and past information in a learned manner. These updates are then integrated into a global scene representation. Using these main contributions our method can enable scenarios with real-time constraints and can scale to arbitrary scene sizes by processing and updating the scene only in a local region defined by the new measurement. Our experiments demonstrate improved results compared to existing online methods that purely operate in local regions and show that complementary sources of information can boost the performance. We provide a thorough ablation study on the benefits of different architectural as well as algorithmic design decisions. Our method yields competitive results on the popular ScanNet benchmark and SceneNN dataset.
Joint Co-Speech Gesture and Expressive Talking Face Generation using Diffusion with Adapters
Authors: Steven Hogue; Chenxu Zhang; Yapeng Tian; Xiaohu Guo
Venue: WACV 2025 (main)

Abstract: Recent advances in co-speech gesture and talking head generation have been impressive yet most methods focus on only one of the two tasks. Those that attempt to generate both often rely on separate models or network modules increasing training complexity and ignoring the inherent relationship between face and body movements. To address the challenges in this paper we propose a novel model architecture that jointly generates face and body motions within a single network. This approach leverages shared weights between modalities facilitated by adapters that enable adaptation to a common latent space. Our experiments demonstrate that the proposed framework not only maintains state-of-the-art co-speech gesture and talking head generation performance but also significantly reduces the number of parameters required.
UnDIVE: Generalized Underwater Video Enhancement using Generative Priors
Authors: Suhas Srinath; Aditya Chandrasekar; Hemang Jamadagni; Rajiv Soundararajan; Prathosh A P
Venue: WACV 2025 (main)

Abstract: With the rise of marine exploration underwater imaging has gained significant attention as a research topic. Underwater video enhancement has become crucial for real-time computer vision tasks in marine exploration. However most existing methods focus on enhancing individual frames and neglect video temporal dynamics leading to visually poor enhancements. Furthermore the lack of ground-truth references limits the use of abundant available underwater video data in many applications. To address these issues we propose a two-stage framework for enhancing underwater videos. The first stage uses a denoising diffusion probabilistic model to learn a generative prior from unlabeled data capturing robust and descriptive feature representations. In the second stage this prior is incorporated into a physics-based image formulation for spatial enhancement while also enforcing temporal consistency between video frames. Our method enables real-time and computationally-efficient processing of high-resolution underwater videos at lower resolutions and offers efficient enhancement in the presence of diverse water-types. Extensive experiments on four datasets show that our approach generalizes well and outperforms existing enhancement methods. Our code is available at github.com/suhas-srinath/undive.
Automated Patient Positioning with Learned 3D Hand Gestures
Authors: Zhongpai Gao; Abhishek Sharma; Meng Zheng; Benjamin Planche; Terrence Chen; Ziyan Wu
Venue: WACV 2025 (main)

Abstract: Positioning patients for scanning and interventional procedures is a critical task that requires high precision and accuracy. The conventional workflow involves manually adjusting the patient support to align the center of the target body part with the laser projector or other guiding devices. This process is not only time-consuming but also prone to inaccuracies. In this work we propose an automated patient positioning system that utilizes a camera to detect specific hand gestures from technicians allowing users to indicate the target patient region to the system and initiate automated positioning. Our approach relies on a novel multi-stage pipeline to recognize and interpret the technicians' gestures translating them into precise motions of medical devices. We evaluate our proposed pipeline during actual MRI scanning procedures using RGB-Depth cameras to capture the process. Results show that our system achieves accurate and precise patient positioning with minimal technician intervention. Furthermore we validate our method on HaGRID a large-scale hand gesture dataset demonstrating its effectiveness in hand detection and gesture recognition.
Class-Conditioned Transformation for Enhanced Robust Image Classification
Authors: Tsachi Blau; Roy Ganz; Chaim Baskin; Michael Elad; Alex Bronstein
Venue: WACV 2025 (main)

Abstract: Robust classification methods predominantly concentrate on algorithms that address a specific threat model resulting in ineffective defenses against other threat models. Real-world applications are exposed to this vulnerability as malicious attackers might exploit alternative threat models. In this work we propose a novel test-time threat model agnostic algorithm that enhances Adversarial-Trained (AT) models. Our method operates through COnditional image transformation and DIstance-based Prediction (CODIP) and includes two main steps: First we transform the input image into each dataset class where the input image might be either clean or attacked. Next we make a prediction based on the shortest transformed distance. The conditional transformation utilizes the perceptually aligned gradients property possessed by AT models and as a result eliminates the need for additional models or additional training. Moreover it allows users to choose the desired balance between clean and robust accuracy without training. The proposed method achieves state-of-the-art results demonstrated through extensive experiments on various models AT methods datasets and attack types. Notably applying CODIP leads to substantial robust accuracy improvement of up to +23% +20% +26% and +22% on CIFAR10 CIFAR100 ImageNet and Flowers datasets respectively.
ColFigPhotoAttnNet: Reliable Finger Photo Presentation Attack Detection Leveraging Window-Attention on Color Spaces
Authors: Anudeep Vurity; Emanuela Marasco; Raghavendra Ramachandra; Jongwoo Park
Venue: WACV 2025 (main)

Abstract: Finger photo Presentation Attack Detection (PAD) can significantly strengthen smartphone device security. Still these algorithms are trained to detect certain attack types. Furthermore they are designed to operate on images acquired by specific capture devices which leads to poor generalization and a lack of robustness in handling mobile hardware's evolving nature. The proposed investigation is the first to systematically analyze the performance degradation of existing deep learning PAD systems convolutional and transformers in cross-capture device settings. In this paper we introduce the ColFigPhotoAttnNet architecture designed based on window attention on color channels followed by the nested residual network as the predictor to achieve a reliable PAD. Extensive experiments using various capture devices including iPhone13 Pro GooglePixel 3 Nokia C5 and OnePlusOne were carried out to evaluate the performance of proposed and existing methods on three publicly available databases. The findings underscore the effectiveness of our approach.
Cap2Aug: Caption Guided Image Data Augmentation
Authors: Aniket Roy; Anshul Shah; Ketul Shah; Anirban Roy; Rama Chellappa
Venue: WACV 2025 (main)

Abstract: Visual recognition in a low-data regime is challenging and often prone to overfitting. To mitigate this issue several data augmentation strategies have been proposed. However standard transformations e.g. rotation cropping and flipping provide limited semantic variations. To this end we propose Cap2Aug an image-to-image diffusion model-based data augmentation strategy using image captions to condition the image synthesis step. We generate a caption for an image and use this caption as an additional input for an image-to-image diffusion model. This increases the semantic diversity of the augmented images due to caption conditioning compared to the usual data augmentation techniques. We show that Cap2Aug is particularly effective where only a few samples are available for an object class. However naively generating the synthetic images is not adequate due to the domain gap between real and synthetic images. Thus we employ a maximum mean discrepancy loss to align the synthetic images to the real images to minimize the domain gap. We evaluate our method on few-shot classification and image classification with long-tail class distribution tasks. Cap2Aug achieves state-of-the-art performance on both tasks while evaluated on eleven benchmarks. Code: https://github.com/aniket004/Cap_2_Aug.git
Meta-Learning for Color-to-Infrared Cross-Modal Style Transfer
Authors: Evelyn A. Stump; Francesco Luzi; Leslie M. Collins; Jordan M. Malof
Venue: WACV 2025 (main)

Abstract: Recent object detection models for infrared (IR) imagery are based upon deep neural networks (DNNs) and require large amounts of labeled training imagery. However publicly available datasets that can be used for such training are limited in their size and diversity. To address this problem we explore cross-modal style transfer (CMST) to leverage large and diverse color imagery datasets so that they can be used to train DNN-based IR image-based object detectors. We evaluate six contemporary stylization methods on four publicly-available IR datasets - the first comparison of its kind - and find that CMST is highly effective for DNN-based detectors. Surprisingly we find that existing data-driven methods are outperformed by a simple grayscale stylization (an average of the color channels). Our analysis reveals that existing data-driven methods are either too simplistic or introduce significant artifacts into the imagery. To overcome these limitations we propose meta-learning style transfer (MLST) which learns a stylization by composing and tuning well-behaved analytic functions. We find that MLST leads to more complex stylizations without introducing significant image artifacts and achieves the best overall detector performance on our benchmark datasets.
Improving Pelvic MR-CT Image Alignment with Self-Supervised Reference-Augmented Pseudo-CT Generation Framework
Authors: Daniel Kim; Mohammed A. Al-masni; Jaehun Lee; Dong-Hyun Kim; Kanghyun Ryu
Venue: WACV 2025 (main)

Abstract: RegistFormer our novel reference-augmented image synthesis framework generates aligned pseudo-CT images (with respect to MR) from misaligned MR and CT pairs. RegistFormer addresses the limitations of intensity-based registration methods which often fail due to dissimilar image features and complex deformation fields. Unlike conventional image-to-image (I2I) translation methods our method uses a misaligned CT scan as an auxiliary input to guide the synthesis task through the Deformation-Aware Cross-Attention (DACA) mechanism. DACA integrates the deformation field from a registration method to aggregate spatially matched features from the misaligned CT into MR spatial coordinates. Additionally we propose a novel combination of loss functions for training with datasets of misaligned MR-CT pairs in a self-supervised manner eliminating the need for pre-aligned training data. Experiments were conducted with the synthRAD2023 MR-CT pelvis pair dataset. RegistFormer outperforms past state-of-the-art methods including I2I registration and hybrid (registration + I2I) across metrics evaluating both structure alignment and distribution similarity. Moreover RegistFormer demonstrates superior performance in zero-shot segmentation downstream tasks highlighting its clinical value. Source code: https://github.com/danny4159/RegistFormer
ActionDiffusion: An Action-Aware Diffusion Model for Procedure Planning in Instructional Videos
Authors: Lei Shi; Paul-Christian Bürkner; Andreas Bulling
Venue: WACV 2025 (main)

Abstract: We present ActionDiffusion - a novel diffusion model for procedure planning in instructional videos that is the first to take temporal inter-dependencies between actions into account. Our approach is in stark contrast to existing methods that fail to exploit the rich information content available in the particular order in which actions are performed. Our method unifies the learning of temporal dependencies between actions and denoising of the action plan in the diffusion process by projecting the action information into the noise space. This is achieved 1) by adding action embeddings in the noise masks in the noise-adding phase and 2) by introducing an attention mechanism in the noise prediction network to learn the correlations between different action steps. We report extensive experiments on three instructional video benchmark datasets (CrossTask Coin and NIV) and show that our method outperforms previous state-of-the-art methods on all metrics on CrossTask and NIV and all metrics except accuracy on Coin dataset. We show that by adding action embeddings into the noise mask the diffusion model can better learn action temporal dependencies and increase the performances on procedure planning. Codes are available at https://www.collaborative-ai.org/publications/shi25_wacv/
SGD: Street View Synthesis with Gaussian Splatting and Diffusion Prior
Authors: Zhongrui Yu; Haoran Wang; Jinze Yang; Hanzhang Wang; Jiale Cao; Zhong Ji; Mingming Sun
Venue: WACV 2025 (main)

Abstract: Novel View Synthesis (NVS) for street scenes plays a critical role in the autonomous driving simulation. Current mainstream methods such as Neural Radiance Fields (NeRF) and 3D Gaussian Splatting (3DGS) struggle to maintain rendering quality at the viewpoint that deviates significantly from the training viewpoints. This issue stems from the sparse training views captured by a fixed camera on a moving vehicle. To tackle this problem we propose a novel approach that enhances the capacity of 3DGS by leveraging prior from a Diffusion Model along with complementary multi-modal data. Specifically we first fine-tune a Diffusion Model by adding images from adjacent frames as condition meanwhile exploiting depth data from LiDAR point clouds to supply additional spatial information. Then we apply the fine-tuned Diffusion Model to regularize the 3DGS at unseen views during training. Experimental results validate the effectiveness of our method compared with current state-of-the-art models and demonstrate its advances in rendering images from broader views.
MSI-NeRF: Linking Omni-Depth with View Synthesis through Multi-Sphere Image Aided Generalizable Neural Radiance Field
Authors: Dongyu Yan; Guanyu Huang; Fengyu Quan; Haoyao Chen
Venue: WACV 2025 (main)

Abstract: Panoramic observation using fisheye cameras is significant in virtual reality (VR) and robot perception. However panoramic images synthesized by traditional methods lack depth information and can only provide three degrees-of-freedom (3DoF) rotation rendering in VR applications. To fully preserve and exploit the parallax information within the original fisheye cameras we introduce MSI-NeRF which combines deep learning omnidirectional depth estimation and novel view synthesis. We construct a multi-sphere image as a cost volume through feature extraction and warping of the input images. We further build an implicit radiance field using spatial points and interpolated 3D feature vectors as input which can simultaneously realize omnidirectional depth estimation and 6DoF view synthesis. Leveraging the knowledge from depth estimation task our method can learn scene appearance by source view supervision only. It does not require novel target views and can be trained conveniently on existing panorama depth estimation datasets. Our network has the generalization ability to reconstruct unknown scenes efficiently using only four images. Experimental results show that our method outperforms existing methods in both depth estimation and novel view synthesis tasks.
ARTeFACT: Benchmarking Segmentation Models on Diverse Analogue Media Damage
Authors: Daniela Ivanova; Marco Aversa; Paul Henderson; John Williamson
Venue: WACV 2025 (main)

Abstract: Accurately detecting and classifying damage in analogue media such as paintings photographs textiles mosaics and frescoes is essential for cultural heritage preservation. While machine learning models excel in correcting degradation if the damage operator is known a priori we show that they fail to robustly predict where the damage is even after supervised training; thus reliable damage detection remains a challenge. Motivated by this we introduce ARTeFACT a dataset for damage detection in diverse types analogue media with over 11000 annotations covering 15 kinds of damage across various subjects media and historical provenance. Furthermore we contribute human-verified text prompts describing the semantic contents of the images and derive additional textual descriptions of the annotated damage. We evaluate CNN Transformer diffusion-based segmentation models and foundation vision models in zero-shot supervised unsupervised and text-guided settings revealing their limitations in generalising across media types. Our dataset is available at https://daniela997.github.io/ARTeFACT/ as the first-of-its kind benchmark for analogue media damage detection and restoration.
Analyzing and Improving the Skin Tone Consistency and Bias in Implicit 3D Relightable Face Generators
Authors: Libing Zeng; Nima Khademi Kalantari
Venue: WACV 2025 (main)

Abstract: With the advances in generative adversarial networks (GANs) and neural rendering 3D relightable face generation has received significant attention. Among the existing methods a particularly successful technique uses an implicit lighting representation and generates relit images through the product of synthesized albedo and light-dependent shading images. While this approach produces high-quality results with intricate shading details it often has difficulty producing relit images with consistent skin tones particularly when the lighting condition is extracted from images of individuals with dark skin. Additionally this technique is biased towards producing albedo images with lighter skin tones. Our main observation is that this problem is rooted in the biased spherical harmonics (SH) coefficients used during training. Following this observation we conduct an analysis and demonstrate that the bias appears not only in band 0 (DC term) but also in the other bands of the estimated SH coefficients. We then propose a simple but effective strategy to mitigate the problem. Specifically we normalize the SH coefficients by their DC term to eliminate the inherent magnitude bias while statistically align the coefficients in the other bands to alleviate the directional bias. We also propose a scaling strategy to match the distribution of illumination magnitude in the generated images with the training data. Through extensive experiments we demonstrate the effectiveness of our solution in increasing the skin tone consistency and mitigating bias.