GenAI4ELab/papercli-papers huggingface dataset:
Generation of Complex 3D Human Motion by Temporal and Spatial Composition of Diffusion Models
Authors: Lorenzo Mandelli; Stefano Berretti
Venue: WACV 2025 (main)

Abstract: In this paper we address the challenge of generating realistic 3D human motions for action classes that were never seen during the training phase. Our approach involves decomposing complex actions into simpler movements specifically those observed during training by leveraging the knowledge of human motion contained in GPTs models. These simpler movements are then combined into a single realistic animation using the properties of diffusion models. Our claim is that this decomposition and subsequent recombination of simple movements can synthesize an animation that accurately represents the complex input action. This method operates during the inference phase and can be integrated with any pre-trained diffusion model enabling the synthesis of motion classes not present in the training data. We evaluate our method by dividing two benchmark human motion datasets into basic and complex actions and then compare its performance against the state-of-the-art. Our code and models are publicly available at our https://github.com/divanoLetto/MotionCompositionDiffusion.
Inverting the Generation Process of Denoising Diffusion Implicit Models: Empirical Evaluation and a Novel Method
Authors: Yan Zeng; Masanori Suganuma; Takayuki Okatani
Venue: WACV 2025 (main)

Abstract: This paper studies the problem of inverting the DDIM image generation process to recover latent variables particularly the initial noise map from a generated image. Existing methods often struggle with accuracy in this task. We propose a novel hybrid approach that combines direct inversion via gradient descent for the first step followed by a fixed-point method for subsequent steps. Empirical evaluations across three datasets demonstrate that our method significantly improves the prediction of initial latent variables while achieving superior reconstruction accuracy. Additionally we introduce a new evaluation called the self-interpolation test which assesses the quality of images generated from interpolated points between the true and predicted latent maps offering deeper insights into performance. Our results reveal that while existing methods perform reasonably well in reconstruction they consistently fail to accurately predict the initial latent variables resulting in poor performance on the self-interpolation test. In contrast our method outperforms all others across all metrics providing valuable insights into diffusion models and enhancing their applications in image generation and editing.
HeightMapNet: Explicit Height Modeling for End-to-End HD Map Learning
Authors: Wenzhao Qiu; Shanmin Pang; Hao Zhang; Jianwu Fang; Jianru Xue
Venue: WACV 2025 (main)

Abstract: Recent advances in high-definition (HD) map construction from surround-view images have highlighted their cost-effectiveness in deployment. However prevailing techniques often fall short in accurately extracting and utilizing road features as well as in the implementation of view transformation. In response we introduce HeightMapNet a novel framework that establishes a dynamic relationship between image features and road surface height distributions. By integrating height priors our approach refines the accuracy of Bird's-Eye-View (BEV) features beyond conventional methods. HeightMapNet also introduces a foreground-background separation network that sharply distinguishes between critical road elements and extraneous background components enabling precise focus on detailed road micro-features. Additionally our method leverages multi-scale features within the BEV space optimally utilizing spatial geometric information to boost model performance. HeightMapNet has shown exceptional results on the challenging nuScenes and Argoverse 2 datasets outperforming several widely recognized approaches. The code will be available at https://github.com/adasfag/HeightMapNet/.
Invariant Shape Representation Learning for Image Classification
Authors: Tonmoy Hossain; Jing Ma; Jundong Li; Miaomiao Zhang
Venue: WACV 2025 (main)

Abstract: Geometric shape features have been widely used as strong predictors for image classification. Nevertheless most existing classifiers such as deep neural networks (DNNs) directly leverage the statistical correlations between these shape features and target variables. However these correlations can often be spurious and unstable across different environments (e.g. in different age groups certain types of brain changes have unstable relations with neurodegenerative disease); hence leading to biased or inaccurate predictions. In this paper we introduce a novel framework that for the first time develops invariant shape representation learning (ISRL) to further strengthen the robustness of image classifiers. In contrast to existing approaches that mainly derive features in the image space our model ISRL is designed to jointly capture invariant features in latent shape spaces parameterized by deformable transformations. To achieve this goal we develop a new learning paradigm based on invariant risk minimization (IRM) to learn invariant representations of image and shape features across multiple training distributions/environments. By embedding the features that are invariant with regard to target variables in different environments our model consistently offers more accurate predictions. We validate our method by performing classification tasks on both simulated 2D images real 3D brain and cine cardiovascular magnetic resonance images (MRIs). Our code is publicly available at https://github.com/tonmoy-hossain/ISRL.
Multi-Resolution Guided 3D GANs for Medical Image Translation
Authors: Juhyung Ha; Jong Sung Park; David Crandall; Eleftherios Garyfallidis; Xuhong Zhang
Venue: WACV 2025 (main)

Abstract: Medical image translation is the process of converting from one imaging modality to another in order to reduce the need for multiple image acquisitions from the same patient. This can enhance the efficiency of treatment by reducing the time equipment and labor needed. In this paper we introduce a multi-resolution guided Generative Adversarial Network (GAN)-based framework for 3D medical image translation. Our framework uses a 3D multi-resolution Dense-Attention UNet (3D-mDAUNet) as the generator and a 3D multi-resolution UNet as the discriminator optimized with a unique combination of loss functions including voxel-wise GAN loss and 2.5D perception loss. Our approach yields promising results in volumetric image quality assessment (IQA) across a variety of imaging modalities body regions and age groups demonstrating its robustness. Furthermore we propose a synthetic-to-real applicability assessment as an additional evaluation to assess the effectiveness of synthetic data in downstream applications such as segmentation. This comprehensive evaluation shows that our method produces synthetic medical images not only of high-quality but also potentially useful in clinical applications. Our code is available at github.com/juhha/3D-mADUNet.
Ada-VE: Training-Free Consistent Video Editing using Adaptive Motion Prior
Authors: Tanvir Mahmud; Mustafa Munir; Radu Marculescu; Diana Marculescu
Venue: WACV 2025 (main)

Abstract: Video-to-video synthesis poses significant challenges in maintaining character consistency smooth temporal transitions and preserving visual quality during fast motion. While recent fully cross-frame self-attention mechanisms have improved character consistency across multiple frames they come with high computational costs and often include redundant operations especially for videos with higher frame rates. To address these inefficiencies we propose an adaptive motion-guided cross-frame attention mechanism that selectively reduces redundant computations. This enables a greater number of cross-frame attentions over more frames within the same computational budget thereby enhancing both video quality and temporal coherence. Our method leverages optical flow to focus on moving regions while sparsely attending to stationary areas allowing for the joint editing of more frames without increasing computational demands. Traditional frame interpolation techniques struggle with motion blur and flickering in intermediate frames which compromises visual fidelity. To mitigate this we introduce KV-caching for jointly edited frames reusing keys and values across intermediate frames to preserve visual quality and maintain temporal consistency throughout the video. With our adaptive cross-frame self-attention approach we achieve a threefold increase in the number of keyframes processed compared to existing methods all within the same computational budget as fully cross-frame attention baselines. This results in significant improvements in prediction accuracy and temporal consistency outperforming state-of-the-art approaches. Code is made publicly available at https://github.com/tanvir-utexas/AdaVE/tree/main.
Cross-Modal Feature Alignment and MMD Improve Robustness of Prompt Tuning
Authors: Jingchen Sun; Rohan Sharma; Vishnu Lokhande; Changyou Chen
Venue: WACV 2025 (main)

Abstract: Prompt Tuning has emerged as a prominent research paradigm for adapting vision-language models to various downstream tasks. However recent research indicates that prompt tuning methods often lead to overfitting due to limited training samples. In this paper we propose a Cross-modal Aligned Feature Tuning (CRAFT) method to address this issue. Cross-modal alignment is conducted by first selecting anchors from the alternative domain and deriving relative representations of the embeddings for the selected anchors. Optimizing for a feature alignment loss over anchor-aligned text and image modalities creates a more unified text-image common space. Overfitting in prompt tuning also deteriorates model performance on out-of-distribution samples. To further improve the prompt model's robustness we propose minimizing Maximum Mean Discrepancy (MMD) over the anchor-aligned feature spaces to mitigate domain shift. The experiment on four different prompt tuning structures consistently shows the improvement of our method with increases of up to 6.1% in the Base-to-Novel generalization task 5.8% in the group robustness task and 2.7% in the out-of-distribution tasks. The code is available at https://github.com/Jingchensun/Craft.
ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Model
Authors: Gaoge Han; Mingjiang Liang; Jinglei Tang; Yongkang Cheng; Wei Liu; Shaoli Huang
Venue: WACV 2025 (main)

Abstract: Generating human motion from textual descriptions is a challenging task. Existing methods either struggle with physical credibility or are limited by the complexities of physics simulations. In this paper we present ReinDiffuse that combines reinforcement learning with motion diffusion model to generate physically credible human motions that align with textual descriptions. Our method adapts Motion Diffusion Model to output a parameterized distribution of actions making them compatible with reinforcement learning paradigms. We employ reinforcement learning with the objective of maximizing physically plausible rewards to optimize motion generation for physical fidelity. Our approach outperforms existing state-of-the-art models on two major datasets HumanML3D and KIT-ML achieving significant improvements in physical plausibility and motion quality. Project: https://reindiffuse.github.io/
VerA: Versatile Anonymization Applicable to Clinical Facial Photographs
Authors: Majed El Helou; Doruk Cetin; Petar Stamenkovic; Niko Benjamin Huber; Fabio Zünd
Venue: WACV 2025 (main)

Abstract: The demand for privacy in facial image dissemination is gaining ground internationally echoed by the proliferation of regulations such as GDPR DPDPA CCPA PIPL and APPI. While recent advances in anonymization surpass pixelation or blur methods additional constraints to the task pose challenges. Largely unaddressed by current anonymization methods are clinical images and pairs of before-and-after clinical images illustrating facial medical interventions e.g. facial surgeries or dental procedures. We present VerA the first Versatile Anonymization framework that solves two challenges in clinical applications: A) it preserves selected semantic areas (e.g. mouth region) to show medical intervention results that is anonymization is only applied to the areas outside the preserved area; and B) it produces anonymized images with consistent personal identity across multiple photographs which is crucial for anonymizing photographs of the same person taken before and after a clinical intervention. We validate our results on both single and paired anonymization of clinical images through extensive quantitative and qualitative evaluation. We also demonstrate that VerA reaches the state of the art on established anonymization tasks in terms of photorealism and de-identification.
Event-Guided Fusion-Mamba for Context-Aware 3D Human Pose Estimation
Authors: Bo Lang; Mooi Choo Chuah
Venue: WACV 2025 (main)

Abstract: 3D human pose estimation (3D HPE) is an important computer vision task with various practical applications. Researchers have proposed various deep learning-based methods for 3D HPE. However the majority of such methods rely on lifting 2D pose sequence to 3D which do not perform well in challenging scenarios and are often computationally expensive. Such methods typically rely on 2D joint coordinates which do not provide much spatial context to solve ambiguity problem. In addition merely relying on information extracted from RGB frames may miss temporal information and structural context. Thus in this paper we propose a framework that incorporates event stream as an additional input since event features provide such useful information. Moreover instead of using 2D joint coordinates in pose sequence our framework uses intermediate visual representations produced by off-the-shelf 2D pose detectors to implicitly encode joint-centric spatial context. Our new framework is a novel state space model (SSM)-based solution called Event-Guided Context Aware MambaPose (CA-MambaPose). In CA-MambaPose framework we design a novel cross modality fusion mamba module to skillfully fuse the RGB and Event features. CA-MambaPose has lower computational cost due to the efficiency of Mamba blocks. We conduct extensive experiments to evaluate CA-MambaPose using two existing datasets. Our experimental results show that CA-MambaPose achieves better performance than SOTA methods.
MegaFusion: Extend Diffusion Models towards Higher-Resolution Image Generation without Further Tuning
Authors: Haoning Wu; Shaocheng Shen; Qiang Hu; Xiaoyun Zhang; Ya Zhang; Yanfeng Wang
Venue: WACV 2025 (main)

Abstract: Diffusion models have emerged as frontrunners in text-to-image generation but their fixed image resolution during training often leads to challenges in high-resolution image generation such as semantic deviations and object replication. This paper introduces MegaFusion a novel approach that extends existing diffusion-based text-to-image models towards efficient higher-resolution generation without additional fine-tuning or adaptation. Specifically we employ an innovative truncate and relay strategy to bridge the denoising processes across different resolutions allowing for high-resolution image generation in a coarse-to-fine manner. Moreover by integrating dilated convolutions and noise re-scheduling we further adapt the model's priors for higher resolution. The versatility and efficacy of MegaFusion make it universally applicable to both latent-space and pixel-space diffusion models along with other derivative models. Extensive experiments confirm that MegaFusion significantly boosts the capability of existing models to produce images of megapixels and various aspect ratios while only requiring about 40% of the original computational cost. Code is available at https://haoningwu3639.github.io/MegaFusion/.
Defending Against Repetitive Backdoor Attacks on Semi-Supervised Learning through Lens of Rate-Distortion-Perception Trade-Off
Authors: Cheng-Yi Lee; Ching-Chia Kao; Cheng-Han Yeh; Chun-Shien Lu; Chia-Mu Yu; Chu-Song Chen
Venue: WACV 2025 (main)

Abstract: Semi-supervised learning (SSL) has achieved remarkable performance with a small fraction of labeled data by leveraging vast amounts of unlabeled data from the Internet. However this large pool of untrusted data is extremely vulnerable to data poisoning leading to potential backdoor attacks. Current backdoor defenses are not yet effective against such a vulnerability in SSL. In this study we propose a novel method Unlabeled Data Purification (UPure) to disrupt the association between trigger patterns and target classes by introducing perturbations in the frequency domain. By leveraging the Rate-Distortion-Perception (RDP) trade-off we further identify the frequency band where the perturbations are added and justify this selection. Notably UPure purifies poisoned unlabeled data without the need of extra clean labeled data. Extensive experiments on four benchmark datasets and five SSL algorithms demonstrate that UPure effectively reduces the attack success rate from 99.78% to 0% while maintaining model accuracy. Code is available here: https://github.com/chengyi-chris/UPure.
BeautyBank: Encoding Facial Makeup in Latent Space
Authors: Qianwen Lu; Xingchao Yang; Takafumi Taketomi
Venue: WACV 2025 (main)

Abstract: The advancement of makeup transfer editing and image encoding has demonstrated their effectiveness and superior quality. However existing makeup works primarily focus on low-dimensional features such as color distributions and patterns limiting their versatillity across a wide range of makeup applications. Futhermore existing high-dimensional latent encoding methods mainly target global features such as structure and style and are less effective for tasks that require detailed attention to local color and pattern features of makeup. To overcome these limitations we propose BeautyBank a novel makeup encoder that disentangles pattern features of bare and makeup faces. Our method encodes makeup features into a high-dimensional space preserving essential details necessary for makeup reconstruction and broadening the scope of potential makeup research applications. We also propose a Progressive Makeup Tuning (PMT) strategy specifically designed to enhance the preservation of detailed makeup features while preventing the inclusion of irrelevant attributes. We further explore novel makeup applications including facial image generation with makeup injection and makeup similarity measure. Extensive empirical experiments validate that our method offers superior task adaptability and holds significant potential for widespread application in various makeup-related fields. Furthermore to address the lack of large-scale high-quality paired makeup datasets in the field we constructed the Bare-Makeup Synthesis Dataset (BMS) comprising 324000 pairs of 512x512 pixel images of bare and makeup-enhanced faces.
MVFNet: Multipurpose Video Forensics Network using Multiple Forms of Forensic Evidence
Authors: Tai D Nguyen; Matthew C Stamm
Venue: WACV 2025 (main)

Abstract: While videos can be falsified in many different ways most existing forensic networks are specialized to detect only a single manipulation type (e.g. deepfake inpainting). This poses a significant issue as the manipulation used to falsify a video is not known a priori. To address this problem we propose MVFNet - a multipurpose video forensics network capable of detecting multiple types of manipulations including inpainting deepfakes splicing and editing. Our network does this by extracting and jointly analyzing a broad set of forensic feature modalities that capture both spatial and temporal anomalies in falsified videos. To reliably detect and localize fake content of all shapes and sizes our network employs a novel Multi-Scale Hierarchical Transformer module to identify forensic inconsistencies across multiple spatial scales. Experimental results show that our network obtains state-of-the-art performance in general scenarios where multiple different manipulations are possible and rivals specialized detectors in targeted scenarios.
PV-VTT: A Privacy-Centric Dataset for Mission-Specific Anomaly Detection and Natural Language Interpretation
Authors: Ryozo Masukawa; Sanggeon Yun; Yoshiki Yamaguchi; Mohsen Imani
Venue: WACV 2025 (main)

Abstract: Video crime detection is a significant application of computer vision and artificial intelligence. However existing datasets primarily focus on detecting severe crimes by analyzing entire video clips often neglecting the precursor activities (i.e. privacy violations) that could potentially prevent these crimes. To address this limitation we present PV-VTT (Privacy Violation Video To Text) a unique multimodal dataset aimed at identifying privacy violations. PV-VTT provides detailed annotations for both video and text in scenarios. To ensure the privacy of individuals in the videos we only provide video feature vectors avoiding the release of any raw video data. This privacy-focused approach allows researchers to use the dataset while protecting participant confidentiality. Recognizing that privacy violations are often ambiguous and context-dependent we propose a Graph Neural Network (GNN)-based video description model. Our model generates a GNN-based prompt with an image for a Large Language Model (LLM) which delivers cost-effective and high-quality video descriptions. By leveraging a single video frame along with relevant text our method reduces the number of input tokens required maintaining descriptive quality while optimizing LLM API usage. Extensive experiments validate the effectiveness and interpretability of our approach in video description tasks and the flexibility of our PV-VTT dataset.
MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training
Authors: Chengyin Li; Hui Zhu; Rafi Ibn Sultan; Hassan Bagher Ebadian; Prashant Khanduri; Chetty Indrin; Kundan Thind; Dongxiao Zhu
Venue: WACV 2025 (main)

Abstract: In the diverse field of medical imaging automatic segmentation has numerous applications and must handle a wide variety of input domains such as different types of Computed Tomography (CT) scans and Magnetic Resonance (MR) images. This heterogeneity challenges automatic segmentation algorithms to maintain consistent performance across different modalities due to the requirement for spatially aligned and paired images. Typically segmentation models are trained using a single modality which limits their ability to generalize to other types of input data without employing transfer learning techniques. Additionally leveraging complementary information from different modalities to enhance segmentation precision often necessitates substantial modifications to popular encoder-decoder designs such as introducing multiple branched encoding or decoding paths for each modality. In this work we propose a simple Multi-Modal Segmentation (MulModSeg) strategy to enhance medical image segmentation across multiple modalities specifically CT and MR. It incorporates two key designs: a modality-conditioned text embedding framework via a frozen text encoder that adds modality awareness to existing segmentation frameworks without significant structural modifications or computational overhead and an alternating training procedure that facilitates the integration of essential features from unpaired CT and MR inputs. Through extensive experiments with both Fully Convolutional Network and Transformer-based backbones MulModSeg consistently outperforms previous methods in segmenting abdominal multi-organ and cardiac substructures for both CT and MR modalities. The code is available in this link.