GenAI4ELab/papercli-papers huggingface dataset:
DiaMond: Dementia Diagnosis with Multi-Modal Vision Transformers using MRI and PET
Authors: Yitong Li; Morteza Ghahremani; Youssef Wally; Christian Wachinger
Venue: WACV 2025 (main)

Abstract: Diagnosing dementia particularly for Alzheimer's Disease (AD) and frontotemporal dementia (FTD) is complex due to overlapping symptoms. While magnetic resonance imaging (MRI) and positron emission tomography (PET) data are critical for the diagnosis integrating these modalities in deep learning faces challenges often resulting in suboptimal performance compared to using single modalities. Moreover the potential of multi-modal approaches in differential diagnosis which holds significant clinical importance remains largely unexplored. We propose a novel framework DiaMond to address these issues with vision Transformers to effectively integrate MRI and PET. DiaMond is equipped with self-attention and a novel bi-attention mechanism that synergistically combine MRI and PET alongside a multi-modal normalization to reduce redundant dependency thereby boosting the performance. DiaMond significantly outperforms existing multi-modal methods across various datasets achieving a balanced accuracy of 92.4% in AD diagnosis 65.2% for AD-MCI-CN classification and 76.5% in differential diagnosis of AD and FTD. We also validated the robustness of DiaMond in a comprehensive ablation study. The code is available at https://github.com/ai-med/DiaMond.
Deep Metric Learning for Unsupervised Remote Sensing Change Detection
Authors: Wele Gedara Chaminda Bandara; Vishal M. Patel
Venue: WACV 2025 (main)

Abstract: Remote Sensing Change Detection (RS-CD) aims to detect relevant changes from Multi-Temporal Remote Sensing Images (MT-RSIs) which aids in various RS applications such as land cover land use human development analysis and disaster response. The performance of existing RS-CD methods is attributed to training on large annotated datasets. Furthermore most of these models are less transferable in the sense that the trained model often performs very poorly when there is a domain gap between training and test datasets. This paper proposes an unsupervised CD method based on deep metric learning that can deal with both of these issues. Given an MT-RSI the proposed method generates corresponding change probability map by iteratively optimizing an unsupervised CD loss without training it on a large dataset. Our unsupervised CD method consists of two interconnected deep networks namely Deep-Change Probability Generator (D-CPG) and Deep-Feature Extractor (D-FE). The D-CPG is designed to predict change and no change probability maps for a given MT-RSI while D-FE is used to extract deep features of MT-RSI that will be further used in the proposed unsupervised CD loss. We use transfer learning capability to initialize the parameters of D-FE. We iteratively optimize the parameters of D-CPG and D-FE for a given MT-RSI by minimizing the proposed unsupervised similarity-dissimilarity loss. This loss is motivated by the principle of metric learning where we simultaneously maximize the distance between change pair-wise pixels while minimizing the distance between no-change pair-wise pixels in bi-temporal image domain and their deep feature domain. The experiments conducted on three CD datasets show that our unsupervised CD method achieves significant improvements over the state-of-the-art supervised and unsupervised CD methods. Code and pre-trained models available at https://github.com/wgcban/Metric-CD
Learning Semi-Supervised Medical Image Segmentation from Spatial Registration
Authors: Qianying Liu; Paul Henderson; Xiao Gu; Hang Dai; Fani Deligianni
Venue: WACV 2025 (main)

Abstract: Semi-supervised medical image segmentation has shown promise in training models with limited labeled data and abundant unlabeled data. However state-of-the-art methods ignore a potentially valuable source of unsupervised semantic information--spatial registration transforms between image volumes. To address this we propose CCT-R a contrastive cross-teaching framework incorporating registration information. To leverage the semantic information available in registrations between volume pairs CCT-R incorporates two proposed modules: Registration Supervision Loss (RSL) and Registration-Enhanced Positive Sampling (REPS). The RSL leverages segmentation knowledge derived from transforms between labeled and unlabeled volume pairs providing an additional source of pseudo-labels. REPS enhances contrastive learning by identifying anatomically-corresponding positives across volumes using registration transforms. Experimental results on two challenging medical segmentation benchmarks demonstrate the effectiveness and superiority of CCT-R across various semi-supervised settings with as few as one labeled case. Our code is available at https://github.com/kathyliu579/ContrastiveCrossteachingWithRegistration.
Towards High-Fidelity Head Blending with Chroma Keying for Industrial Applications
Authors: Hah Min Lew; Sahng-Min Yoo; Hyunwoo Kang; Gyeong-Moon Park
Venue: WACV 2025 (main)

Abstract: We introduce an industrial Head Blending pipeline for the task of seamlessly integrating an actor's head onto a target body in digital content creation. The key challenge stems from discrepancies in head shape and hair structure which lead to unnatural boundaries and blending artifacts. Existing methods treat foreground and background as a single task resulting in suboptimal blending quality. To address this problem we propose CHANGER a novel pipeline that decouples background integration from foreground blending. By utilizing chroma keying for artifact-free background generation and introducing Head shape and long Hair augmentation (H2 augmentation) to simulate a wide range of head shapes and hair styles CHANGER improves generalization on innumerable various real-world cases. Furthermore our Foreground Predictive Attention Transformer (FPAT) module enhances foreground blending by predicting and focusing on key head and body regions. Quantitative and qualitative evaluations on benchmark datasets demonstrate that our CHANGER outperforms state-of-the-art methods delivering high-fidelity industrial-grade results.
LiGAR: LiDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognition
Authors: Naga Venkata Sai Raviteja Chappa; Khoa Luu
Venue: WACV 2025 (main)

Abstract: Group Activity Recognition (GAR) remains challenging in computer vision due to the complex nature of multi-agent interactions. This paper introduces LiGAR a LIDAR-Guided Hierarchical Transformer for Multi-Modal Group Activity Recognition. LiGAR leverages LiDAR data as a structural backbone to guide the processing of visual and textual information enabling robust handling of occlusions and complex spatial arrangements. Our framework incorporates a Multi-Scale LIDAR Transformer Cross-Modal Guided Attention and an Adaptive Fusion Module to integrate multi-modal data at different semantic levels effectively. LiGAR's hierarchical architecture captures group activities at various granularities from individual actions to scene-level dynamics. Extensive experiments on the JRDB-PAR Volleyball and NBA datasets demonstrate LiGAR's superior performance achieving state-of-the-art results with improvements of up to 10.6% in F1-score on JRDB-PAR and 5.9% in Mean Per Class Accuracy on the NBA dataset. Notably LiGAR maintains high performance even when LiDAR data is unavailable during inference showcasing its adaptability. Our ablation studies highlight the significant contributions of each component and the effectiveness of our multi-modal multi-scale approach in advancing the field of group activity recognition.
Unsupervised Video Highlight Detection by Learning from Audio and Visual Recurrence
Authors: Zahidul Islam; Sujoy Paul; Mrigank Rochan
Venue: WACV 2025 (main)

Abstract: With the exponential growth of video content the need for automated video highlight detection to extract key moments or highlights from lengthy videos has become increasingly pressing. This technology has the potential to enhance user experiences by allowing quick access to relevant content across diverse domains. Existing methods typically rely either on expensive manually labeled frame-level annotations or on a large external dataset of videos for weak supervision through category information. To overcome this we focus on unsupervised video highlight detection eliminating the need for manual annotations. We propose a novel unsupervised approach which capitalizes on the premise that significant moments tend to recur across multiple videos of the similar category in both audio and visual modalities. Surprisingly audio remains under-explored especially in unsupervised algorithms despite its potential to detect key moments. Through a clustering technique we identify pseudo-categories of videos and compute audio pseudo-highlight scores for each video by measuring the similarities of audio features among audio clips of all the videos within each pseudo-category. Similarly we also compute visual pseudo-highlight scores for each video using visual features. Then we combine audio and visual pseudo-highlights to create the audio-visual pseudo ground-truth highlight of each video for training an audio-visual highlight detection network. Extensive experiments and ablation studies on three benchmarks showcase the superior performance of our method over prior work.
Attention-Based Class-Conditioned Alignment for Multi-Source Domain Adaptation of Object Detectors
Authors: Atif Belal; Akhil Meethal; Francisco Perdigon Romero; Marco Pedersoli; Eric Granger
Venue: WACV 2025 (main)

Abstract: Domain adaptation methods for object detection (OD) strive to mitigate the impact of distribution shifts by promoting feature alignment across source and target domains. Multi-source domain adaptation (MSDA) allows leveraging multiple annotated source datasets and unlabeled target data to improve the accuracy and robustness of the detection model. Most state-of-the-art MSDA methods for OD perform feature alignment in a class-agnostic manner. This is challenging since the objects have unique modality information due to variations in object appearance across domains. A recent prototype-based approach proposed a class-wise alignment yet it suffers from error accumulation caused by noisy pseudo-labels that can negatively affect adaptation with imbalanced data. To overcome these limitations we propose an attention-based class-conditioned alignment method for MSDA designed to align instances of each object category across domains. In particular an attention module combined with an adversarial domain classifier allows learning domain-invariant and class-specific instance representations. Experimental results on multiple benchmarking MSDA datasets indicate that our method outperforms state-of-the-art methods and exhibits robustness to class imbalance achieved through a conceptually simple class-conditioning strategy. Our code is available at: https://github.com/imatif17/ACIA.
Stable Autofocus with Focal Consistency Loss
Authors: Sangwon Lee; Myungsub Choi; Nagyeong Lee; Hyong-Euk Lee
Venue: WACV 2025 (main)

Abstract: Autofocus aims to accurately position the camera lens to bring the desired region of interest into focus. Conventional works search for the sharpest frame within the lens movement. However sharpness measure in many real-world settings is ambiguous and may cause a focus hunting problem where the lens continuously moves back and forth to search for the accurate position. To mitigate this problem we introduce a simple yet powerful loss function specifically designed to produce consistent outputs in autofocus systems. The proposed Focal Consistency Loss (FCL) allows autofocus models to better learn the geometric cues relative to each initial position of the lens significantly reducing distracting lens movement and enhancing the user experience when taking a photo. Furthermore we improve autofocus stability by utilizing multiple consecutive frames in a practical way. Experimental results show the effectiveness of FCL in various practical scenarios including multi-frame autofocus for both conventional and dual-pixel images.
Bandit Based Attention Mechanism in Vision Transformers
Authors: Amartya Roy Chowdhury; Raghuram Bharadwaj Diddigi; Prabuchandran K J; Achyut Mani Tripathi
Venue: WACV 2025 (main)

Abstract: Vision Transformers (ViT) have demonstrated remarkable performance on many computer vision tasks. However their high computational cost and quadratic complexity pose challenges for deployment in resource-constrained environments. The core of Vision Transformers is the self-attention mechanism which aggregates information from different image regions or patches. In a conventional ViT processing involves attention to all patches creating a substantial computational bottleneck and extended training times. We hypothesize that applying soft attention to all patches may be unnecessary and instead focusing on relevant and significant patches (hard attention) would be sufficient. To address this we introduce a module within the Vision Transformer that allows the attention mechanism to selectively process only the essential patches. We propose a novel bandit-based attention mechanism that leverages the idea of exploration and exploitation. The extensive experimentation across various datasets illustrates that the proposed bandit attention-based ViT not only achieves superior performance compared to the existing state-of-the-art vision transformer models but also results in greater throughput and lower computational time in the training as well as the inference. The code is publicly available at https://github.com/aquorio15/bandit wacv
Semantically Conditioned Prompts for Visual Recognition under Missing Modality Scenarios
Authors: Vittorio Pipoli; Federico Bolelli; Sara Sarto; Marcella Cornia; Lorenzo Baraldi; Costantino Grana; Rita Cucchiara; Elisa Ficarra
Venue: WACV 2025 (main)

Abstract: This paper tackles the domain of multimodal prompting for visual recognition specifically when dealing with missing modalities through multimodal Transformers. It presents two main contributions: (i) we introduce a novel prompt learning module which is designed to produce sample-specific prompts and (ii) we show that modality-agnostic prompts can effectively adjust to diverse missing modality scenarios. Our model termed SCP exploits the semantic representation of available modalities to query a learnable memory bank which allows the generation of prompts based on the semantics of the input. Notably SCP distinguishes itself from existing methodologies for its capacity of self-adjusting to both the missing modality scenario and the semantic context of the input without prior knowledge about the specific missing modality and the number of modalities. Through extensive experiments we show the effectiveness of the proposed prompt learning framework and demonstrate enhanced performance and robustness across a spectrum of missing modality cases. Our source code is available at https://github.com/vittoriopipoli/SCP_WACV2025.
OccFlowNet: Occupancy Estimation via Differentiable Rendering and Occupancy Flow
Authors: Simon Boeder; Benjamin Risse
Venue: WACV 2025 (main)

Abstract: Semantic occupancy has recently gained significant traction as a prominent 3D scene representation. However most existing camera-based methods rely on large and costly datasets with fine-grained 3D voxel labels for training which limits their practicality and scalability. Furthermore approaches in this domain lack the modelling of scene dynamics. In this work we present a novel approach to occupancy estimation inspired by neural radiance field (NeRF) using supervision in 2D based on 3D labels provided by LiDAR that offers a more natural way of supervision than voxel labels. In particular we employ differentiable volumetric rendering to predict depth and semantic maps and train a 3D network based on supervision in 2D space only. To enhance geometric accuracy and increase the supervisory signal we introduce temporal rendering of adjacent time steps. Additionally we introduce occupancy flow as a mechanism to handle dynamic objects in the scene and ensure their temporal consistency. Through extensive experimentation we demonstrate that supervision in 2D with LiDAR can achieve state-of-the-art performance compared to methods using voxel labels and when combining it with voxel supervision in 3D temporal rendering and occupancy flow we outperform all previous occupancy estimation models significantly. We conclude that the proposed rendering supervision and occupancy flow advances occupancy estimation.
Cross-Domain Multi-Modal Few-Shot Object Detection via Rich Text
Authors: Zeyu Shangguan; Daniel Seita; Mohammad Rostami
Venue: WACV 2025 (main)

Abstract: Cross-modal feature extraction and integration have led to steady performance improvements in few-shot learning tasks. However existing multi-modal object detection (MM-OD) methods degrade when facing significant domain shift and are sample insufficient. We hypothesize that rich text information could more effectively help the model to build a knowledge relationship between the vision instance and its language description and can help mitigate domain shift. Specifically we study the Cross-Domain few-shot generalization of MM-OD (CDMM-FSOD) and propose a meta-learning based multi-modal few-shot object detection method that utilizes rich text semantic information as an auxiliary modality to achieve domain adaptation. Our proposed novel neural network contains a multi-modal feature aggregation module that aligns the vision and language support feature embeddings and a rich text semantic rectify module that utilizes bidirectional text feature generation to reinforce multi-modal feature alignment and thus to enhance the model's language understanding capability. We evaluate our model on common standard cross-domain object detection datasets and demonstrate that our approach considerably outperforms existing FSOD methods. Our implementation is publicly available: https://github.com/zshanggu/CDMM
Local Masked Reconstruction for Efficient Self-Supervised Learning on High-Resolution Images
Authors: Jun Chen; Faizan Farooq Khan; Ming Hu; Ammar Sherif; Zongyuan Ge; Boyang Li; Mohamed Elhoseiny
Venue: WACV 2025 (main)

Abstract: Self-supervised learning for computer vision has progressed tremendously and improved many downstream vision tasks such as image classification semantic segmentation and object detection. Among these generative self-supervised vision learning approaches such as MAE and BEiT show promising performance. However their global reconstruction mechanism is computationally demanding especially for high-resolution images. The computational cost increases extensively when scaled to a large-scale dataset. To address this issue we propose local masked reconstruction (LoMaR) a simple yet effective approach that reconstructs image patches from small neighboring regions. The strategy can be easily integrated into any generative self-supervised learning techniques and improves the trade-off between efficiency and accuracy compared to reconstruction over the entire image. LoMaR is 2.5x faster than MAE and 5.0x faster than BEiT on 384x384 ImageNet pretraining and surpasses them by 0.2% and 0.8% in accuracy respectively. It is 2.1x faster than MAE on iNaturalist pretraining and gains 0.2% in accuracy. On MS COCO LoMaR outperforms MAE by 0.5 APbox on object detection and 0.5 APmask on instance segmentation. It also outperforms MAE by 0.2% on semantic segmentation. Our code and pretrained models are available at: https://github.com/junchen14/LoMaR.
PureForest: A Large-Scale Aerial Lidar and Aerial Imagery Dataset for Tree Species Classification in Monospecific Forests
Authors: Charles Gaydon; Floryne Roche
Venue: WACV 2025 (main)

Abstract: Sustainable forest management is a cornerstone of climate and environmental action. Responsible management relies on forest models such as biomass or fire vulnerability estimates which depend on mapping the spatial distribution of tree species. Because forest mapping relies heavily on visual identification it is a time-consuming endeavor that would benefit from more automation. To develop scalable automated methods researchers need suitable benchmarks but unfortunately tree species classification datasets are scarce. Most of them contain only imagery data which capture signal only at the canopy level and ignore the three-dimensional structure of the forest making it very difficult to distinguish between different species. Lidar which penetrates the canopy and captures the geometry of trees provides a rich signal for distinguishing tree species. As an increasingly available and cost-effective remote sensing modality it could become the new standard for large-scale tree species mapping. However current Lidar benchmarks for tree species classification are extremely limited and all insufficient in size and diversity; with less than 1500 annotated trees from at most a dozen different sites they fall short from the requirements of deep learning development. To fill this data gap we release PureForest: a large-scale open multimodal dataset designed for tree species classification from both Aerial Lidar Scanning point clouds and Very High Resolution aerial images. PureForest covers 339 km2 in 449 different monospecific forests with verified labels for 18 tree species grouped into 13 semantic classes. It is the largest and most comprehensive Lidar dataset for tree species identification. By making PureForest publicly available we hope to provide a challenging benchmark to support the development of deep learning approaches for tree species identification from Lidar and aerial imagery. In this data paper we describe the annotation workflow and the dataset and we establish a baseline performance from both 2D and 3D modalities.
ACE: Anatomically Consistent Embeddings in Composition and Decomposition
Authors: Ziyu Zhou; Haozhe Luo; Mohammad Reza Hosseinzadeh Taher; Jiaxuan Pang; Xiaowei Ding; Michael Gotway; Jianming Liang
Venue: WACV 2025 (main)

Abstract: Medical images acquired from standardized protocols show consistent macroscopic or microscopic anatomical structures and these structures consist of composable/decomposable organs and tissues but existing self-supervised learning (SSL) methods do not appreciate such composable/decomposable structure attributes inherent to medical images. To overcome this limitation this paper introduces a novel SSL approach called ACE to learn anatomically consistent embedding via composition and decomposition with two key branches: (1) global consistency capturing discriminative macro-structures via extracting global features; (2) local consistency learning fine-grained anatomical details from composable/decomposable patch features via corresponding matrix matching. Experimental results across 6 datasets and 2 backbones evaluated in few-shot learning fine-tuning and property analysis show ACE's superior robustness transferability and clinical potential. The innovations of our ACE lie in grid-wise image cropping leveraging the intrinsic properties of compositionality and decompositionality of medical images bridging the semantic gap from high-level pathologies to low-level tissue anomalies and providing a new SSL method for medical imaging.