GenAI4ELab/papercli-papers huggingface dataset:
SoundLoc3D: Invisible 3D Sound Source Localization and Classification using a Multimodal RGB-D Acoustic Camera
Authors: Yuhang He; Sangyun Shin; Anoop Cherian; Niki Trigoni; Andrew Markham
Venue: WACV 2025 (main)

Abstract: Accurately localizing 3D sound sources and estimating their semantic labels - where the sources may not be visible but are assumed to lie on the physical surface of objects in the scene - have many real applications including detecting gas leak and machinery malfunction. The audio-visual weak- correlation in such setting poses new challenges in deriving innovative methods to answer if or how we can use cross- modal information to solve the task. Towards this end we propose to use an acoustic-camera rig consisting of a pinhole RGB-D camera and a coplanar four-channel microphone array (Mic-Array). By using this rig to record audio-visual signals from multiviews we can use the cross-modal cues to estimate the sound sources 3D locations. Specifically our framework SoundLoc3D treats the task as a set prediction problem each element in the set corresponds to a potential sound source. Given the audio-visual weak-correlation the set representation is initially learned from a single view microphone array signal and then refined by actively incorporating physical surface cues revealed from multiview RGB-D images. We demonstrate the efficiency and superiority of SoundLoc3D on large-scale simulated dataset and further show its robustness to RGB-D measurement inaccuracy and ambient noise interference.
Enhancing Monocular Depth Estimation with Multi-Source Auxiliary Tasks
Authors: Alessio Quercia; Erenus Yildiz; Zhuo Cao; Kai Krajsek; Abigail Morrison; Ira Assent; Hanno Scharr
Venue: WACV 2025 (main)

Abstract: Monocular depth estimation (MDE) is a challenging task in computer vision often hindered by the cost and scarcity of high-quality labeled datasets. We tackle this challenge using auxiliary datasets from related vision tasks for an alternating training scheme with a shared decoder built on top of a pre-trained vision foundation model while giving a higher weight to MDE. Through extensive experiments we demonstrate the benefits of incorporating various in-domain auxiliary datasets and tasks to improve MDE quality on average by  11%. Our experimental analysis shows that auxiliary tasks have different impacts confirming the importance of task selection highlighting that quality gains are not achieved by merely adding data. Remarkably our study reveals that using semantic segmentation datasets as Multi-Label Dense Classification (MLDC) often results in additional quality gains. Lastly our method significantly improves the data efficiency for the considered MDE datasets enhancing their quality while reducing their size by at least 80%. This paves the way for using auxiliary data from related tasks to improve MDE quality despite limited availability of high-quality labeled data. Code is available at https://jugit.fz-juelich.de/ias-8/mdeaux.
Neural SDF for Shadow-Aware Unsupervised Structured Light
Authors: Kazuto Ichimaru; Diego Thomas; Takafumi Iwaguchi; Hiroshi Kawasaki
Venue: WACV 2025 (main)

Abstract: Among various active 3D measurement techniques Structured Light (SL) is one of the most popular method for its robustness and high accuracy. Ordinary SL system consists of a camera and a projector and by projecting a pre-defined pattern we can obtain pixel-to-pixel correspondences between the camera and the projector for triangulation. However if we lack the knowledge of the projected pattern for some reason e.g. the projected pattern is not as expected due to lens distortion inaccurate calibration undesired optical phenomenon like inter-reflection and so on the accuracy of conventional SL is severally degraded. As a remedy we propose unsupervised structured light (USSL) which does not explicitly use prior knowledge on the pattern. Inspired by the fact that human can recognize the scene structure illuminated by unknown light source (e.g. rotating mirror ball) and some prior work have succeeded in novel-view-synthesis under unknown illumination conditions we implement USSL on Neural Signed Distance Fields (Neural SDF) pipeline with implicit reflection module powered by neural network. Additionally since every SL method causes occlusion (shadow) by pattern projection we must take it into account for accurate shape reconstruction. To this end we integrate shadow volume rendering into the proposed pipeline. Experiments with synthetic and real dataset are conducted to confirm the feasibility of the proposed method.
Can Multimodal Large Language Models Truly Perform Multimodal In-Context Learning?
Authors: Shuo Chen; Zhen Han; Bailan He; Jianzhe Liu; Mark Buckley; Yao Qin; Philip Torr; Volker Tresp; Jindong Gu
Venue: WACV 2025 (main)

Abstract: Large Language Models (LLMs) with in-context learning (ICL) ability can quickly adapt to a specific context given a few demonstrations (demos). Recently Multimodal Large Language Models (MLLMs) built upon LLMs have also shown multimodal ICL ability i.e. responding to queries given a few multimodal demos including images queries and answers. While ICL has been extensively studied on LLMs its research on MLLMs remains limited. One essential question is whether these MLLMs can truly conduct multimodal ICL or if only the textual modality is necessary. We investigate this question by examining two primary factors that influence ICL: 1) Demo content i.e. understanding the influences of demo content in different modalities. 2) Demo selection strategy i.e. how to select better multimodal demos for improved performance. Experiments revealed that multimodal ICL is predominantly driven by the textual content whereas the visual information in the demos has little influence. Interestingly visual content is still necessary and useful for selecting demos to increase performance. Motivated by our analysis we propose a simple yet effective approach termed Mixed Modality In-Context Example Selection (MMICES) which considers both visual and language modalities when selecting demos. Extensive experiments are conducted to support our findings and verify the improvement brought by our method.
I Spy with My Little Eye A Minimum Cost Multicut Investigation of Dataset Frames
Authors: Katharina Prasse; Isaac Bravo; Stefanie Walter; Margret Keuper
Venue: WACV 2025 (main)

Abstract: Visual framing analysis is a key method in social sciences for determining common themes and concepts in a given discourse. To reduce manual effort image clustering can significantly speed up the annotation process. In this work we phrase the clustering task as a Minimum Cost Multicut Problem [MP]. Solutions to the MP have been shown to provide clusterings that maximize the posterior probability solely from provided local pairwise probabilities of two images belonging to the same cluster. We discuss the efficacy of numerous embedding spaces to detect visual frames and show its superiority over other clustering methods. To this end we employ the climate change dataset ClimateTV which contains images commonly used for visual frame analysis. For broad visual frames DINOv2 is a suitable embedding space while ConvNeXt V2 returns a larger number of clusters which contain fine-grain differences i.e. speech and protest. Our insights into embedding space differences in combination with the optimal clustering - by definition - advances automated visual frame detection. Our code can be found at https://github.com/KathPra/MP4VisualFrameDetection.
A Semantically Impactful Image Manipulation Dataset: Characterizing Image Manipulations using Semantic Significance
Authors: Yuwei Chen; Ming-Ching Chang; Mattias Kirchner; Zhenfei Zhang; Xin Li; Arslan Basharat; Anthony Hoogs
Venue: WACV 2025 (main)

Abstract: We investigate how to characterize semantic significance (SS) in detecting image manipulations (IMD) for media forensics. We introduce the Characterization of Semantic Impact for IMD (CSI-IMD) dataset which focuses on localizing and evaluating the semantic impact of image manipulations to counter advanced generative techniques. Our evaluation of 10 state-of-the-art IMD and localization methods on CSI-IMD reveals key insights. Unlike existing datasets CSI-IMD provides detailed semantic annotations beyond traditional manipulation masks aiding in the development of new defensive strategies. The dataset features manipulations from advanced generation methods offering various levels of semantic significance. It is divided into two parts: a gold-standard set of 1000 manually annotated manipulations with high-quality control and an extended set of 500000 automated manipulations for large-scale training and analysis. We also propose a new SS-focused task to assess the impact of semantically targeted manipulations. Our experiments show that current IMD methods struggle with manipulations created using stable diffusion with TruFor and Cat-Net performing the best among those tested. The CSI-IMD dataset will become available at https://github.com/csiimd/csiimd.
Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformers
Authors: Tobias Christian Nauen; Sebastian Palacio; Federico Raue; Andreas Dengel
Venue: WACV 2025 (main)

Abstract: Self-attention in Transformers comes with a high computational cost because of their quadratic computational complexity but their effectiveness in addressing problems in language and vision has sparked extensive research aimed at enhancing their efficiency. However diverse experimental conditions spanning multiple input domains prevent a fair comparison based solely on reported results posing challenges for model selection. To address this gap in comparability we perform a large-scale benchmark of more than 45 models for image classification evaluating key efficiency aspects including accuracy speed and memory usage. Our benchmark provides a standardized baseline for efficiency-oriented transformers. We analyze the results based on the Pareto front - the boundary of optimal models. Surprisingly despite claims of other models being more efficient ViT remains Pareto optimal across multiple metrics. We observe that hybrid attention-CNN models exhibit remarkable inference memory- and parameter-efficiency. Moreover our benchmark shows that using a larger model in general is more efficient than using higher resolution images. Thanks to our holistic evaluation we provide a centralized resource for practitioners and researchers facilitating informed decisions when selecting or developing efficient transformers.
MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations
Authors: Reza Akbarian Bafghi; Nidhin Harilal; Maziar Raissi; Claire Monteleoni
Venue: WACV 2025 (main)

Abstract: This paper introduces MixDiff a new self-supervised learning (SSL) pre-training framework that combines real and synthetic images. Unlike traditional SSL methods that predominantly use real images MixDiff uses a variant of Stable Diffusion to replace an augmented instance of a real image facilitating the learning of cross real-synthetic image representations.  Our key insight is that while models trained solely on synthetic images underperform combining real and synthetic data leads to more robust and adaptable representations. Experiments show MixDiff enhances SimCLR BarlowTwins and DINO across various robustness datasets and domain transfer tasks boosting SimCLR's ImageNet-1K accuracy by 4.56%. Our framework also demonstrates comparable performance without needing any augmentations a surprising finding in SSL where augmentations are typically crucial. Furthermore MixDiff achieves similar results to SimCLR while requiring less real data highlighting its efficiency in representation learning
Covariance-Based Space Regularization for Few-Shot Class Incremental Learning
Authors: Yijie Hu; Guanyu Yang; Zhaorui Tan; Xiaowei Wang; Kaizhu Huang; Qiu-Feng Wang
Venue: WACV 2025 (main)

Abstract: Few-shot Class Incremental Learning (FSCIL) presents a challenging yet realistic scenario which requires the model to continually learn new classes with limited labeled data (i.e. incremental sessions) while retaining knowledge of previously learned base classes (i.e. base sessions). Due to the limited data in incremental sessions models are prone to overfitting new classes and suffering catastrophic forgetting of base classes. To tackle these issues recent advancements resort to prototype-based approaches to constrain the base class distribution and learn discriminative representations of new classes. Despite the progress the limited data issue still induces ill-divided feature space leading the model to confuse the new class with old classes or fail to facilitate good separation among new classes. In this paper we aim to mitigate these issues by directly constraining the span of each class distribution from a covariance perspective. In detail we propose a simple yet effective covariance constraint loss to force the model to learn each class distribution with the same covariance matrix. In addition we propose a perturbation approach to perturb the few-shot training samples in the feature space which encourages the samples to be away from the weighted distribution of other classes. Regarding perturbed samples as new class data the classifier is forced to establish explicit boundaries between each new class and the existing ones. Our approach is easy to integrate into existing FSCIL approaches to boost performance. Experiments on three benchmarks validate the effectiveness of our approach achieving state-of-the-art performance.
Structured Human Assessment of Text-to-Image Generative Models
Authors: Ciprian A. Corneanu; Qianli Feng; Aleix M. Martinez
Venue: WACV 2025 (main)

Abstract: Following the great progress in text-conditioned image generation there is a dire need for establishing clear comparison benchmarks. Unfortunately assessing performance of such models is highly subjective and notoriously difficult. Current automatic assessment of generated images quality and their alignment to text are approximate at best while human assessment is subjective poorly calibrated and not very well defined. To address these concerns we propose GenomeBench a new framework for assessing quality of text-to-image generative models. It consists of a prompt dataset richly annotated with semantic components based on a formalized grounding of language and images. On top of it we define a procedure to collect human assessment through a carefully guided question answering process. Finally these assessments are summarized into a novel score built around quality and alignment to text. We show the proposal achieves higher inter-annotator agreement with respect to the baseline human assessment and better correlation between quality and alignment compared to automatic assessment. Finally we use this framework to dissect the performance of recent text-to-image models providing insights on strength and weakness of each.
Unsupervised Single-Image Intrinsic Image Decomposition with LiDAR Intensity Enhanced Training
Authors: Shogo Sato; Takuhiro Kaneko; Kazuhiko Murasaki; Taiga Yoshida; Ryuichi Tanida; Akisato Kimura
Venue: WACV 2025 (main)

Abstract: Unsupervised intrinsic image decomposition (IID) is the task of separating a natural image into albedo and shade without ground truth during training. Although a recent model employing light detection and ranging (LiDAR) intensity demonstrated impressive performance the necessity of LiDAR intensity during inference restricts its practicality. To expand the usage scenario while maintaining the IID quality achieved by using both an image and its corresponding LiDAR intensity we propose a novel approach that utilizes an image without LiDAR intensity during inference while utilizing both an image and LiDAR intensity during training. Specifically our proposed model processes an image and LiDAR intensity individually using distinct encoder paths during training but utilizes only an image-encoder path during inference. Additionally we introduce an albedo-alignment loss aligning the gray-scale albedo from an image to that from its corresponding LiDAR intensity. LiDAR intensity is not affected by illumination effects including cast shadows thus albedo-alignment loss transfers the illumination-invariant property of LiDAR intensity to the image-encoder path. Furthermore we also propose image-LiDAR conversion (ILC) paths that mutually translates the style of an image and LiDAR intensity. IID models translate an image into albedo and shade styles while keeping the image contents thus it is important to separate the image into contents and style. Trained with pairs of an image and its corresponding LiDAR intensity which share contents but differ in style the mutual translation in ILC paths improve the accuracy of the separation. Consequently our model achieves comparable IID quality to the existing model with LiDAR intensity while utilizing only an image without LiDAR intensity during inference.
3D Synthesis for Architectural Design
Authors: I-Ting Tsai; Bharath Hariharan
Venue: WACV 2025 (main)

Abstract: We introduce a 3D synthesis method for architectural design to allow for the efficient generation of diverse and realistic building designs. In spite of advances in 3D synthesis current off-the-shelf 3D synthesis techniques are inappropriate for architectural design: they are trained primarily on isolated objects have limited diversity blend building facades with background and produce overly complex geometry that is difficult to edit or manipulate a major issue in an iterative design process. We propose an alternative pipeline that integrates auto-generated coarse models with segment-wise texture inpainting and semantics-based editing resulting in diverse style-consistent and shape-precise designs. We show through qualitative and quantitative experiments that our pipeline generates more diverse visually appealing architectures with clean geometries without the need for any extensive training.
Assessing Visually-Continuous Corruption Robustness of Neural Networks Relative to Human Performance
Authors: Huakun Shen; Boyue Hu; Krzysztof Czarnecki; Lina Marsso; Marsha Chechik
Venue: WACV 2025 (main)

Abstract: Neural Networks (NNs) have surpassed human accuracy in image classification on ImageNet yet they often lack robustness against image corruption i.e. corruption robustness with such robustness being seemingly effortless for human perception. In this paper we propose visually-continuous corruption robustness (VCR) - an extension of corruption robustness to allow assessing it over the wide and continuous range of changes that correspond to the human perceptive quality (i.e. from the original image to the full distortion of all perceived visual information) along with two novel human-aware metrics for NN evaluation. To compare VCR of NNs with human perception we conducted extensive experiments on 14 commonly used image corruptions with 7718 human participants and state-of-the-art robust NN models with different training objectives (e.g. standard adversarial corruption robustness) different architectures (e.g. convolution NNs vision transformers) and different amounts of training data augmentation. Our study showed that: 1) assessing robustness against continuous corruption can reveal insufficient robustness undetected by existing benchmarks; as a result 2) the gap between NN and human robustness is larger than previously known; and finally 3) some image corruptions have a similar impact on human perception offering opportunities for more cost-effective robustness assessments.
SMDAF: A Scalable Sidewalk Material Data Acquisition Framework with Bidirectional Cross-Modal Knowledge Distillation
Authors: Jiawei Liu; Wayne Lam; Zhigang Zhu; Hao Tang
Venue: WACV 2025 (main)

Abstract: Ensuring safe and independent navigation poses considerable difficulties for individuals who are blind or have low vision (BLV) as it requires detailed knowledge of their immediate environment. Our research highlights the critical need for accessible data on sidewalk materials and objects which is currently lacking in existing map services. To bridge this gap we present the Sidewalk Material Data Acquisition Framework (SMDAF) designed for large-scale data collection. This framework includes (1) a lightweight data collection system embedded in a white cane which captures audio data through the interaction of the cane tip with the sidewalk surface and a mobile app that facilitates data storage and management resulting in a novel multimodal dataset comprising both image and audio data; and (2) a unique Cross-Modal Knowledge Distillation (CMKD) technique for an enhanced audio material classifier. Our CMKD approach employs an image-based model as the teacher to improve the audio model incorporating an Enhanced Bidirectional learning method with an intuitive filtering technique: Bidirectional Correct Sample Filtering (BCSF). BCSF filters correct samples to prevent the distillation of incorrect knowledge addressing the issue of inaccurate cross-modal learning. This novel approach has resulted in a 1.84% improvement in Macro Accuracy achieving an overall accuracy of 87.62% surpassing all state-of-the-art KD and CMKD methods. This study underscores the efficacy of SMDAF and provides a practical CMKD technique for future cross-modal learning tasks. Code and dataset are available https://github.com/FgSurewin/SMDAF-CMKD.
Epipolar Attention Field Transformers for Bird's Eye View Semantic Segmentation
Authors: Christian Witte; Jens Behley; Cyrill Stachniss; Marvin Raaijmakers
Venue: WACV 2025 (main)

Abstract: Spatial understanding of the semantics of the surroundings is a key capability needed by autonomous cars to enable safe driving decisions. Recently purely vision-based solutions have gained increasing research interest. In particular approaches extracting a bird's eye view (BEV) from multiple cameras have demonstrated great performance for spatial understanding. This paper addresses the dependency on learned positional encodings to correlate image and BEV feature map elements for transformer-based methods. We propose leveraging epipolar geometric constraints to model the relationship between cameras and the BEV by Epipolar Attention Fields. They are incorporated into the attention mechanism as a novel attribution term serving as an alternative to learned positional encodings. Experiments show that our method EAFormer outperforms previous BEV approaches by 2% mIoU for map semantic segmentation and exhibits superior generalization capabilities compared to implicitly learning the camera configuration.