GenAI4ELab/papercli-papers huggingface dataset:
Multi-HexPlanes: A Lightweight Map Representation for Rendering and 3D Reconstruction
Authors: Jianhao Zheng; Gábor Valasek; Daniel Barath; Iro Armeni
Venue: WACV 2025 (main)

Abstract: Creating maps of the world around us is paramount to many applications including those related to robotics such as navigation and inspection. Given the computational resource limitations typical of robotic platforms there is a pressing need for lightweight 3D representations that capture detailed texture and geometric information with minimal storage. Traditional voxel-based approaches require substantial memory resources. On the other hand neural implicit and 3D Gaussian splatting representations require significant computational power (GPUs) and can hardly run in real-time. In this paper we introduce a novel scene representation Multi-HexPlanes that divides 3D environments into large boxes and utilizes the faces of the boxes to encapsulate texture and geometric information. This representation reduces the memory requirement to store the map making our approach especially suitable for systems with limited memory. Through extensive evaluations on large-scale datasets we find that our method achieves better performance on rendering and more complete 3D reconstruction. We also demonstrate that our map representation can output dense feature points with rich geometric information for downstream tasks such as training 3D Gaussian splats. The proposed technique promises substantial improvements in real-time 3D mapping applications particularly for devices constrained by processing power and storage.
MLLM-Tool: A Multimodal Large Language Model for Tool Agent Learning
Authors: Chenyu Wang; Weixin Luo; Sixun Dong; Xiaohua Xuan; Zhengxin Li; Lin Ma; Shenghua Gao
Venue: WACV 2025 (main)

Abstract: Recently the astonishing performance of large language models (LLMs) in natural language comprehension and generation tasks triggered lots of exploration of using them as central controllers to build agent systems. Multiple studies focus on bridging the LLMs to external tools to extend the application scenarios. However the current LLMs' ability to perceive tool use is limited to a single text query which may result in ambiguity in understanding the users' real intentions. LLMs are expected to eliminate that by perceiving the information in the visual- or auditory-grounded instructions. Therefore in this paper we propose MLLM-Tool a system incorporating open-source LLMs and multi-modal encoders so that the learned LLMs can be conscious of multi-modal input instruction and then select the function-matched tool correctly. To facilitate the evaluation of the model's capability we collect a dataset featuring multi-modal input tools from HuggingFace. Another essential feature of our dataset is that it also contains multiple potential choices for the same instruction due to the existence of identical functions and synonymous functions which provides more potential solutions for the same query. The experiments reveal that our MLLM-Tool is capable of recommending appropriate tools for multi-modal instructions. Codes and data are available at github.com/MLLMTool/MLLM-Tool.
Skyeyes: Ground Roaming using Aerial View Images
Authors: Zhiyuan Gao; Wenbin Teng; Gonglin Chen; Jinsen Wu; Ningli Xu; Rongjun Qin; Andrew Feng; Yajie Zhao
Venue: WACV 2025 (main)

Abstract: Integrating aerial imagery-based scene generation into applications like autonomous driving and gaming enhances realism in 3D environments but challenges remain in creating detailed content for occluded areas and ensuring real-time consistent rendering. In this paper we introduce Skyeyes a novel framework that can generate photorealistic sequences of ground view images using only aerial view inputs thereby creating a ground roaming experience. More specifically we combine a 3D representation with a view consistent generation model which ensures coherence between generated images. This method allows for the creation of geometrically consistent ground view images even with large view gaps. The images maintain improved spatial-temporal coherence and realism enhancing scene comprehension and visualization from aerial perspectives. To the best of our knowledge there are no publicly available datasets that contain pairwise geo-aligned aerial and ground view imagery. Therefore we build a large synthetic and geo-aligned dataset using Unreal Engine. Both qualitative and quantitative analyses on this synthetic dataset display superior results compared to other leading synthesis approaches. See the project page for more results: https://chaoren2357.github.io/website-skyeyes/
Closing the Domain Gap in Manga Colorization via Aligned Paired Dataset
Authors: Maksim Golyadkin; Ianis Plevokas; Ilya Makarov
Venue: WACV 2025 (main)

Abstract: This paper addresses the challenge of artwork colorization by proposing a benchmark for manga colorization using real black-and-white and colorized image pairs. Color images are widely recognized for their ability to capture attention and improve memory retention yet the manual process of colorization is labor-intensive. Deep learning methods for supervised image-to-image translation offer a promising solution relying on aligned pairs of black-and-white and color images for training. However these pairs are often generated synthetically introducing a domain gap that limits model performance. To address this we explore the use of real data proposing a method for creating such datasets. Our benchmarks reveal that models trained on real data significantly outperform those trained on synthetic pairs. Furthermore we present a pipeline for text removal and panel segmentation streamlining the comic colorization process. These contributions aim to enhance the generalization and applicability of deep learning models for artwork colorization.
VaLID: Variable-Length Input Diffusion for Novel View Synthesis
Authors: Shijie Li; Farhad G. Zanjani; Haitam Ben Yahia; Yuki Asano; Juergen Gall; Amirhossein Habibian
Venue: WACV 2025 (main)

Abstract: Novel View Synthesis (NVS) which tries to produce a realistic image at the target view given source view images and their corresponding poses is a fundamental problem in 3D Vision. As this task is heavily under-constrained some recent work like Zero123 [18] tries to solve this problem with generative modeling specifically using pre-trained diffusion models. Although this strategy generalizes well to new scenes compared to neural radiance field-based methods it offers low levels of flexibility. For example it can only accept a single-view image as input despite realistic applications often offering multiple input images. This is because the source-view images and corresponding poses are processed separately and injected into the model at different stages. Thus it is not trivial to generalize the model into multi-view source images once they are available. To solve this issue we try to process each pose image pair separately and then fuse them as a unified visual representation which will be injected into the model to guide image synthesis at the target-views. However inconsistency and computation costs increase as the number of input source-view images increases. To solve these issues the Multi-view Cross Former module is proposed which maps variable-length input data to fix-size output data. A two-stage training strategy is introduced to further improve the efficiency during training time. Qualitative and quantitative evaluation over multiple datasets demonstrates the effectiveness of the proposed method against previous approaches. The code will be released according to the acceptance.
Divergent Domains Convergent Grading: Enhancing Generalization in Diabetic Retinopathy Grading
Authors: Sharon Chokuwa; Muhammad Haris Khan
Venue: WACV 2025 (main)

Abstract: Diabetic Retinopathy (DR) constitutes 5% of global blindness cases. While numerous deep learning approaches have sought to enhance traditional DR grading methods they often falter when confronted with new out-of-distribution data thereby impeding their widespread application. In this study we introduce a novel deep learning method for achieving domain generalization (DG) in DR grading and make the following contributions. First we propose a new way of generating image-to-image diagnostically relevant fundus augmentations conditioned on the grade of the original fundus image. These augmentations are tailored to emulate the types of shifts in DR datasets thus increase the model's robustness. Second we address the limitations of the standard classification loss in DG for DR fundus datasets by proposing a new DG-specific loss - domain alignment loss; which ensures that the feature vectors from all domains corresponding to the same class converge onto the same manifold for better domain generalization. Third we tackle the coupled problem of data imbalance across DR domains and classes by proposing to employ Focal loss which seamlessly integrates with our new alignment loss. Fourth due to inevitable observer variability in DR diagnosis that induces label noise we propose leveraging self-supervised pretraining. This approach ensures that our DG model remains robust against early susceptibility to label noise even when only a limited dataset of non-DR fundus images is available for pretraining. Our method demonstrates significant improvements over the strong Empirical Risk Minimization baseline and other recently proposed state-of-the-art DG methods for DR grading. Code is available at https://github.com/sharonchokuwa/dg-adr.
Relational Self-Supervised Distillation with Compact Descriptors for Image Copy Detection
Authors: Juntae Kim; Sungwon Woo; Jongho Nang
Venue: WACV 2025 (main)

Abstract: Image copy detection is the task of detecting edited copies of any image within a reference database. While previous approaches have shown remarkable progress the large size of their networks and descriptors remains a dis-advantage complicating their practical application. In this paper we propose a novel method that achieves competitive performance by using a lightweight network and compact descriptors. By utilizing relational self-supervised distillation to transfer knowledge from a large network to a small network we enable the training of lightweight networks with smaller descriptor sizes. We introduce relational self-supervised distillation for flexible representation in a smaller feature space and apply contrastive learning with a hard negative loss to prevent dimensional collapse. For the DISC2021 benchmark ResNet-50 and EfficientNet-B0 are used as the teacher and student models respectively with micro average precision improving by 5.0%/4.9%/5.9% for 64/128/256 descriptor sizes compared to the baseline method. The code is available at https://github.com/juntae9926/RDCD.
FlashMix: Fast Map-Free LiDAR Localization via Feature Mixing and Contrastive-Constrained Accelerated Training
Authors: Raktim Gautam Goswami; Naman Patel; Prashanth Krishnamurthy; Farshad Khorrami
Venue: WACV 2025 (main)

Abstract: Map-free LiDAR localization systems accurately localize within known environments by predicting sensor position and orientation directly from raw point clouds eliminating the need for large maps and descriptors. However their long training times hinder rapid adaptation to new environments. To address this we propose FlashMix which uses a frozen scene-agnostic backbone to extract local point descriptors aggregated with an MLP mixer to predict sensor pose. A buffer of local descriptors is used to accelerate training by orders of magnitude combined with metric learning or contrastive loss regularization of aggregated descriptors to improve performance and convergence. We evaluate FlashMix on various LiDAR localization benchmarks examining different regularizations and aggregators and demonstrating its effectiveness for rapid and accurate LiDAR localization in real-world scenarios. The code is available at https://github.com/raktimgg/FlashMix.
EI-Nexus: Towards Unmediated and Flexible Inter-Modality Local Feature Extraction and Matching for Event-Image Data
Authors: Zhonghua Yi; Hao Shi; Qi Jiang; Kailun Yang; Ze Wang; Diyang Gu; Yufan Zhang; Kaiwei Wang
Venue: WACV 2025 (main)

Abstract: Event cameras with high temporal resolution and high dynamic range have limited research on the inter-modality local feature extraction and matching of event-image data. We propose EI-Nexus an unmediated and flexible framework that integrates two modality-specific keypoint extractors and a feature matcher. To achieve keypoint extraction across viewpoint and modality changes we bring Local Feature Distillation (LFD) which transfers the viewpoint consistency from a well-learned image extractor to the event extractor ensuring robust feature correspondence. Furthermore with the help of Context Aggregation (CA) a remarkable enhancement is observed in feature matching. We further establish the first two inter-modality feature matching benchmarks MVSEC-RPE and EC-RPE to assess relative pose estimation on event-image data. Our approach outperforms traditional methods that rely on explicit modal transformation offering more unmediated and adaptable feature extraction and matching achieving better keypoint similarity and state-of-the-art results on the MVSEC-RPE and EC-RPE benchmarks. The source code and benchmarks will be made publicly available at EI-Nexus.
Towards Robust Training via Gradient-Diversified Backpropagation
Authors: Xilin He; Cheng Luo; Qinliang Lin; Weicheng Xie; Muhammad Haris Khan; Siyang Song; Linlin Shen
Venue: WACV 2025 (main)

Abstract: Neural networks are prone to be vulnerable to adversarial attacks and domain shifts. Adversarial-driven methods including adversarial training and adversarial augmentation have been frequently proposed to improve the model's robustness against adversarial attacks and distribution-shifted samples. Nonetheless recent research on adversarial attacks has cast a spotlight on the robustness lacuna against attacks targeted at deep semantic layers. Our analysis reveals that previous adversarial-driven methods tend to generate overpowering perturbations in deep semantic layers leading to distortion of the training for these layers. This can be primarily attributed to the exclusive utilization of loss functions on the output layer for adversarial gradient generation. This inherent practice projects an excessive adversarial impact on the deep semantic layers elevating the difficulty of training such layers. Therefore from the standing point of relaxing the excessive perturbations in the deep semantic layer and diversifying the adversarial gradients to ensure robust training for deep semantic layers this paper proposes a novel Stochastic Loss Integration Method (SLIM) which can be instantiated into the existing adversarial-driven methods in a plug-and-play manner. Experimental results across diverse tasks including classification and segmentation as well as various areas such as adversarial robustness and domain generalization validate the effectiveness of our proposed method. Furthermore we provide an in-depth analysis to offer a comprehensive understanding of layer-wise training involving various loss terms.
A Spatio-Temporal Representation Learning as an Alternative to Traditional Glosses in Sign Language Translation and Production
Authors: Eui Jun Hwang; Sukmin Cho; Huije Lee; Youngwoo Yoon; Jong C. Park
Venue: WACV 2025 (main)

Abstract: This work addresses the challenges associated with the use of glosses in both Sign Language Translation (SLT) and Sign Language Production (SLP). While the glosses have long been used as a bridge between sign language and spoken language they come with two major limitations that impede the advancement of sign language systems. First annotating the glosses is a labor-intensive and time-consuming process which limits the scalability of datasets. Second the glosses oversimplify sign language by stripping away its spatio-temporal dynamics reducing complex signs to basic labels and missing the subtle movements essential for precise interpretation. To address these limitations we introduce Universal Gloss-level Representation (UniGloR) a framework designed to capture the spatio-temporal features inherent in sign language providing a more dynamic and detailed alternative to the use of the glosses. The core idea of UniGloR is simple yet effective: We derive dense spatio-temporal representations from sign keypoint sequences using self-supervised learning and seamlessly integrate them into SLT and SLP tasks. Our experiments in a keypoint-based setting demonstrate that UniGloR either outperforms or matches the performance of previous SLT and SLP methods on two widely-used datasets: PHOENIX14T and How2Sign. Code is available at https://github.com/eddie-euijun-hwang/UniGloR.
Benchmarking VLMs' Reasoning About Persuasive Atypical Images
Authors: Sina Malakouti; Aysan Aghazadeh; Ashmit Khandelwal; Adriana Kovashka
Venue: WACV 2025 (main)

Abstract: Vision language models (VLMs) have shown strong zero-shot generalization across various tasks especially when integrated with large language models (LLMs). However their ability to comprehend rhetorical and persuasive visual media such as advertisements remains understudied. Ads often employ atypical imagery using surprising object juxtapositions to convey shared properties. For example Fig. 1 (e) shows a beer with a feather-like texture. This requires advanced reasoning to deduce that this atypical representation signifies the beer's lightness. We introduce three novel tasks Multi-label Atypicality Classification Atypicality Statement Retrieval and Atypical Object Recognition to benchmark VLMs' understanding of atypicality in persuasive images. We evaluate how well VLMs use atypicality to infer an ad's message and test their reasoning abilities by employing semantically challenging negatives. Finally we pioneer atypicality-aware verbalization by extracting comprehensive image descriptions sensitive to atypical elements. Findings reveal that: (1) VLMs lack advanced reasoning capabilities compared to LLMs; (2) simple effective strategies can extract atypicality-aware information leading to comprehensive image verbalization; (3) atypicality aids persuasive ad understanding. Code and data is available at aysanaghazadeh.github.io/PersuasiveAdVLMBenchmark/
Uncertainty Awareness Enables Efficient Labeling for Cancer Subtyping in Digital Pathology
Authors: Nirhoshan Sivaroopan; Chamuditha Jayanga Galappaththige; Chalani Ekanayake; Hasindri Watawana; Ranga Rodrigo; Chamira U.S. Edussooriya; Dushan N. Wadduwage
Venue: WACV 2025 (main)

Abstract: Machine-learning-assisted cancer subtyping is a promising avenue in digital pathology. Cancer subtyping models however require careful training using expert annotations so that they can be inferred with a degree of known certainty (or uncertainty). To this end we introduce the concept of uncertainty awareness into a self-supervised contrastive learning model. This is achieved by computing an evidence vector at every epoch which assesses the model's confidence in its predictions. The derived uncertainty score is then utilized as a metric to selectively label the most crucial images that require further annotation thus iteratively refining the training process. With just 1-10% of strategically selected annotations we attain state-of-the-art performance in cancer subtyping on benchmark datasets. Our method not only strategically guides the annotation process to minimize the need for extensive labeled datasets but also improve the precision and efficiency of classifications. This development is particularly beneficial in settings where the availability of labeled data is limited offering a promising direction for future research and application in digital pathology. Our code is available at https://github.com/Nirhoshan/AI-for-histopathology
3D Part Segmentation via Geometric Aggregation of 2D Visual Features
Authors: Marco Garosi; Riccardo Tedoldi; Davide Boscaini; Massimiliano Mancini; Nicu Sebe; Fabio Poiesi
Venue: WACV 2025 (main)

Abstract: Supervised 3D part segmentation models are tailored for a fixed set of objects and parts limiting their transferability to open-set real-world scenarios. Recent works have explored vision-language models (VLMs) as a promising alternative using multi-view rendering and textual prompting to identify object parts. However naively applying VLMs in this context introduces several drawbacks such as the need for meticulous prompt engineering and fails to leverage the 3D geometric structure of objects. To address these limitations we propose COPS a COmprehensive model for Parts Segmentation that blends the semantics extracted from visual concepts and 3D geometry to effectively identify object parts. COPS renders a point cloud from multiple viewpoints extracts 2D features projects them back to 3D and uses a novel geometric-aware feature aggregation procedure to ensure spatial and semantic consistency. Finally it clusters points into parts and labels them. We demonstrate that COPS is efficient scalable and achieves zero-shot state-of-the-art performance across five datasets covering synthetic and real-world data texture-less and coloured objects as well as rigid and non-rigid shapes. The code is available at https://3d-cops.github.io.
BioPose: Biomechanically-Accurate 3D Pose Estimation from Monocular Videos
Authors: Farnoosh Koleini; Muhammad Usama Saleem; Pu Wang; Hongfei Xue; Ahmed Helmy; Abbey Fenwick
Venue: WACV 2025 (main)

Abstract: Recent advancements in 3D human pose estimation from single-camera images and videos have relied on parametric models like SMPL. However these models oversimplify anatomical structures limiting their accuracy in capturing true joint locations and movements which reduces their applicability in biomechanics healthcare and robotics. Biomechanically accurate pose estimation on the other hand typically requires costly marker-based motion capture systems and optimization techniques in specialized labs. To bridge this gap we propose BioPose a novel learning-based framework for predicting biomechanically accurate 3D human pose directly from monocular videos. BioPose includes three key components: a multi-query human mesh recovery model (MQ-HMR) a neural inverse kinematics (NeurIK) model and a 2D-informed pose refinement technique. MQ-HMR leverages a multi-query deformable transformer to extract multi-scale fine-grained image features enabling precise human mesh recovery. NeurIK treats the mesh vertices as virtual markers applying a spatial-temporal network to regress biomechanically accurate 3D poses under anatomical constraints. To further improve 3D pose estimations a 2D-informed refinement step optimizes the query tokens during inference by aligning the 3D structure with 2D pose observations. Experiments on benchmark datasets demonstrate that BioPose significantly outperforms state-of-the-art methods.
DPA: Dual Prototypes Alignment for Unsupervised Adaptation of Vision-Language Models
Authors: Eman Ali; Sathira Silva; Muhammad Haris Khan
Venue: WACV 2025 (main)

Abstract: Vision-language models (VLMs) e.g. CLIP have shown remarkable potential in zero-shot image classification. However adapting these models to new domains remains challenging especially in unsupervised settings where labeled data is unavailable. Recent research has proposed pseudo-labeling approaches to adapt CLIP in an unsupervised manner using unlabeled target data. Nonetheless these methods struggle due to noisy pseudo-labels resulting from the misalignment between CLIP's visual and textual representations. This study introduces DPA an unsupervised domain adaptation method for VLMs. DPA introduces the concept of dual prototypes acting as distinct classifiers along with the convex combination of their outputs thereby leading to accurate pseudo-label construction. Next it ranks pseudo-labels to facilitate robust self-training particularly during early training. Finally it addresses visual-textual misalignment by aligning textual prototypes with image prototypes to further improve the adaptation performance. Experiments on 13 downstream vision tasks demonstrate that DPA significantly outperforms zero-shot CLIP and the state-of-the-art unsupervised adaptation baselines.