GenAI4ELab/papercli-papers huggingface dataset:
FlashVTG: Feature Layering and Adaptive Score Handling Network for Video Temporal Grounding
Authors: Zhuo Cao; Bingqing Zhang; Heming Du; Xin Yu; Xue Li; Sen Wang
Venue: WACV 2025 (main)

Abstract: Text-guided Video Temporal Grounding (VTG) aims to localize relevant segments in untrimmed videos based on textual descriptions encompassing two subtasks: Moment Retrieval (MR) and Highlight Detection (HD). Although previous typical methods have achieved commendable results it is still challenging to retrieve short video moments. This is primarily due to the reliance on sparse and limited decoder queries which significantly constrain the accuracy of predictions. Furthermore suboptimal outcomes often arise because previous methods rank predictions based on isolated predictions neglecting the broader video context. To tackle these issues we introduce FlashVTG a framework featuring a Temporal Feature Layering (TFL) module and an Adaptive Score Refinement (ASR) module. The TFL module replaces the traditional decoder structure to capture nuanced video content variations across multiple temporal scales while the ASR module improves prediction ranking by integrating context from adjacent moments and multi-temporal-scale features. Extensive experiments demonstrate that FlashVTG achieves state-of-the-art performance on four widely adopted datasets in both MR and HD. Specifically on the QVHighlights dataset it boosts mAP by 5.8% for MR and 3.3% for HD. For short-moment retrieval FlashVTG increases mAP to 125% of previous SOTA performance. All these improvements are made without adding training burdens underscoring its effectiveness.
Fine-Grained Spatial and Verbal Losses for 3D Visual Grounding
Authors: Sombit Dey; Ozan Unal; Christos Sakaridis; Luc Van Gool
Venue: WACV 2025 (main)

Abstract: 3D visual grounding consists of identifying the instance in a 3D scene which is referred to by an accompanying language description. While several architectures have been proposed within the commonly employed grounding-by-selection framework the utilized losses are comparatively under-explored. In particular most methods rely on a basic supervised cross-entropy loss on the predicted distribution over candidate instances which fails to model both spatial relations between instances and the internal fine-grained word-level structure of the verbal referral. Sparse attempts to additionally supervise verbal embeddings globally by learning the class of the referred instance from the description or employing verbo-visual contrast to better separate instance embeddings do not fundamentally lift the aforementioned limitations. Responding to these shortcomings we introduce two novel losses for 3D visual grounding: a visual-level offset loss on regressed vector offsets from each instance to the ground-truth referred instance and a language-related span loss on predictions for the word-level span of the referred instance in the description. In addition we equip the verbo-visual fusion module of our new 3D visual grounding architecture AsphaltNet with a top-down bidirectional attentive fusion block which enables the supervisory signals from our two losses to propagate to the respective converse branches of the network and thus aid the latter to learn context-aware instance embeddings and grounding-aware verbal embeddings. AsphaltNet proposes novel auxiliary losses to aid 3D visual grounding with competitive results compared to the state-of-the-art on the ReferIt3D benchmark.
Towards On-the-Fly Novel Category Discovery in Dynamic Long-Tailed Distributions
Authors: Hoin Jung; Xiaoqian Wang
Venue: WACV 2025 (main)

Abstract: As the diversity of real-world object categories increases the need for sophisticated classification methods also grows. However Novel Category Discovery (NCD) which aims to predict unseen categories often falls short in scenarios where new categories are constantly updated and data distributions are potentially biased. In addition existing dynamic NCD approaches assume that incremental stages introduce a fixed number of new classes and often overlook distributional biases in real-world classes. To address these limitations we propose a novel framework Novel Category Discovery for Dynamic Long-Tailed distribution (NCD-DLT) which deals with the more realistic and challenging scenario where imbalanced unlabeled data are introduced incrementally and sporadically over time. Unlike conventional methods requiring k-means clustering on all test samples our approach identifies novel categories on-the-fly predicting categories for individual data points as they arrive. We propose an advanced hash-based clustering technique leveraging a double-hashing strategy to mitigate collisions and incorporating a greedy hash regularization loss for sparse representations to enhance clustering capabilities. Furthermore we implement distillation losses during training to preserve the model's discriminative power across stages without forgetting prior knowledge. Finally we introduce a novel graph merging algorithm based on the Hash Hamming Graph revealing the dataset's clustering structure. It serves as a mechanism for pseudo-labeling in training and acts as a post-processing tool reallocating less confident samples to more appropriate clusters. Our comprehensive approach addresses the limitations of existing NCD methods in the dynamic scenario of novel category discovery in long-tailed distributions demonstrating improved accuracy for both uniform and long-tailed scenarios.
Survival Prediction in Lung Cancer through Multi-Modal Representation Learning
Authors: Aiman Farooq; Deepak Mishra; Santanu Chaudhury
Venue: WACV 2025 (main)

Abstract: Survival prediction is a crucial task associated with cancer diagnosis and treatment planning. This paper presents a novel approach to survival prediction by harnessing comprehensive information from CT and PET scans along with associated Genomic data. Current methods rely on either a single modality or the integration of multiple modalities for prediction without adequately addressing associations across patients or modalities. We aim to develop a robust predictive model for survival outcomes by integrating multi-modal imaging data with genetic information while accounting for associations across patients and modalities. We learn representations for each modality via a self-supervised module and harness the semantic similarities across the patients to ensure the embeddings are aligned closely. However optimizing solely for global relevance is inadequate as many pairs sharing similar high-level semantics such as tumor type are inadvertently pushed apart in the embedding space. To address this issue we use a cross-patient module (CPM) designed to harness inter-subject correspondences. The CPM module aims to bring together embeddings from patients with similar disease characteristics. Our experimental evaluation of the dataset of Non-Small Cell Lung Cancer (NSCLC) patients demonstrates the effectiveness of our approach in predicting survival outcomes outperforming state-of-the-art methods.
PALO: A Polyglot Large Multimodal Model for 5B People
Authors: Hanoona Rasheed; Muhammad Maaz; Abdelrahman Shaker; Salman Khan; Hisham Cholakkal; Rao M. Anwer; Tim Baldwin; Michael Felsberg; Fahad S. Khan
Venue: WACV 2025 (main)

Abstract: In pursuit of more inclusive Vision-Language Models (VLMs) this study introduces a Large Multilingual Multimodal Model called PALO. PALO offers visual reasoning capabilities in 10 major languages including English Chinese Hindi Spanish French Arabic Bengali Russian Urdu and Japanese that span a total of  5B people (65% of the world population). Our approach involves a semi-automated translation approach to adapt the multimodal instruction dataset from English to the target languages using a fine-tuned Large Language Model thereby ensuring high linguistic fidelity while allowing scalability due to minimal manual effort. The incorporation of diverse instruction sets helps us boost overall performance across multiple languages especially those that are underrepresented like Hindi Arabic Bengali and Urdu. The resulting models are trained across three scales (1.7B 7B and 13B parameters) to show the generalization and scalability where we observe substantial improvements compared to strong baselines. We also propose the first multilingual multimodal benchmark for the forthcoming approaches to evaluate their vision-language reasoning capabilities across languages. Code: https://github.com/mbzuai-oryx/PALO.
Uncertainty-Aware Regularization for Image-to-Image Translation
Authors: Anuja Vats; Ivar Farup; Marius Pedersen; Kiran Raja
Venue: WACV 2025 (main)

Abstract: The importance of quantifying uncertainty in deep networks has become paramount for reliable real-world applications. In this paper we propose a method to improve uncertainty estimation in medical Image-to-Image (I2I) translation. Our model integrates aleatoric uncertainty and employs Uncertainty-Aware Regularization (UAR) inspired by simple priors to refine uncertainty estimates and enhance reconstruction quality. We show that by leveraging simple priors on parameters our approach captures more robust uncertainty maps effectively refining them to indicate precisely where the network encounters difficulties while being less affected by noise. Our experiments demonstrate that UAR not only improves translation performance but also provides better uncertainty estimations particularly in the presence of noise and artifacts. We validate our approach using two medical imaging datasets showcasing its effectiveness in maintaining high confidence in familiar regions while accurately identifying areas of uncertainty in novel/ambiguous scenarios.
Revisiting Deep Archetypal Analysis for Phenotype Discovery in High Content Imaging
Authors: Mario Wieser; Daniel Siegismund; Stephan Steigele
Venue: WACV 2025 (main)

Abstract: The discovery of unique treatment candidates for complex diseases is a challenging task for current drug discovery programs. Biopharma research has developed automated and scalable screening assays of cell culture models to screen thousands of drug candidates in parallel e.g. by considering bio-image based assays. However the large amount of data hinders a systematic review by human experts to distinguish between different disease and healthy phenotypes. A prevalent approach to uncover phenotypic endpoints in a dataset is based on the concept of archetypal analysis which seeks for extremal points in a dataset. State-of-the-art non-linear archetypal methods based on variational autoencoders require k - 1 latent dimensions to encode k archetypes. However in high content imaging we frequently require a significantly larger number of latent dimensions than archetypes to encode HCIs which results in weak latent representations and ambiguous archetypes. To overcome this limitation we propose to relax the simplex constraint in the latent space to a unit hypersphere and learn the respective archetypes based on online dictionary learning. Extensive experiments on two industry-relevant assays and a synthetic MNIST example demonstrate that our method outperforms state-of-the-art deep archetypal analysis approaches.
GaussianBeV : 3D Gaussian Representation Meets Perception Models for BeV Segmentation
Authors: Florian Chabot; Nicolas Granger; Guillaume Lapouge
Venue: WACV 2025 (main)

Abstract: The Bird's-eye View (BeV) representation is widely used for 3D perception from multi-view camera images. It allows to merge features from different cameras into a common space providing a unified representation of the 3D scene. The key component is the view transformer which transforms image views into the BeV. However actual view transformer methods based on geometry or cross-attention do not provide a sufficiently detailed representation of the scene as they use a sub-sampling of the 3D space that is non-optimal for modeling the fine structures of the environment. In this paper we propose GaussianBeV a novel method for transforming image features to BeV by finely representing the scene using a set of 3D gaussians located and oriented in 3D space. This representation is then splattered to produce the BeV feature map by adapting recent advances in 3D representation rendering based on gaussian splatting. GaussianBeV is the first approach to use this 3D gaussian modeling and 3D scene rendering process in an optimization free manner i.e. without optimizing it on a specific scene and directly integrated into a single stage model for BeV scene understanding. Experiments show that the proposed representation is highly effective and place GaussianBeV as the new state-of-the-art on the BeV semantic segmentation task on the nuScenes dataset.
PC-GZSL: Prior Correction for Generalized Zero Shot Learning
Authors: S Divakar Bhat; Amit More; Mudit Soni; Bhuvan Aggarwal
Venue: WACV 2025 (main)

Abstract: Generalized Zero Shot Learning (GZSL) aims at achieving a good accuracy on both seen and unseen classes by relying on the information acquired from auxiliary attributes. Existing approaches have devised many frameworks to make this knowledge transfer more efficient and informative. Despite their effectiveness on boosting the overall performance there has always been a strong bias in the model towards the seen classes which makes GZSL problem more challenging. The effect of this bias on the model performance has never been properly explored. We observe that GZSL algorithms in literature have an evident bias towards the seen classes. Further we also show that techniques like calibrated stacking fall short of resolving this conflict between the seen and unseen classes effectively. In this work we analyze and develop a logit-adjustment approach in GZSL setting and propose a simple yet effective method to remove the bias from trained models in a post-hoc manner. Moreover as a consequence of the post-hoc nature of the proposed approach there is no additional training cost. We exhaustively compare the proposed method on both embedding-based and generative-based GZSL frameworks surpassing the SOTA results by 3.1% 4.6% and 3.1% on CUB SUN and AwA2 datasets. We also present theoretical analysis showing effectiveness of proposed approach.
Multi-Scale Grouped Prototypes for Interpretable Semantic Segmentation
Authors: Hugo Porta; Emanuele Dalsasso; Diego Marcos; Devis Tuia
Venue: WACV 2025 (main)

Abstract: Prototypical part learning is emerging as a promising approach for making semantic segmentation interpretable. The model selects real patches seen during training as prototypes and constructs the dense prediction map based on the similarity between parts of the test image and the prototypes. This improves interpretability since the user can inspect the link between the predicted output and the patterns learned by the model in terms of prototypical information. In this paper we propose a method for interpretable semantic segmentation that leverages multi-scale image representation for prototypical part learning. First we introduce a prototype layer that explicitly learns diverse prototypical parts at several scales leading to multi-scale representations in the prototype activation output. Then we propose a sparse grouping mechanism that produces multi-scale sparse groups of these scale-specific prototypical parts. This provides a deeper understanding of the interactions between multi-scale object representations while enhancing the interpretability of the segmentation model. The experiments conducted on Pascal VOC Cityscapes and ADE20K demonstrate that the proposed method increases model sparsity improves interpretability over existing prototype-based methods and narrows the performance gap with the non-interpretable counterpart models. Code is available at github.com/eceo-epfl/ScaleProtoSeg.
Effective and Efficient Medical Image Segmentation with Hierarchical Context Interaction
Authors: Zehua Cheng; Di Yuan; Wenhu Zhang; Thomas Lukasiewicz
Venue: WACV 2025 (main)

Abstract: The U-Net models have become the predominant architecture within the domain of medical image segmentation. Recent advancements have showcased the potential of incorporating attention-based techniques into U-Net structures. Nevertheless the inclusion of attention mechanisms often leads to a substantial increase in both computational demands and the number of parameters with only a marginal improvement in the performance. This observation raises a critical evaluation of the efficiency associated with the integration of attention modules. In this paper we propose a novel methodology termed Hierarchical Context Interaction (HCI) a parameter-efficient attention-free enhancement that can be seamlessly incorporated into U-Net-based models. Experimental results demonstrate that our proposed HCI module attains state-of-the-art performance on two widely used benchmarks i.e. Medical Segmentation Decathlon Datasets and Synapse Datasets while concurrently sustaining a computationally efficient profile comparable to conventional U-Net configurations.
A 0-Shot Self-Attention Mechanism for Accelerated Diagonal Attention
Authors: Mario Viti; Nadiya Shvai; Arcadi Llanza; Amir Nakib
Venue: WACV 2025 (main)

Abstract: The ability of Transformers to process longer sequences has led to unprecedented levels of generalization in visual tasks. However the complexity of Transformers is dominated by the quadratic cost associated with the computation of the attention blocks posing a bottleneck that impedes the scaling of sequence length and the realization of more advanced AI solutions. We propose and explore the hypothesis that the self-attention mechanism exhibits regularities that can be exploited to enhance performance and achieve linear-cost attention without significant loss of effectiveness. Specifically we investigate the attention matrix of Visual Transformers to identify and leverage these regularities in order to simplify the computation process. The resulting procedure significantly reduces the computational cost of Transformers by directly reducing attention block complexity. Moreover the designed procedure is 0-shot self-supervised thus it requires no retraining additional data or parameters as all Transformer parameters remain unchanged. Consequently the proposed method can be seamlessly applied to pre-trained Visual Transformers without the need for retraining. Experiments conducted on a series of Vision Transformers pre-trainedon ImageNet-1K dataset demonstrate the effectiveness of our proposed approach.
Temporal Dynamics in Visual Data: Analyzing the Impact of Time on Classification Accuracy
Authors: Tom Pégeot; Eva Feillet; Adrian Popescu; Inna Kucher; Bertrand Delezoide
Venue: WACV 2025 (main)

Abstract: Visual datasets are generally constructed from the samples available at the time of their collection and are not further updated. However these static datasets do not reflect the distribution changes that occur in real data. We analyze how different collection times lead to a shift in class distribution by collecting a set of Flickr images published over 14 years. The proposed "Visual Classes through Time" (VCT-107) dataset contains images tagged by their publication date and includes 107 classes covering various topics (human-made objects animals plants food etc.). Images from each class are divided into five collection periods to study the impact of time on classification accuracy. When training different classification models using linear probing we observe an accuracy loss when training on data from one period and testing on other periods. This happens even in the case of a strongly pre-trained model like DinoV2 ViT-B/14. Intuitively the performance loss is generally more significant when the collection periods between the training and test data are further apart. Our analysis reveals that the temporal shift varies between classes with the largest shifts observed for human-made objects and the smallest for natural concepts such as animal species. Our results stress the importance of regularly updating models to adapt to time-induced changes in the distribution of visual classes even when using a strongly pre-trained model. We release the VCT-107 dataset to facilitate research on temporal shifts.
Bidirectional Multi-Step Domain Generalization for Visible-Infrared Person Re-Identification
Authors: Mahdi Alehdaghi; Pourya Shamsolmoali; Rafael M. O. Cruz; Eric Granger
Venue: WACV 2025 (main)

Abstract: A key challenge in visible-infrared person re-identification (V-I ReID) is training a backbone model capable of effectively addressing the significant discrepancies across modalities. State-of-the-art methods that generate a single intermediate bridging domain are often less effective as this generated domain may not adequately capture sufficient common discriminant information. This paper introduces Bidirectional Multi-step Domain Generalization (BMDG) a novel approach for unifying feature representations across diverse modalities. BMDG creates multiple virtual intermediate domains by learning and aligning body part features extracted from both I and V modalities. In particular our method aims to minimize the cross-modal gap in two steps. First BMDG aligns modalities in the feature space by learning shared and modality-invariant body part prototypes from V and I images. Then it generalizes the feature representation by applying bidirectional multi-step learning which progressively refines feature representations in each step and incorporates more prototypes from both modalities. Based on these prototypes multiple bridging steps enhance the feature representation. Experiments conducted on V-I ReID datasets indicate that our BMDG approach can outperform state-of-the-art part-based and intermediate generation methods and can be integrated into other part-based methods to enhance their V-I ReID performance.
Pre-Trained Multiple Latent Variable Generative Models are Good Defenders Against Adversarial Attacks
Authors: Dario Serez; Marco Cristani; Alessio Del Bue; Vittorio Murino; Pietro Morerio
Venue: WACV 2025 (main)

Abstract: Attackers can deliberately perturb classifiers' input with subtle noise altering final predictions. Among proposed countermeasures adversarial purification employs generative networks to preprocess input images filtering out adversarial noise. In this study we propose specific generators defined Multiple Latent Variable Generative Models (MLVGMs) for adversarial purification. These models possess multiple latent variables that naturally disentangle coarse from fine features. Taking advantage of these properties we autoencode images to maintain class-relevant information while discarding and re-sampling any detail including adversarial noise. The procedure is completely training-free exploring the generalization abilities of pre-trained MLVGMs on the adversarial purification downstream task. Despite the lack of large models trained on billions of samples we show that smaller MLVGMs are already competitive with traditional methods and can be used as foundation models. Official code released at https://github.com/SerezD/gen_adversarial.