GenAI4ELab/papercli-papers huggingface dataset:
CRAFT: Class Ranking Aware Fine-Tuning for Enhanced Out-of-Distribution Detection
Authors: Naveen Karunanayake; Suranga Seneviratne; Sanjay Chawla
Venue: WACV 2025 (main)

Abstract: Out-of-distribution (OOD) detection remains a key challenge preventing the rollout of key AI technologies like autonomous vehicles into the mainstream as classifiers trained on in-distribution (ID) data are unable to gracefully handle OOD data. While OOD detection remains an active area of research, current post-hoc methods often suffer from limited separability between ID and OOD, and outlier exposure-based methods lack generalisation to unseen outlier types. We present CRAFT, a fine-tuning approach for arming pre-trained classifiers against OOD inputs without requiring access to outliers. The key insight that underpins our approach is that during pre-training, classifiers implicitly learn a ranking across the ID classes that is not respected by OOD data. Therefore, a form of fine-tuning without outliers of a pre-trained classifier can sharpen the rank order of the classes, making them sensitive to the presence of OOD data. Furthermore, the fine-tuned model does not impact the ability of the classifier to correctly classify ID inputs to their respective classes. Experiments on CIFAR-10, CIFAR-100, and ImageNet-200 demonstrate that CRAFT outperforms 33 existing methods, particularly in the more challenging near-OOD detection, as well as in overall OOD detection consistency and ID classification accuracy.
Improving Faithfulness of Text-to-Image Diffusion Models through Inference Intervention
Authors: Danfeng Guo; Sanchit Agarwal; Yu-Hsiang Lin; Jiun-Yu Kao; Tagyoung Chung; Nanyun Peng; Mohit Bansal
Venue: WACV 2025 (main)

Abstract: Text-to-Image diffusion models have shown remarkable capabilities in generating high-quality images. However current models often struggle to adhere to the complete set of conditions specified in the input text and return unfaithful generations. Existing works address this problem by either fine-tuning the base model or modifying the latent representations during the inference stage with gradient-based updates. Not only are these approaches computationally expensive but also they usually only improve limited kinds of errors (e.g. the count of objects). In this work we propose an intervention-based mechanism to enhance the faithfulness of diffusion models by controlling the denoising process. Starting with layout-conditional diffusion models our approach first detects incorrectly-generated/missing objects during denoising steps. Next a layout is constructed from the erroneous objects (feedback). Finally we return to an earlier denoising step. The new layout is fed to the diffusion model to obtain its latent representation. Correction is applied by composing the new latents with the original ones and continuing the generation process thereby driving the generation away from erroneous directions. As additional feedback and correction strategy we also explore retrieval-augmented generation to help the model recover missing objects. We conduct experiments on VPEval and HRS-Bench datasets and measure faithfulness across four dimensions; presence of objects object counts scale of objects and spatial relations between objects. Compared to GLIGEN the state-of-the-art model on the VPEval dataset our approach significantly improves on all metrics (+6.7% average accuracy increase). On HRS-Bench dataset it also outperforms existing models in count and scale metrics.
Learning the Power of "No": Foundation Models with Negations
Authors: Jaisidh Singh; Ishaan Shrivastava; Mayank Vatsa; Richa Singh; Aparna Bharati
Venue: WACV 2025 (main)

Abstract: Negation is a fundamental aspect of natural language reasoning yet foundational vision-language models (VLMs) like CLIP face significant challenges in accurately interpreting it. These models often process text prompts holistically making it difficult to isolate and understand the role of negated terms. To overcome this limitation we present CC-Neg: a novel dataset consisting of 228246 images each paired with both true captions and their corresponding negated versions. CC-Neg provides a critical benchmark to assess and improve foundational VLMs' ability to process negations focusing specifically on how the presence of terms like 'not' alters the semantic relationship between images and their textual descriptions. To illustrate the effectiveness of the CC-Neg dataset in enhancing negation understanding we introduce the CoN-CLIP framework which incorporates targeted modifications to CLIP's contrastive loss function. When trained with CC-Neg CoN-CLIP achieves a 3.85% average improvement in top-1 accuracy for zero-shot image classification across eight datasets and a 4.4% performance boost on challenging compositionality benchmarks such as SugarCREPE. These results highlight CoN-CLIP's enhanced understanding of the nuanced semantic relationships involving negation. Our code and the CC-Neg benchmark are available at: https://github.com/jaisidhsingh/CoN-CLIP.
Temporally Streaming Audio-Visual Synchronization for Real-World Videos
Authors: Jordan G Voas; Wei-Cheng Tseng; Layne Berry; Xixi Hu; Puyuan Peng; James Stuedemann; David Harwath
Venue: WACV 2025 (main)

Abstract: We introduce RealSync a novel dataset designed to significantly enhance the training and evaluation of models for audio-visual synchronization (AV Sync) tasks. Sourced from high-quality YouTube channels RealSync covers a wide range of content domains providing an improved scale diversity and alignment with broadcast content compared to existing datasets. It features extended-length video samples catering to the critical need for more comprehensive real-world training and evaluation materials. Alongside this dataset we present StreamSync a model tailored for real-world AV Sync applications. StreamSync is designed to be backbone agnostic and incorporates a streaming mechanism that processes consecutive video segments dynamically iteratively refining synchronization predictions. This innovative approach enables StreamSync to outperform existing models offering superior synchronization accuracy with minimal computational cost per iteration. Together our dataset and the StreamSync model establish a new benchmark for AVSync research promising to drive the development of more robust and practical AVSync methods. https://github.com/jvoas655/StreamSync
SenCLIP: Enhancing Zero-Shot Land-Use Mapping for Sentinel-2 with Ground-Level Prompting
Authors: Pallavi Jain; Dino Ienco; Roberto Interdonato; Tristan Berchoux; Diego Marcos
Venue: WACV 2025 (main)

Abstract: Pre-trained vision-language models (VLMs) such as CLIP demonstrate impressive zero-shot classification capabilities with free-form prompts and even show some generalization in specialized domains. However their performance on satellite imagery is limited due to the under representation of such data in their training sets which predominantly consist of ground-level images. Existing prompting techniques for satellite imagery are often restricted to generic phrases like "a satellite image of..." limiting their effectiveness for zero-shot land-use/land-cover (LULC) mapping. To address these challenges we introduce SenCLIP which transfers CLIP's representation to Sentinel-2 imagery by leveraging a large dataset of Sentinel-2 images paired with geotagged ground-level photos from across Europe. We evaluate SenCLIP alongside other state-of-the-art remote sensing VLMs on zero-shot LULC mapping tasks using the EuroSAT and BigEarthNet datasets with both aerial and ground-level prompting styles. Our approach which aligns ground-level representations with satellite imagery demonstrates significant improvements in classification accuracy across both prompt styles opening new possibilities for applying free-form textual descriptions in zero-shot LULC mapping. Code dataset and pretrained models are available at https://github.com/pallavijain-pj/SenCLIP
ENAF: A Multi-Exit Network with an Adaptive Patch Fusion for Large Image Super Resolution
Authors: Manh Duong Nguyen; Tuan Nghia Nguyen; Xuan Truong Nguyen
Venue: WACV 2025 (main)

Abstract: To accelerate single image super-resolution (SISR) networks on large images (2K-8K) many recent approaches decompose an image into small patches and dynamically determine an execution path according to its difficulty (referred to as a dynamic network). To quantify the hardness of a patch they mainly rely on a handcrafted assessment score e.g. edge which weakly associates a patch's texture with the computational complexity of a SISR model. To address the problem we introduce ENAF - a dynamic network for SISR with an adaptive patch fusion. Built on top of a backbone ENAF incorporates multiple early exits (EEs) to tackle the over-parameterized SISR model. More importantly ENAF plugs a tiny network that estimates PSNR to associate data texture with a computation cost at an EE. Based on the scores ENAF effectively assigns image patches to an exit enhancing the quality-complexity trade-off. Extensive experiments on common datasets with popular SISR backbones demonstrate the effectiveness of ENAF in various settings. The source code will be available.
DiffQRCoder: Diffusion-Based Aesthetic QR Code Generation with Scanning Robustness Guided Iterative Refinement
Authors: Jia-Wei Liao; Winston Wang; Tzu-Sian Wang; Li-Xuan Peng; Ju-Hsuan Weng; Cheng-Fu Chou; Jun-Cheng Chen
Venue: WACV 2025 (main)

Abstract: With the success of Diffusion Models for image generation the technologies also have revolutionized the aesthetic Quick Response (QR) code generation. Despite significant improvements in visual attractiveness for the beautified codes their scannabilities are usually sacrificed and thus hinder their practical uses in real-world scenarios. To address this issue we propose a novel training-free Diffusion-based QR Code generator (DiffQRCoder) to effectively craft both scannable and visually pleasing QR codes. The proposed approach introduces Scanning-Robust Perceptual Guidance (SRPG) a new diffusion guidance for Diffusion Models to guarantee the generated aesthetic codes to obey the ground-truth QR codes while maintaining their attractiveness during the denoising process. Additionally we present another post-processing technique Scanning Robust Manifold Projected Gradient Descent (SR-MPGD) to further enhance their scanning robustness through iterative latent space optimization. With extensive experiments the results demonstrate that our approach not only outperforms other compared methods in Scanning Success Rate (SSR) with better or comparable CLIP aesthetic score (CLIP-aes.) but also significantly improves the SSR of the ControlNet-only approach from 60% to 99%. The subjective evaluation indicates that our approach achieves promising visual attractiveness to users as well. Finally even with different scanning angles and the most rigorous error tolerance settings our approach robustly achieves over 95% SSR demonstrating its capability for real-world applications. Our project page is available at https://jwliao1209.github.io/DiffQRCoder.
SeCo-INR: Semantically Conditioned Implicit Neural Representations for Improved Medical Image Super-Resolution
Authors: Mevan Ekanayake; Zhifeng Chen; Gary Egan; Mehrtash Harandi; Zhaolin Chen
Venue: WACV 2025 (main)

Abstract: Implicit Neural Representations (INRs) have recently advanced the field of deep learning due to their ability to learn continuous representations of signals without the need for large training datasets. Although INR methods have been studied for medical image super-resolution their adaptability to localized priors in medical images has not been extensively explored. Medical images contain rich anatomical divisions that could provide valuable local prior information to enhance the accuracy and robustness of INRs. In this work we propose a novel framework referred to as the Semantically Conditioned INR (SeCo-INR) that conditions an INR using local priors from a medical image enabling accurate model fitting and interpolation capabilities to achieve super-resolution. Our framework learns a continuous representation of the semantic segmentation features of a medical image and utilizes it to derive the optimal INR for each semantic region of the image. We tested our framework using several medical imaging modalities and achieved higher quantitative scores and more realistic super-resolution outputs compared to state-of-the-art methods.
Disentangle Source and Target Knowledge for Continual Test-Time Adaptation
Authors: Tianyi Ma; Maoying Qiao
Venue: WACV 2025 (main)

Abstract: Continual Test-Time Adaptation (CoTTA) task is proposed to tackle the challenges of constant domain shifts during testing. The goals are twofold: 1) to preserve the knowledge from the source domain without source data and 2) to effectively extract target knowledge using unlabeled target domain data. Existing works primarily focus on either source or target knowledge attempting to learn both in a mixed manner. We argue that this may harm the source knowledge preservation and target knowledge extraction. To this end this paper proposes a Source and Target knowledge Disentangle Transformer (SoTa-DiT) with the prompting mechanism. Specifically in a vision transformer (ViT) we incorporate source and target prompts supervised by two groups of deliberately designed loss functions to learn source and target knowledge separately. The source prompt focuses on anti-source-forgetting by extracting and preserving knowledge from the source model while the target prompt focuses on pro-target-extracting using target data contrastive learning. With comprehensive evaluations across various datasets using different ViT backbones we demonstrate that this dual-prompt architecture of SoTa-DiT is effective and that disentangling knowledge with the prompts benefits CoTTA. As a result SoTa-DiT significantly improves image classification accuracy under the CoTTA setting.
LiLMaps: Learnable Implicit Language Maps
Authors: Evgenii Kruzhkov; Sven Behnke
Venue: WACV 2025 (main)

Abstract: One of the current trends in robotics is to employ large language models (LLMs) to provide non-predefined commands execution and natural human-robot interaction. It is useful to have an environment map together with its language representation which can be further utilized by LLMs. Such a comprehensive scene representation enables numerous ways of interaction with the map for autonomously operating robots. In this work we present an approach that enhances incremental implicit mapping through the integration of visual-language features. Specifically we (i) propose a decoder optimization technique for implicit language maps which can be used when new objects appear on the scene and (ii) address the problem of inconsistent visual-language predictions between different viewing positions. Our experiments demonstrate the effectiveness of LiLMaps and solid improvements in performance.
Boosting Diffusion Guidance via Learning Degradation-Aware Models for Blind Super Resolution
Authors: Shao-Hao Lu; Ren Wang; Ching-Chun Huang; Wei-Chen Chiu
Venue: WACV 2025 (main)

Abstract: Recently diffusion-based blind super-resolution (SR) methods have shown great ability to generate high-resolution images with abundant high-frequency detail but the detail is often achieved at the expense of fidelity. Meanwhile another line of research focusing on rectifying the reverse process of diffusion models (i.e. diffusion guidance) has demonstrated the power to generate high-fidelity results for non-blind SR. However these methods rely on known degradation kernels making them difficult to apply to blind SR. To address these issues we introduce degradation-aware models that can be integrated into the diffusion guidance framework eliminating the need to know degradation kernels. Additionally we propose two novel techniques--input perturbation and guidance scalar--to further improve our performance. Extensive experimental results show that our proposed method has superior performance over state-of-the-art methods on blind SR benchmarks.
AdaPrefix++: Integrating Adapters Prefixes and Hypernetwork for Continual Learning
Authors: Sayanta Adhikari; Dupati Srikar Chandra; P. K. Srijith; Pankaj Wasnik; Naoyuki Oneo
Venue: WACV 2025 (main)

Abstract: Continual learning allows systems to continuously learn and adapt to the tasks in an evolving real-world environment without forgetting previous tasks. Developing deep learning models that can continually learn over a sequence of tasks is challenging. We propose a novel method AdaPrefix which addresses this and empowers continual learning capability in pretrained large models (PLMs). AdaPrefix provide a continual learning method for transformer-based deep learning models by appropriately integrating the parameter-efficient methods adapters and prefixes. AdaPrefix is an effective approach for smaller PLMs and achieves better results than state-of-the-art approaches. We further improve upon AdaPrefix by proposing AdaPrefix++ enabling knowledge transfer across the tasks. It leverages hypernetworks to generate prefixes and continually learns the hypernetwork parameters to facilitate knowledge transfer. AdaPrefix++ has a smaller parameter growth compared to AdaPrefix and is more effective and valuable for continual learning in PLMs. We performed several experiments on various benchmark datasets to demonstrate the performance of our approach for different PLMs and continual learning scenarios.
Enhancing Novel Object Detection via Cooperative Foundational Models
Authors: Rohit Bharadwaj; Muzammal Naseer; Salman Khan; Fahad Shahbaz Khan
Venue: WACV 2025 (main)

Abstract: In this work we address the challenging and emergent problem of novel object detection (NOD) focusing on the accurate detection of both known and novel object categories during inference. Traditional object detection algorithms are inherently closed-set limiting their capability to handle NOD. We present a novel approach to transform existing closed-set detectors into open-set detectors. This transformation is achieved by leveraging the complementary strengths of pre-trained foundational models specifically CLIP and SAM through our cooperative mechanism. Furthermore by integrating this mechanism with state-of-the-art open-set detectors such as GDINO we establish new benchmarks in object detection performance. Our method achieves 17.42 mAP in novel object detection and 42.08 mAP for known objects on the challenging LVIS dataset. Adapting our approach to the COCO OVD split we obtain an impressive result of 49.6 Novel AP50 which outperforms existing SOTA methods with similar backbone. Our code is available at: https://rohit901.github.io/coop-foundation-models/
Feature-Level and Spatial-Level Activation Expansion for Weakly-Supervised Semantic Segmentation
Authors: Junsu Choi; Jin-Seop Lee; Noo-ri Kim; SuHyun Yoon; Jee-Hyong Lee
Venue: WACV 2025 (main)

Abstract: Weakly-supervised Semantic Segmentation (WSSS) aims to provide a precise semantic segmentation results without expensive pixel-wise segmentation labels. With the supervision gap between classification and segmentation Image-level WSSS mainly relies on Class Activation Maps (CAMs) from the classification model to emulate the pixel-wise annotations. However CAMs often fail to cover the entire object region because classification models tend to focus on narrow discriminative regions in an object. Towards accurate CAM coverage Existing WSSS methods have tried to boost feature representation learning or impose consistency regularization to the classification models but still there are limitation in activating non-discriminative area where the focus of the models is weak. To tackle this issue we propose FSAE framework which provides explicit supervision of non-discriminative area encouraging the CAMs to activate on various object features. We leverage weak-strong consistency with pseudo-label expansion strategy for reliable supervision and enhance learning of non-discriminative object boundaries. Specifically we use strong perturbation to make challenging inference target and focus on generating reliable pixel-wise supervision signal for broad object regions. Extensive experiments on the WSSS benchmark datasets show that our method boosts initial seed quality and segmentation performance by large margin achieving new state-of-the-art performance on benchmark WSSS datasets. Our public code is available at https://github.com/obeychoi0120/FSAE.
SpaGBOL: Spatial-Graph-Based Orientated Localisation
Authors: Tavis Shore; Oscar Mendez; Simon Hadfield
Venue: WACV 2025 (main)

Abstract: Cross-View Geo-Localisation within urban regions is challenging in part due to the lack of geo-spatial structuring within current datasets and techniques. We propose utilising graph representations to model sequences of local observations and the connectivity of the target location. Modelling as a graph enables generating previously unseen sequences by sampling with new parameter configurations. To leverage this newly available information we propose a GNN-based architecture producing spatially strong embeddings and improving discriminability over isolated image embeddings. We outline SpaGBOL introducing three novel contributions. 1) The first graph-structured dataset for Cross-View Geo-Localisation containing multiple streetview images per node to improve generalisation. 2) Introducing GNNs to the problem we develop the first system that exploits the correlation between node proximity and feature similarity. 3) Leveraging the unique properties of the graph representation - we demonstrate a novel retrieval filtering approach based on neighbourhood bearings. SpaGBOL achieves state-of-the-art accuracies on the unseen test graph - with relative Top-1 retrieval improvements on previous techniques of 11% and 50% when filtering with Bearing Vector Matching on the SpaGBOL dataset. Code and dataset available: https://www.github.com/tavisshore/SpaGBOL.
SADA: Semantic Adversarial Unsupervised Domain Adaptation for Temporal Action Localization
Authors: David Pujol-Perich; Albert Clapés; Sergio Escalera
Venue: WACV 2025 (main)

Abstract: Temporal Action Localization (TAL) is a complex task that poses relevant challenges particularly when attempting to generalize on new - unseen - domains in real-world applications. These scenarios despite realistic are often neglected in the literature exposing these solutions to important performance degradation. In this work we tackle this issue by introducing for the first time an approach for Unsupervised Domain Adaptation (UDA) in sparse TAL which we refer to as Semantic Adversarial unsupervised Domain Adaptation (SADA). Our contributions are threefold: (1) we pioneer the development of a domain adaptation model that operates on realistic sparse action detection benchmarks; (2) we tackle the limitations of global-distribution alignment techniques by introducing a novel adversarial loss that is sensitive to local class distributions ensuring finer-grained adaptation; and (3) we present a novel set of benchmarks based on EpicKitchens100 and CharadesEgo that evaluate multiple domain shifts in a comprehensive manner. Our experiments indicate that SADA improves the adaptation across domains when compared to fully supervised state-of-the-art and alternative UDA methods attaining a performance boost of up to 6.14% mAP. The code is publicly available at https://github.com/davidpujol/SADA.