GenAI4ELab/papercli-papers huggingface dataset:
CLIPArTT: Adaptation of CLIP to New Domains at Test Time
Authors: Gustavo A Vargas Hakim; David Osowiechi; Mehrdad Noori; Milad Cheraghalikhani; Ali Bahri; Moslem Yazdanpanah; Ismail Ben Ayed; Christian Desrosiers
Venue: WACV 2025 (main)

Abstract: Pre-trained vision-language models (VLMs) exemplified by CLIP demonstrate remarkable adaptability across zero-shot classification tasks without additional training. However their performance diminishes in the presence of domain shifts. In this study we introduce CLIP Adaptation duRing Test-Time (CLIPArTT) a fully test-time adaptation (TTA) approach for CLIP which involves automatic text prompts construction during inference for their use as text supervision. Our method employs a unique minimally invasive text prompt tuning process wherein multiple predicted classes are aggregated into a single new text prompt used as pseudo label to re-classify inputs in a transductive manner. Additionally we pioneer the standardization of TTA benchmarks (e.g. TENT) in the realm of VLMs. Our findings demonstrate that without requiring additional transformations nor new trainable modules CLIPArTT enhances performance dynamically across non-corrupted datasets such as CIFAR-100 corrupted datasets like CIFAR-100-C and ImageNet-C alongside synthetic datasets such as VisDA-C. This research underscores the potential for improving VLMs' adaptability through novel test-time strategies offering insights for robust performance across varied datasets and environments. The code can be found at: https://github.com/dosowiechi/CLIPArTT.git
Crafting Distribution Shifts for Validation and Training in Single Source Domain Generalization
Authors: Nikos Efthymiadis; Giorgos Tolias; Ondřej Chum
Venue: WACV 2025 (main)

Abstract: Single-source domain generalization attempts to learn a model on a source domain and deploy it to unseen target domains. Limiting access only to source domain data imposes two key challenges - how to train a model that can generalize and how to verify that it does. The standard practice of validation on the training distribution does not accurately reflect the model's generalization ability while validation on the test distribution is a malpractice to avoid. In this work we construct an independent validation set by transforming source domain images with a comprehensive list of augmentations covering a broad spectrum of potential distribution shifts in target domains. We demonstrate a high correlation between validation and test performance for multiple methods and across various datasets. The proposed validation achieves a relative accuracy improvement over the standard validation equal to 15.4% or 1.6% when used for method selection or learning rate tuning respectively. Furthermore we introduce a novel family of methods that increase the shape bias through enhanced edge maps. To benefit from the augmentations during training and preserve the independence of the validation set a k-fold validation process is designed to separate the augmentation types used in training and validation. The method that achieves the best performance on the augmented validation is selected from the proposed family. It achieves state-of-the-art performance on various standard benchmarks. Code at: https://github.com/NikosEfth/crafting-shifts
High-Fidelity Document Stain Removal via A Large-Scale Real-World Dataset and A Memory-Augmented Transformer
Authors: Mingxian Li; Hao Sun; Yingtie Lei; Xiaofeng Zhang; Yihang Dong; Yilin Zhou; Zimeng Li; Xuhang Chen
Venue: WACV 2025 (main)

Abstract: Document images are often degraded by various stains significantly impacting their readability and hindering downstream applications such as document digitization and analysis. The absence of a comprehensive stained document dataset has limited the effectiveness of existing document enhancement methods in removing stains while preserving fine-grained details. To address this challenge we construct StainDoc the first large-scale high-resolution (2145x2245) dataset specifically designed for document stain removal. StainDoc comprises over 5000 pairs of stained and clean document images across multiple scenes. This dataset encompasses a diverse range of stain types severities and document backgrounds facilitating robust training and evaluation of document stain removal algorithms. Furthermore we propose StainRestorer a Transformer-based document stain removal approach. StainRestorer employs a memory-augmented Transformer architecture that captures hierarchical stain representations at part instance and semantic levels via the DocMemory module. The Stain Removal Transformer (SRTransformer) leverages these feature representations through a dual attention mechanism: an enhanced spatial attention with an expanded receptive field and a channel attention captures channel-wise feature importance. This combination enables precise stain removal while preserving document content integrity. Extensive experiments demonstrate StainRestorer's superior performance over state-of-the-art methods on the StainDoc dataset and its variants StainDoc_Mark and StainDoc_Seal establishing a new benchmark for document stain removal. Our work highlights the potential of memory-augmented Transformers for this task and contributes a valuable dataset to advance future research.
Distillation of Diffusion Features for Semantic Correspondence
Authors: Frank Fundel; Johannes Schusterbauer; Vincent Tao Hu; Björn Ommer
Venue: WACV 2025 (main)

Abstract: Semantic correspondence the task of determining relationships between different parts of images underpins various applications including 3D reconstruction image-to-image translation object tracking and visual place recognition. Recent studies have begun to explore representations learned in large generative image models for semantic correspondence demonstrating promising results. Building on this progress current state-of-the-art methods rely on combining multiple large models resulting in high computational demands and reduced efficiency. In this work we address this challenge by proposing a more computationally efficient approach. We propose a novel knowledge distillation technique to overcome the problem of reduced efficiency. We show how to use two large vision foundation models and distill the capabilities of these complementary models into one smaller model that maintains high accuracy at reduced computational cost. Furthermore we demonstrate that by incorporating 3D data we are able to further improve performance without the need for human-annotated correspondences. Overall our empirical results demonstrate that our distilled model with 3D data augmentation achieves performance superior to current state-of-the-art methods while significantly reducing computational load and enhancing practicality for real-world applications such as semantic video correspondence. Our code and weights are publicly available on our project page.
Clarity Amidst Blur: A Deterministic Method for Synthetic Generation of Water Droplets on Camera Lenses
Authors: Tim Dieter Eberhardt; Tim Brühl; Robin Schwager; Tin Stribor Sohn; Wilhelm Stork
Venue: WACV 2025 (main)

Abstract: In computer vision image clarity is crucial particularly when challenges like water droplets on camera lenses can significantly impair the accurate analysis of visual data. While existing methods mainly focus on small droplets the impact of larger droplets has been largely overlooked. This paper introduces a novel approach that models water droplets using randomly generated points and Bezier curves to simulate their shape on the lens. Based on this geometric framework we developed a classifier to distinguish between two visual scenarios within the droplet. For larger droplets we use a heuristic method to simulate various lens blockages. We evaluate this simulation framework using a real stereo dataset from [19] with clear and soiled images. Our method evaluated using Mean Squared Error (MSE) and Structural Similarity Index Measure (SSIM) demonstrates superior performance in MSE while also achieving competitive results in SSIM compared to existing techniques for generating realistic water droplets. Additionally we applied this technique for data augmentation in object detection tasks using the YOLOv7 [25] model. The results show improved robustness especially in challenging conditions where large droplets obstruct the lens.
A Pipeline and NIR-Enhanced Dataset for Parking Lot Segmentation
Authors: Shirin Qiam; Saipraneeth Devunuri; Lewis J. Lehe
Venue: WACV 2025 (main)

Abstract: Discussions of minimum parking requirement policies often include maps of parking lots which are time-consuming to construct manually. Open-source datasets for such parking lots are scarce particularly for US cities. This paper introduces the idea of using Near-Infrared (NIR) channels as input and several post-processing techniques to improve the prediction of off-street surface parking lots using satellite imagery. We constructed two datasets with 12617 image-mask pairs each: one with 3-channel (RGB) and another with 4-channel (RGB + NIR). The datasets were used to train five deep learning models (OneFormer Mask2Former SegFormer DeepLabV3 and FCN) for semantic segmentation classifying images to differentiate between parking and non-parking pixels. Our results demonstrate that the NIR channel improved accuracy because parking lots are often surrounded by grass--even though the NIR channel needed to be upsampled from a lower resolution. Post-processing including eliminating erroneous "holes" simplifying edges and removing road and building footprints further improved the accuracy. Best model OneFormer trained on 4-channel input and paired with post-processing techniques achieves a mean Intersection over Union (mIoU) of 84.9% and a pixel-wise accuracy of 96.3%.
A Realistic Protocol for Evaluation of Weakly Supervised Object Localization
Authors: Shakeeb Murtaza; Soufiane Belharbi; Marco Pedersoli; Eric Granger
Venue: WACV 2025 (main)

Abstract: Weakly Supervised Object Localization (WSOL) allows training deep learning models for classification and localization (LOC) using only global class-level labels. The absence of bounding box (bbox) supervision during training raises challenges in the literature for hyper-parameter tuning model selection and evaluation. WSOL methods rely on a validation set with bbox annotations for model selection and a test set with bbox annotations for threshold estimation for producing bboxes from localization maps. This approach however is not aligned with the WSOL setting as these annotations are typically unavailable in real-world scenarios. Our initial empirical analysis shows a significant decline in LOC performance when model selection and threshold estimation rely solely on class labels and the image itself respectively compared to using manual bbox annotations. This highlights the importance of incorporating bbox labels for optimal model performance. In this paper a new WSOL evaluation protocol is proposed that provides LOC information without the need for manual bbox annotations. In particular we generated noisy pseudo-boxes from a pretrained off-the-shelf region proposal method such as Selective Search CLIP and RPN for model selection. These bboxes are also employed to estimate the threshold from LOC maps circumventing the need for test-set bbox annotations. Our experiments with several WSOL methods on challenging natural and medical image datasets show that using the proposed pseudo-bboxes for validation facilitates the model selection and threshold estimation with LOC performance comparable to models selected using GT bboxes on the validation set and threshold estimation on the test set. It also outperforms models selected using class-level labels and then dynamically thresholded with only LOC maps.
AlignIT: Enhancing Prompt Alignment in Customization of Text-to-Image Models
Authors: Aishwarya Agarwal; Srikrishna Karanam; Balaji Vasan Srinivasan
Venue: WACV 2025 (main)

Abstract: We consider the problem of customizing text-to-image diffusion models with user-supplied reference images. Given new prompts the existing methods can capture the key concept from the reference images but fail to align the generated image with the prompt. In this work we seek to address this key issue by proposing new methods that can easily be used in conjunction with existing customization methods that optimize the embeddings and weights at various intermediate stages of the text encoding process before being fed into the noise prediction model of a text-to-image diffusion model. The first contribution of this paper is a dissection of the various stages of the text encoding process leading up to the conditioning vector for text-to-image models. We take a holistic view of existing customization methods and notice that key and value outputs from this process differs substantially from their corresponding baseline (non-customized) models. While this difference does not impact the concept being customized it leads to other parts of the generated image not being aligned with the prompt. Further we also observe that these keys and values allow independent control various aspects of the final generation enabling semantic manipulation of the output. Taken together the features spanning these keys and values serve as the basis for our next contribution where we fix the aforementioned issues with existing methods. We propose a new post-processing algorithm AlignIT that infuses the keys and values for the concept of interest while ensuring the keys and values for all other tokens in the input prompt are unchanged. Our proposed method can be plugged in directly to existing customization methods leading to a substantial performance improvement in the alignment of the final result with the input prompt while retaining the customization quality. We conduct extensive experiments across various different customization methods and a wide variety of reference images and show consistent improvements both qualitatively and quantitatively.
DiffuseKronA: A Parameter Efficient Fine-Tuning Method for Personalized Diffusion Models
Authors: Shyam Marjit; Harshit Singh; Nityanand Mathur; Sayak Paul; Chia-Mu Yu; Pin-Yu Chen
Venue: WACV 2025 (main)

Abstract: In the realm of subject-driven text-to-image (T2I) generative models recent developments like DreamBooth and BLIP-Diffusion have led to impressive results yet encounter limitations due to their intensive fine-tuning demands and substantial parameter requirements. While the low-rank adaptation (LoRA) module within DreamBooth offers a reduction in trainable parameters it introduces a pronounced sensitivity to hyperparameters leading to a compromise between parameter efficiency and the quality of T2I personalized image synthesis. Addressing these constraints we introduce DiffuseKronA a novel Kronecker product-based adaptation module that not only significantly reduces the parameter count by 35% and 99.947% compared to LoRA-DreamBooth and the original DreamBooth respectively but also enhances the quality of image synthesis. Crucially DiffuseKronA mitigates the issue of hyperparameter sensitivity delivering consistent high-quality generations across a wide range of hyperparameters thereby diminishing the necessity for extensive fine-tuning. Furthermore a more controllable decomposition makes DiffuseKronA more interpretable and can even achieve up to a 50% reduction with results comparable to LoRA-Dreambooth. Evaluated against diverse and complex input images and text prompts DiffuseKronA consistently outperforms existing low-rank models producing diverse images of higher quality with improved fidelity and a more accurate color distribution of objects all the while upholding exceptional parameter efficiency thus presenting a substantial advancement in the field of T2I generative modeling.
Effective Backdoor Learning on Open-Set Face Recognition Systems
Authors: Diana Voth; Leonidas Dane; Jonas Grebe; Sebastian Peitz; Philipp Terhörst
Venue: WACV 2025 (main)

Abstract: Backdoor attacks pose a serious threat to the security of face recognition systems. These involve the insertion of poisoned inputs into the training data to manipulate the model's behavior at inference time and can cause severe consequences such as unauthorized access to secure systems or impersonation of legitimate users. Previous works on backdoor attacks have primarily focused on closed-set classification systems. However open-set face recognition systems are commonly utilized in practical applications which operate fundamentally differently from closedset systems. In this paper we propose two main contributions. First we demonstrate that closed-set backdoor attacks are effective in basic classification scenarios but fail to perform well in complex open-set face recognition tasks. Second we introduce Feature Stabilized Trigger Loss (FSTL) a novel loss function designed to facilitate the learning of backdoors in open-set recognition models. The experiments were conducted on two large-scale datasets using a variety of high-performing face recognition systems and by training with both physical and digital triggers. Since developing effective attack countermeasures requires knowledge of effective attacks this work will enable future works on developing more secure recognition systems.
Uncertainty Aware Interest Point Detection and Description
Authors: Jingbo Zeng; Zaiwang Gu; Weide Liu; Lile Cai; Jun Cheng
Venue: WACV 2025 (main)

Abstract: Interest point detection and description play an important role in many visual tasks including image registration pose estimation 3D reconstruction and more. State-of-the-art interest point detection techniques are based on deep neural networks (NNs) which are prone to produce overconfident predictions. However calibrated and robust uncertainty measurement is crucial when deploying deep NN models in safety critical applications. In this work we propose a novel Uncertainty-Aware interest Point (UAPoint) detection method to address this problem. Our method leverages evidential learning to learn both aleatoric and epistemic uncertainty. We further propose a constrained sampling scheme to construct more efficient training pairs for the descriptor decoder. We evaluate our method on a wide range of benchmarks and show that our method achieves state-of-the-art performance. Code will be released upon publication. Code will be released in https://github.com/JingboZeng/ UAPoint.
Strategic Base Representation Learning via Feature Augmentations for Few-Shot Class Incremental Learning
Authors: Parinita Nema; Vinod K Kurmi
Venue: WACV 2025 (main)

Abstract: Few-shot class incremental learning implies the model to learn new classes while retaining knowledge of previously learned classes with a small number of training instances. Existing frameworks typically freeze the parameters of the previously learned classes during the incorporation of new classes. However this approach often results in suboptimal class separation of previously learned classes leading to overlap between old and new classes. Consequently the performance of old classes degrades on new classes. To address these challenges we propose a novel feature augmentation driven contrastive learning framework designed to enhance the separation of previously learned classes to accommodate new classes. Our approach involves augmenting feature vectors and assigning proxy labels to these vectors. This strategy expands the feature space ensuring seamless integration of new classes within the expanded space. Additionally we employ a self-supervised contrastive loss to improve the separation between previous classes. We validate our framework through experiments on three FSCIL benchmark datasets: CIFAR100 miniImageNet and CUB200. The results demonstrate that our Feature Augmentation driven Contrastive Learning framework significantly outperforms other approaches achieving state-of-the-art performance.
Adaptive Deviation Learning for Visual Anomaly Detection with Data Contamination
Authors: Anindya Sundar Das; Guansong Pang; Monowar Bhuyan
Venue: WACV 2025 (main)

Abstract: Visual anomaly detection targets to detect images that notably differ from normal pattern and it has found extensive application in identifying defective parts within the manufacturing industry. These anomaly detection paradigms predominantly focus on training detection models using only clean unlabeled normal samples assuming an absence of contamination; a condition often unmet in real-world scenarios. The performance of these methods significantly depends on the quality of the data and usually decreases when exposed to noise. We introduce a systematic adaptive method that employs deviation learning to compute anomaly scores end-to-end while addressing data contamination by assigning relative importance to the weights of individual instances. In this approach the anomaly scores for normal instances are designed to approximate scalar scores obtained from the known prior distribution. Meanwhile anomaly scores for anomaly examples are adjusted to exhibit statistically significant deviations from these reference scores. Our approach incorporates a constrained optimization problem within the deviation learning framework to update instance weights resolving this problem for each mini-batch. Comprehensive experiments on the MVTec and VisA benchmark datasets indicate that our proposed method surpasses competing techniques and exhibits both stability and robustness in the presence of data contamination.
Self-Relaxed Joint Training: Sample Selection for Severity Estimation with Ordinal Noisy Labels
Authors: Shumpei Takezaki; Kiyohito Tanaka; Seiichi Uchida
Venue: WACV 2025 (main)

Abstract: Severity level estimation is a crucial task in medical image diagnosis. However accurately assigning severity class labels to individual images is very costly and challenging. Consequently the attached labels tend to be noisy. In this paper we propose a new framework for training with "ordinal" noisy labels. Since severity levels have an ordinal relationship we can leverage this to train a classifier while mitigating the negative effects of noisy labels. Our framework uses two techniques: clean sample selection and dual-network architecture. A technical highlight of our approach is the use of soft labels derived from noisy hard labels. By appropriately using the soft and hard labels in the two techniques we achieve more accurate sample selection and robust network training. The proposed method outperforms various state-of-the-art methods in experiments using two endoscopic ulcerative colitis (UC) datasets and a retinal Diabetic Retinopathy (DR) dataset. Our codes are available at https://github.com/shumpei-takezaki/Self-Relaxed-Joint-Training.