GenAI4ELab/papercli-papers huggingface dataset:
Feed-Forward Latent Domain Adaptation
Authors: Ondrej Bohdal; Da Li; Shell Xu Hu; Timothy Hospedales
Venue: WACV 2024 (main)

Abstract: We study a new highly-practical problem setting that enables resource-constrained edge devices to adapt a pre-trained model to their local data distributions. Recognizing that device's data are likely to come from multiple latent domains that include a mixture of unlabelled domain-relevant and domain-irrelevant examples, we focus on the comparatively under-studied problem of latent domain adaptation. Considering limitations of edge devices, we aim to only use a pre-trained model and adapt it in a feed-forward way, without using back-propagation and without access to the source data. Modelling these realistic constraints bring us to the novel and practically important problem setting of feed-forward latent domain adaptation. Our solution is to meta-learn a network capable of embedding the mixed-relevance target dataset and dynamically adapting inference for target examples using cross-attention. The resulting framework leads to consistent improvements over strong ERM baselines. We also show that our framework sometimes even improves on the upper bound of domain-supervised adaptation, where only domain-relevant instances are provided for adaptation. This suggests that human annotated domain labels may not always be optimal, and raises the possibility of doing better through automated instance selection.
Triplet Attention Transformer for Spatiotemporal Predictive Learning
Authors: Xuesong Nie; Xi Chen; Haoyuan Jin; Zhihang Zhu; Yunfeng Yan; Donglian Qi
Venue: WACV 2024 (main)

Abstract: Spatiotemporal predictive learning offers a self-supervised learning paradigm that enables models to learn both spatial and temporal patterns by predicting future sequences based on historical sequences. Mainstream methods are dominated by recurrent units, yet they are limited by their lack of parallelization and often underperform in real-world scenarios. To improve prediction quality while maintaining computational efficiency, we propose an innovative triplet attention transformer designed to capture both inter-frame dynamics and intra-frame static features. Specifically, the model incorporates the Triplet Attention Module (TAM), which replaces traditional recurrent units by exploring self-attention mechanisms in temporal, spatial, and channel dimensions. In this configuration: (i) temporal tokens contain abstract representations of inter-frame, facilitating the capture of inherent temporal dependencies; (ii) spatial and channel attention combine to refine the intra-frame representation by performing fine-grained interactions across spatial and channel dimensions. Alternating temporal, spatial, and channel-level attention allows our approach to learn more complex short- and long-range spatiotemporal dependencies. Extensive experiments demonstrate performance surpassing existing recurrent-based and recurrent-free methods, achieving state-of-the-art under multi-scenario examination including moving object trajectory prediction, traffic flow prediction, driving scene prediction, and human motion capture.
Arbitrary-Resolution and Arbitrary-Scale Face Super-Resolution With Implicit Representation Networks
Authors: Yi Ting Tsai; Yu Wei Chen; Hong-Han Shuai; Ching-Chun Huang
Venue: WACV 2024 (main)

Abstract: Face super-resolution (FSR) is a critical technique for enhancing low-resolution facial images and has significant implications for face-related tasks. However, existing FSR methods are limited by fixed up-sampling scales and sensitivity to input size variations. To address these limitations, this paper introduces an Arbitrary-Resolution and Arbitrary-Scale FSR method with implicit representation networks (ARASFSR), featuring three novel designs. First, ARASFSR employs 2D deep features, local relative coordinates, and up-sampling scale ratios to predict RGB values for each target pixel, allowing super-resolution at any up-sampling scale. Second, a local frequency estimation module captures high-frequency facial texture information to reduce the spectral bias effect. Lastly, a global coordinate modulation module guides FSR to leverage prior knowledge of facial structure effectively and achieve resolution adaptation. Quantitative and qualitative evaluations demonstrate the robustness of ARASFSR over existing state-of-the-art methods while super-resolving facial images across various input sizes and up-sampling scales.
FPGAN-Control: A Controllable Fingerprint Generator for Training With Synthetic Data
Authors: Alon Shoshan; Nadav Bhonker; Emanuel Ben Baruch; Ori Nizan; Igor Kviatkovsky; Joshua Engelsma; Manoj Aggarwal; Gérard Medioni
Venue: WACV 2024 (main)

Abstract: Training fingerprint recognition models using synthetic data has recently gained increased attention in the biometric community as it alleviates the dependency on sensitive personal data. Existing approaches for fingerprint generation are limited in their ability to generate diverse impressions of the same finger, a key property for providing effective data for training recognition models. To address this gap, we present FPGAN-Control, an identity preserving image generation framework which enables control over the fingerprint's image appearance (e.g., fingerprint type, acquisition device, pressure level) of generated fingerprints. We introduce a novel appearance loss that encourages disentanglement between the fingerprint's identity and appearance properties. In our experiments, we used the publicly available NIST SD302 (N2N) dataset for training the FPGAN-Control model. We demonstrate the merits of FPGAN-Control, both quantitatively and qualitatively, in terms of identity preservation level, degree of appearance control, and low synthetic-to-real domain gap. Finally, training recognition models using only synthetic datasets generated by FPGAN-Control lead to recognition accuracies that are on par or even surpass models trained using real data. To the best of our knowledge, this is the first work to demonstrate this.
Continual Learning of Unsupervised Monocular Depth From Videos
Authors: Hemang Chawla; Arnav Varma; Elahe Arani; Bahram Zonooz
Venue: WACV 2024 (main)

Abstract: Spatial scene understanding, including monocular depth estimation, is an important problem with various applications such as robotics and autonomous driving. While improvements in unsupervised monocular depth estimation have potentially allowed models to be trained on diverse crowdsourced videos, this remains under-explored as most methods utilize the standard training protocol wherein the models are trained from scratch on all data after new data is collected. Instead, continual training of models on sequentially collected data would significantly reduce computational and memory costs. Nevertheless, naive continual training leads to catastrophic forgetting, where the model performance deteriorates on older domains as it learns on newer domains, highlighting the trade-off between model stability and plasticity. While several techniques have been proposed to address this issue in image classification, the high-dimensional and spatiotemporally correlated outputs of depth estimation make it a distinct challenge. To the best of our knowledge, no framework or method currently exists focusing on the problem of continual learning in depth estimation. Thus, we introduce a framework that captures the challenges of continual unsupervised depth estimation (CUDE), and define the necessary metrics for evaluating model performance. We propose a rehearsal-based dual-memory method MonoDepthCL, which utilizes spatiotemporal consistency for continual learning in depth estimation, even when the camera intrinsics are unknown.
CXR-IRGen: An Integrated Vision and Language Model for the Generation of Clinically Accurate Chest X-Ray Image-Report Pairs
Authors: Junjie Shentu; Noura Al Moubayed
Venue: WACV 2024 (main)

Abstract: Chest X-Ray (CXR) images play a crucial role in clinical practice, providing vital support for diagnosis and treatment. Augmenting the CXR dataset with synthetically generated CXR images annotated with radiology reports can enhance the performance of deep learning models for various tasks. However, existing studies have primarily focused on generating unimodal data of either images or reports. In this study, we propose an integrated model, CXR-IRGen, designed specifically for generating CXR image-report pairs. Our model follows a modularized structure consisting of a vision module and a language module. Notably, we present a novel prompt design for the vision module by combining both text embedding and image embedding of a reference image. Additionally, we propose a new CXR report generation model as the language module, which effectively leverages a large language model and self-supervised learning strategy. Experimental results demonstrate that our new prompt is capable of improving the general quality (FID) and clinical efficacy (AUROC) of the generated images, with average improvements of 15.84% and 1.84%, respectively. Moreover, the proposed CXR report generation model outperforms baseline models in terms of clinical efficacy (F1 score) and exhibits a high-level alignment of image and text, as the best F1 score of our model is 6.93% higher than the state-of-the-art CXR report generation model. Our code is available at https://github.com/junjie-shentu/CXR-IRGen.
Overcoming Catastrophic Forgetting for Multi-Label Class-Incremental Learning
Authors: Xiang Song; Kuang Shu; Songlin Dong; Jie Cheng; Xing Wei; Yihong Gong
Venue: WACV 2024 (main)

Abstract: Despite the recent progress of class-incremental learning (CIL) methods, their capabilities in real-world scenarios such as multi-label settings remain unexplored. This paper focuses on a more practical CIL problem named multi-label class-incremental learning (MLCIL). MLCIL requires the vision models to overcome catastrophic forgetting of old knowledge while learning new classes from multi-label samples. Direct application of existing CIL methods to MLCIL leads to label absence, representative sample selection, and feature dilution problems. To address these problems, we present a novel AdaPtive Pseudo-Label-drivEn (APPLE) framework consisting of three components. First, the adaptive pseudo-label strategy is proposed to solve the label absence problem, which leverages the old model to annotate old classes for new samples. Second, a cluster sampling strategy is proposed to obtain more diverse samples to alleviate catastrophic forgetting under the MLCIL setting better. Finally, a class attention decoder is designed to mitigate the object feature dilution problem in multi-label samples. The extensive experiments on PASCAL VOC 2007 and MS-COCO demonstrate that our proposed method significantly outperforms other representative state-of-the-art CIL methods.
PatchRefineNet: Improving Binary Segmentation by Incorporating Signals From Optimal Patch-Wise Binarization
Authors: Savinay Nagendra; Daniel Kifer
Venue: WACV 2024 (main)

Abstract: The purpose of binary segmentation models is to determine which pixels belong to an object of interest (e.g., which pixels in an image are part of roads). The models assign a logit score (i.e., probability) to each pixel and these are converted into predictions by thresholding (i.e., each pixel with logit score >= t is predicted to be part of a road). However, a common phenomenon in current and former state-of-the-art segmentation models is spatial bias -- in some patches, the logit scores are consistently biased upwards and in others they are consistently biased downwards. These biases cause false positives and false negatives in the final predictions. In this paper, we propose PatchRefineNet (PRN), a small network that sits on top of a base segmentation model and learns to correct its patch-specific biases. Across a wide variety of base models, PRN consistently helps them improve mIoU by 2-3%. One of the key ideas behind PRN is the addition of a novel supervision signal during training. Given the logit scores produced by the base segmentation model, each pixel is given a pseudo-label that is obtained by optimally thresholding the logit scores in each image patch. Incorporating these pseudo-labels into the loss function of PRN helps correct systematic biases and reduce false positives/negatives. Although we mainly focus on binary segmentation, we also show how PRN can be extended to saliency detection and few-shot segmentation. We also discuss how the ideas can be extended to multi-class segmentation. Source code is available at https://github.com/savinay95n/PatchRefineNet.
Feature Augmentation Based Test-Time Adaptation
Authors: Younggeol Cho; Youngrae Kim; Junho Yoon; Seunghoon Hong; Dongman Lee
Venue: WACV 2025 (main)

Abstract: Test-time adaptation (TTA) allows a model to be adapted to an unseen domain without accessing the source data. Due to the nature of practical environments TTA has a limited amount of data for adaptation. Recent TTA methods further restrict this by filtering input data for reliability making the effective data size even smaller and limiting adaptation potential. To address this issue We propose Feature Augmentation based Test-time Adaptation (FATA) a simple method that fully utilizes the limited amount of input data through feature augmentation. FATA employs Normalization Perturbation to augment features and adapts the model using the FATA loss which makes the outputs of the augmented and original features similar. FATA is model-agnostic and can be seamlessly integrated into existing models without altering the model architecture. We demonstrate the effectiveness of FATA on various models and scenarios on ImageNet-C and Office-Home validating its superiority in diverse real-world conditions. Code is available at https://github.com/RangeWING/FATA.
Agtech Framework for Cranberry-Ripening Analysis using Vision Foundation Models
Authors: Faith Johnson; Ryan Meegan; Jack Lowry; Peter Oudemans; Kristin Dana
Venue: WACV 2025 (main)

Abstract: Agricultural domains are being transformed by recent advances in AI and computer vision that support quantitative visual evaluation. Using aerial and ground imaging over a time series we develop a framework for characterizing the ripening process of cranberry crops a crucial component for precision agriculture tasks such as comparing crop breeds (high-throughput phenotyping) and detecting disease. Using drone imaging we capture images from 20 waypoints across multiple bogs and using ground-based imaging (hand-held camera) we image same bog patch using fixed fiducial markers. Both imaging methods are repeated to gather a multi-week time series spanning the entire growing season. Aerial imaging provides multiple samples to compute a distribution of albedo values. Ground imaging enables tracking of individual berries for a detailed view of berry appearance changes. Using vision transformers (ViT) for feature detection after segmentation we extract a high dimensional feature descriptor of berry appearance. Interpretability of appearance is critical for plant biologists and cranberry growers to support crop breeding decisions (e.g. comparison of berry varieties from breeding programs). For interpretability we create a 2D manifold of cranberry appearance by using a UMAP dimensionality reduction on ViT features. This projection enables quantification of ripening paths and a useful metric of ripening rate. We demonstrate the comparison of four cranberry varieties based on our ripening assessments. This work is the first of its kind and has future impact for cranberries and for other crops including wine grapes olives blueberries and maize. Aerial and ground datasets are made publicly available.
Test-Time Adaptation in Point Clouds: Leveraging Sampling Variation with Weight Averaging
Authors: Ali Bahri; Moslem Yazdanpanah; Mehrdad Noori; Sahar Dastani Oghani; Milad Cheraghalikhani; David Osowiechi; Farzad Beizaee; Gustavo A. Vargas Hakim; Ismail Ben Ayed; Christian Desrosiers
Venue: WACV 2025 (main)

Abstract: Test-Time Adaptation (TTA) addresses distribution shifts during testing by adapting a pretrained model without access to source data. In this work we propose a novel TTA approach for 3D point cloud classification combining sampling variation with weight averaging. Our method leverages Farthest Point Sampling (FPS) and K-Nearest Neighbors (KNN) to create multiple point cloud representations adapting the model for each variation using the TENT algorithm. The final model parameters are obtained by averaging the adapted weights leading to improved robustness against distribution shifts. Extensive experiments on ModelNet40-C ShapeNet-C and ScanObjectNN-C datasets with different backbones (Point-MAE PointNet DGCNN) demonstrate that our approach consistently outperforms existing methods while maintaining minimal resource overhead. The proposed method effectively enhances model generalization and stability in challenging real-world conditions. The implementation is available at: https://github.com/AliBahri94/SVWA_TTA.git.
Self-Supervised Anomaly Segmentation via Diffusion Models with Dynamic Transformer UNet
Authors: Komal Kumar; Snehashis Chakraborty; Dwarikanath Mahapatra; Behzad Bozorgtabar; Sudipta Roy
Venue: WACV 2025 (main)

Abstract: A robust anomaly detection mechanism should possess the capability to effectively remediate anomalies restoring them to a healthy state while preserving essential healthy information. Despite the efficacy of existing generative models in learning the underlying distribution of healthy reference data they face primary challenges when it comes to efficiently repair larger anomalies or anomalies situated near high pixel-density regions. In this paper we introduce a self-supervised anomaly detection method based on a diffusion model that samples from multi-frequency four dimensional simplex noise and makes predictions using our proposed Dynamic Transformer UNet (DTUNet). This simplex-based noise function helps address primary problems to some extent and is scalable for three-dimensional and colored images. In the evolution of ViT our developed architecture serving as the backbone for the diffusion model is tailored to treat time and noise image patches as tokens. We incorporate long skip connections bridging the shallow and deep layers along with smaller skip connections within these layers. Furthermore we integrate a partial diffusion Markov process which reduces sampling time thus enhancing scalability. Our method surpasses existing generative-based anomaly detection methods across three diverse datasets which include BrainMRI Brats2021 and the MVtec dataset. It achieves an average improvement of +10.1% in Dice coefficient +10.4% in IOU and +9.6% in AUC. Our source code is made publicly available on Github.
RGB2Point: 3D Point Cloud Generation from Single RGB Images
Authors: Jae Joong Lee; Bedrich Benes
Venue: WACV 2025 (main)

Abstract: We introduce RGB2Point an unposed single-view RGB image to a 3D point cloud generation based on Transformer. RGB2Point takes an input image of an object and generates a dense 3D point cloud. Contrary to prior works based on CNN layers and diffusion-denoising approaches we use pre-trained Transformer layers that are fast and generate high-quality point clouds with consistent quality over available categories. Our generated point clouds demonstrate high quality on a real-world dataset as evidenced by improved Chamfer distance (51.15%) and Earth Mover's distance (36.17%) metrics compared to the current state-of-the-art. Additionally our approach shows a better quality on a synthetic dataset achieving better Chamfer distance (39.26%) Earth Mover's distance (26.95%) and F-score (47.16%). Moreover our method produces 63.1% more consistent high-quality results across various object categories compared to prior works. Furthermore RGB2Point is computationally efficient requiring only 2.3GB of VRAM to reconstruct a 3D point cloud from a single RGB image and our implementation generates the results 15133x faster than a SOTA diffusion-based model.
Shadow Removal Refinement via Material-Consistent Shadow Edges
Authors: Shilin Hu; Hieu Le; ShahRukh Athar; Sagnik Das; Dimitris Samaras
Venue: WACV 2025 (main)

Abstract: Shadow boundaries can be confused with material boundaries as both exhibit sharp changes in luminance or contrast within a scene. However shadows do not modify the intrinsic color or texture of surfaces. Therefore on both sides of shadow edges traversing regions with the same material the original color and texture should be the same if the shadow is removed properly. These shadow/shadow-free pairs are very useful but difficult-to-collect supervision signals. The crucial contribution of this paper is to learn how to identify those shadow edges that traverse material-consistent regions and how to use them as self-supervision for shadow removal refinement during test time. To achieve this we fine-tune SAM an image segmentation foundation model to produce a shadow-invariant segmentation and then extract material-consistent shadow edges by comparing the SAM segmentation with the shadow mask. Utilizing these shadow edges we introduce color- and texture-consistency losses to enhance the shadow removal process. We demonstrate the effectiveness of our method in improving shadow removal results on more challenging in-the-wild images outperforming the state-of-the-art shadow removal methods. Additionally we propose a new metric and an annotated dataset for evaluating the performance of shadow removal methods without the need for paired shadow/shadow-free data. Our code and dataset are available at: https://github.com/cvlab-stonybrook/ShadowRemovalRefine
ChromaDistill : Colorizing Monochrome Radiance Fields with Knowledge Distillation
Authors: Ankit Dhiman; Srinath R; Srinjay Sarkar; Lokesh Boregowda; Venkatesh Babu Radhakrishnan
Venue: WACV 2025 (main)

Abstract: Colorization is a well-explored problem in the domains of image and video processing. However extending colorization to 3D scenes presents significant challenges. Recent Neural Radiance Field (NeRF) and Gaussian-Splatting(3DGS) methods enable high-quality novel-view synthesis for multi-view images. However the question arises: How can we colorize these 3D representations? This work presents a method for synthesizing colorized novel views from input grayscale multi-view images. Using image or video colorization methods to colorize novel views from these 3D representations naively will yield output with severe inconsistencies. We introduce a novel method to use powerful image colorization models for colorizing 3D representations. We propose a distillation-based method that transfers color from these networks trained on natural images to the target 3D representation. Notably this strategy does not add any additional weights or computational overhead to the original representation during inference. Extensive experiments demonstrate that our method produces high-quality colorized views for indoor and outdoor scenes showcasing significant cross-view consistency advantages over baseline approaches. Our method is agnostic to the underlying 3D representation and easily generalizable to NeRF and 3DGS methods. Further we validate the efficacy of our approach in several diverse applications: 1.) Infra-Red (IR) multi-view images and 2.) Legacy grayscale multi-view image sequences. Project Webpage: https://val.cds.iisc.ac.in/chroma-distill.github.io/