GenAI4ELab/papercli-papers huggingface dataset:
PatchFinder: Leveraging Visual Language Models for Accurate Information Retrieval using Model Uncertainty
Authors: Roman Colman; Minh Vu; Manish Bhattarai; Martin Ma; Hari Viswanathan; Daniel O'Malley; Javier Santos
Venue: WACV 2025 (main)

Abstract: For decades corporations and governments have relied on scanned documents to record vast amounts of information. However extracting this information is a slow and tedious process due to the sheer volume and complexity of these records. The rise of Vision Language Models (VLMs) presents a way to efficiently and accurately extract the information out of these documents. The current automated workflow often requires a two-step approach involving the extraction of information using optical character recognition software and subsequent usage of large language models for processing this information. Unfortunately these methods encounter significant challenges when dealing with noisy scanned documents often requiring computationally expensive language models to handle high information density effectively. In this study we propose PatchFinder an algorithm that builds upon VLMs to improve information extraction. First we devise a confidence-based score called Patch Confidence based on the Maximum Softmax Probability of the VLMs' output to measure the model's confidence in its predictions. Using this metric PatchFinder determines a suitable patch size partitions the input document into overlapping patches and generates confidence-based predictions for the target information. Our experimental results show that PatchFinder leveraging Phi-3v a 4.2 billion parameter VLM achieves an accuracy of 94% on our dataset of 190 noisy scanned documents outperforming ChatGPT-4o by 18.5 percentage points.
DiTAS: Quantizing Diffusion Transformers via Enhanced Activation Smoothing
Authors: Zhenyuan Dong; Sai Qian Zhang
Venue: WACV 2025 (main)

Abstract: Diffusion Transformers (DiTs) have recently attracted significant interest from both industry and academia due to their enhanced capabilities in visual generation surpassing the performance of traditional diffusion models that employ U-Net. However the improved performance of DiTs comes at the expense of higher parameter counts and implementation costs which significantly limits their deployment on resource-constrained devices like mobile phones. We propose DiTAS a data-free post-training quantization (PTQ) method for efficient DiT inference. DiTAS relies on the proposed temporal-aggregated smoothing techniques to mitigate the impact of the channel-wise outliers within the input activations leading to much lower quantization error under extremely low bitwidth. To further enhance the performance of the quantized DiT we adopt the layer-wise grid search strategy to optimize the smoothing factor. Moreover we integrate a training-free LoRA module for weight quantization leveraging alternating optimization to minimize quantization errors without additional fine-tuning. Experimental results demonstrate that our approach enables 4-bit weight 8-bit activation (W4A8) quantization for DiTs while maintaining comparable performance as the full-precision model. Code is available at https://github.com/DZY122/DiTAS.
Improving Conditional Diffusion Models through Re-Noising from Unconditional Diffusion Priors
Authors: Kangfu Mei; Nithin Gopalakrishnan Nair; Vishal Patel
Venue: WACV 2025 (main)

Abstract: Conditional diffusion probabilistic models can model the distribution of natural images and can generate diverse and realistic samples based on given conditions. However oftentimes their results can be unrealistic with observable color shifts and textures. We believe that this issue results from the divergence between the probabilistic distribution learned by the model and the distribution of natural images. The delicate conditions gradually enlarge the divergence during each sampling timestep. To address this issue we introduce a new method that brings the predicted samples to the training data manifold using a pretrained unconditional diffusion model. The unconditional model acts as a regularizer and reduces the divergence introduced by the conditional model at each sampling step. We perform comprehensive experiments to demonstrate the effectiveness of our approach on super-resolution colorization turbulence removal and image-deraining tasks. The improvements obtained by our method suggest that the priors can be incorporated as a general plugin for improving conditional diffusion models.
SmartKC++: Improving Performance of Smartphone-Based Corneal Topographers
Authors: Vaibhav Ganatra; Siddhartha Gairola; Pallavi Joshi; Anand Balasubramaniam; Kaushik Murali; Arivunithi Varadharajan; Bellamkonda Mallikarjuna; Nipun Kwatra; Mohit Jain
Venue: WACV 2025 (main)

Abstract: Keratoconus an ocular condition marked by progressive corneal thinning and outward bulging presents diagnostic challenges due to the high cost and lack of portability in conventional corneal topographers. These limitations restrict accessibility for many necessitating affordable and mobile alternatives. Innovations like SmartKC offer a low-cost and portable alternative however there still remains some gaps in performance when compared to commercial topographers. In this paper we introduce SmartKC++ a series of innovative methodological improvements to the image processing pipeline of SmartKC aimed at significantly enhancing its diagnostic precision and reliability. Our comprehensive evaluation on a dataset comprising 303 eye images reveals that SmartKC++ boosts the accuracy of automated keratoconus diagnosis by 7.69% relative to SmartKC.
Partial Texture VAE: Color and Texture Encoder for Rock Particle Images
Authors: Tetsushi Yamada; Simone Di Santo
Venue: WACV 2025 (main)

Abstract: We propose Partial Texture VAE (PT-VAE) for rock particle image analysis a variant of the variational autoencoder (VAE) specialized in encoding color and texture properties from arbitrarily sized and shaped unresized images containing invalid background pixels into the vectors with the same size. PT-VAE integrates partial convolution and the mask aware feature aggregation layer in the encoder and is trained by minimizing a simplified style loss in an unsupervised manner. Input texture images do not require resizing or cropping thus preventing deformation and under-representation of the overall texture pattern. This consideration is often overlooked in data-driven texture analysis. Experiments using our textured rock particle images show that the encoded features efficiently capture color and texture information and are invariant to size and shape of the image as well as invalid pixels. Image retrieval tests show that the PT-VAE incorporating the mask aware Texture Encoding Layer (TEL) outperformed other configurations and existing methods.
Learning Keypoints for Multi-Agent Behavior Analysis using Self-Supervision
Authors: Daniel Khalil; Christina Liu; Pietro Perona; Jennifer Sun; Markus Marks
Venue: WACV 2025 (main)

Abstract: The study of social interactions and collective behaviors through multi-agent video analysis is crucial in biology. While self-supervised keypoint discovery has emerged as a promising solution to reduce the need for manual keypoint annotations existing methods often struggle with videos containing multiple interacting agents especially those of the same species and color. To address this we introduce B-KinD-multi a novel approach that leverages pre-trained video segmentation models to guide keypoint discovery in multi-agent scenarios. This eliminates the need for time-consuming manual annotations on new experimental settings and organisms. Extensive evaluations demonstrate improved keypoint regression and downstream behavioral classification in videos of flies mice and rats. Furthermore our method generalizes well to other species including ants bees and humans highlighting its potential for broad applications in automated keypoint annotation for multi-agent behavior analysis. Code available under: B-KinD-Multi
GaitContour: Efficient Gait Recognition Based on a Contour-Pose Representation
Authors: Yuxiang Guo; Anshul Shah; Jiang Liu; Ayush Gupta; Rama Chellappa; Cheng Peng
Venue: WACV 2025 (main)

Abstract: Gait recognition holds the promise to robustly identify subjects based on walking patterns instead of appearance information. In recent years this field has been dominated by learning methods based on two input formats: silhouette images and sparse keypoints. Compared to image-based approaches keypoint-based methods can achieve significantly higher efficiency due to their sparsity. However sparsity also results in information loss thereby reducing performance. In this work we propose a novel keypoint-based Contour-Pose representation which compactly encodes both body shape and parts information. We further propose a local-to-global architecture called GaitContour to leverage this novel representation and efficiently compute subject embedding in two stages. The first stage consists of a local transformer that extracts features from five different body regions. The second stage then aggregates the regional features to estimate a global human gait representation. Such a design significantly reduces the complexity of the attention operation and improves both efficiency and performance. Through large scale experiments GaitContour is shown to perform significantly better than previous keypoint-based methods. Furthermore the Contour-Pose representation also achieves new SoTA performances on fusion-based gait recognition methods.
LIPIDS: Learning-Based Illumination Planning in Discretized (Light) Space for Photometric Stereo
Authors: Ashish Tiwari; Mihirkumar Sutariya; Shanmuganathan Raman
Venue: WACV 2025 (main)

Abstract: Photometric stereo is a powerful technique for estimating per-pixel surface normals from images under varied illumination. Although several methods address photometric stereo with different image (or light) counts ranging from one to two to a hundred very few focus on learning optimal lighting configuration. Finding an optimal configuration is challenging due to the large number of possible lighting directions. Moreover exhaustive sampling of all possibilities is impractical due to time and resource constraints. Photometric stereo methods have demonstrated promising performance on existing datasets which feature limited light directions sparsely sampled from the light space. Therefore can we optimally utilize these datasets for illumination planning? In this work we introduce LIPIDS - Learning-based Illumination Planning In Discretized light Space to achieve minimal and optimal lighting configurations for photometric stereo under arbitrary light distribution. We propose a Light Sampling Network (LSNet) that optimizes the lighting direction for a fixed number of lights by minimizing the normal loss through a normal regression network. The learned light configurations can directly estimate surface normals during inference even using an off-the-shelf photometric stereo method. Extensive qualitative and quantitative analysis on synthetic and real-world datasets show that photometric stereo under learned lighting configurations through LIPIDS either surpasses or is nearly comparable to existing illumination planning methods across different photometric stereo backbones.
Multi-View Image Diffusion via Coordinate Noise and Fourier Attention
Authors: Justin Theiss; Norman Müller; Daeil Kim; Aayush Prakash
Venue: WACV 2025 (main)

Abstract: Recently text-to-image generation with diffusion models has made significant advancements in both higher fidelity and generalization capabilities compared to previous baselines. However generating holistic multi-view consistent images from prompts still remains an important and challenging task. To address this challenge we propose a diffusion process that attends to time-dependent spatial frequencies of features with a novel attention mechanism as well as novel noise initialization technique and cross-attention loss. This Fourier-based attention block focuses on features from non-overlapping regions of the generated scene in order to better align the global appearance. Our noise initialization technique incorporates shared noise and low spatial frequency information derived from pixel coordinates and depth maps to induce noise correlations across views. The cross-attention loss further aligns features sharing the same prompt across the scene. Our technique improves SOTA on several quantitative metrics with qualitatively better results when compared to other state-of-the-art approaches for multi-view consistency.
Tumor Synthesis Conditioned on Radiomics
Authors: Jonghun Kim; Inye Na; Eun Sook Ko; Hyunjin Park
Venue: WACV 2025 (main)

Abstract: Due to privacy concerns obtaining large datasets is challenging in medical image analysis especially with 3D modalities like Computed Tomography (CT) and Magnetic Resonance Imaging (MRI). Existing generative models developed to address this issue often face limitations in output diversity and thus cannot accurately represent 3D medical images. We propose a tumor-generation model that utilizes radiomics features as generative conditions. Radiomics features are high-dimensional handcrafted semantic features that are biologically well-grounded and thus are good candidates for conditioning. Our model employs a GAN-based model to generate tumor masks and a diffusion-based approach to generate tumor texture conditioned on radiomics features. Our method allows the user to generate tumor images according to user-specified radiomics features such as size shape and texture at an arbitrary location. This enables the physicians to easily visualize tumor images to better understand tumors according to changing radiomics features. Our approach allows for the removal manipulation and repositioning of tumors generating various tumor types in different scenarios. The model has been tested on tumors in four different organs (kidney lung breast and brain) across CT and MRI. The synthesized images are shown to effectively aid in training for downstream tasks and their authenticity was also evaluated through expert evaluations. Our method has potential usage in treatment planning with diverse synthesized tumors. Our code is available at github.com/jongdory/TS-Radiomics.
PTQ4VM: Post-Training Quantization for Visual Mamba
Authors: Younghyun Cho; Changhun Lee; Seonggon Kim; Eunhyeok Park
Venue: WACV 2025 (main)

Abstract: Visual Mamba is an approach that extends the selective space state model Mamba to vision tasks. It processes image tokens sequentially in a fixed order accumulating information to generate outputs. Despite its growing popularity for delivering high-quality outputs at a low computational cost across various tasks Visual Mamba is highly susceptible to quantization which makes further performance improvements challenging. Our analysis reveals that the fixed token access order in Visual Mamba introduces unique quantization challenges which we categorize into three main issues: 1) token-wise variance 2) channel-wise outliers and 3) a long tail of activations. To address these challenges we propose Post-Training Quantization for Visual Mamba (PTQ4VM) which introduces two key strategies: Per-Token Static (PTS) quantization and Joint Learning of Smoothing Scale and Step Size (JLSS). To the our best knowledge this is the first quantization study on Visual Mamba. PTQ4VM can be applied to various Visual Mamba backbones converting the pretrained model to a quantized format in under 15 minutes without notable quality degradation. Extensive experiments on large-scale classification and regression tasks demonstrate its effectiveness achieving up to 1.83X speedup on GPUs with negligible accuracy loss compared to FP16. Our code is available at https://github.com/YoungHyun197/ptq4vm.
AIDE: Improving 3D Open-Vocabulary Semantic Segmentation by Aligned Vision-Language Learning
Authors: Yimu Wang; Krzysztof Czarnecki
Venue: WACV 2025 (main)

Abstract: 3D open-vocabulary semantic segmentation aims at recognizing countless categories beyond the limited set of annotations used in traditional settings. Due to the lack of large-scale 3D-vision-language segmentation data instead of training models from scratch the current solutions distill knowledge from pre-trained 2D vision-language models (VLMs) into 3D models. However this distillation is supervised by misaligned 3D-scene-image-to-text data pairs consequently leading to suboptimal performance. Moreover as 2D VLMs are trained on 2D datasets text encoders of VLMs which serve as the bridge between 3D models and an unbounded set of categories lack 3D semantics. In this paper to address these issues and improve generalization performance we propose an AlIgned 3D Open-Vocabulary SEmantic Segmentation framework called AIDE with two novel modules. To collect high-quality and well-aligned 3D-scene-image-to-text pairs our CLIP-rewarded alignment module (i) generates diverse captions of multi-view images of 3D scenes to capture details by varying the temperatures and then (ii) samples captions based on their similarity to corresponding images for rich and accurate associations. Next to adapt 2D VLMs to 3D contexts our adaptive segmentation module introduces (iii) trainable tokens within the input space and each layer of the text encoder while freezing the text encoder to avoid catastrophic forgetting. Extensive experiments show that AIDE outperforms previous methods by a large margin on three representative benchmarks demonstrating its effectiveness.
Recurrence-Based Vanishing Point Detection
Authors: Skanda Bharadwaj; Robert T. Collins; Yanxi Liu
Venue: WACV 2025 (main)

Abstract: Classical approaches to Vanishing Point Detection (VPD) rely solely on the presence of explicit straight lines in images. Recent supervised deep learning methods rely on learned filters from labeled datasets. We propose an alternative unsupervised approach: Recurrence-based Vanishing Point Detection (R-VPD) that uses implicit lines derived from discovered recurring correspondences in addition to explicit lines. Furthermore we contribute two new VPD datasets: 1) a Synthetic Image dataset with 3200 ground truth vanishing points and 2) a Real-World Image dataset with 1400 human annotated vanishing points. We compare our method with two classical methods and two state-of-the-art deep learning-based VPD methods. We demonstrate that our unsupervised approach outperforms all the methods on the synthetic images dataset outperforms the classical methods and is on par with the supervised learning approaches on real-world images. Code and data can be found here: http://vision.cse.psu.edu/data/data.shtml
AMP-ViT: Optimizing Vision Transformer Efficiency with Adaptive Mixed-Precision Post-Training Quantization
Authors: Yu-Shan Tai; An-Yeu Wu
Venue: WACV 2025 (main)

Abstract: Vision transformers (ViTs) have revolutionized computer vision but face significant challenges due to their high computational and memory demands. Existing post-training quantization methods struggle to maintain performance at low bit-widths due to activation asymmetry and reliance on manual configurations. To overcome these challenges we introduce SymAlign to address activation asymmetry and reduce clamping loss. Additionally we propose AutoScale an automatic and data-driven mechanism that adapts to variant activations. We incorporate the above-mentioned techniques and propose an adaptive mixed-precision post-training quantization framework for vision transformers (AMP-ViT). Our comprehensive approach addresses asymmetry variant distribution and uneven sensitivities making it the first to tackle these challenges thoroughly. Our experiments on ViT DeiT and Swin demonstrate significant accuracy improvements compared with SOTA on the ImageNet dataset. Specifically our proposed methods achieve accuracy improvements ranging from 0.90% to 23.35% on 4-bit ViTs with single-precision and from 3.82% to 78.14% on 5-bit fully quantized ViTs with mixed-precision.
Delta-NAS: Difference of Architecture Encoding for Predictor-Based Evolutionary Neural Architecture Search
Authors: Arjun Sridhar; Yiran Chen
Venue: WACV 2025 (main)

Abstract: Neural Architecture Search (NAS) continues to serve a key roll in the design and development of neural networks for task specific deployment. Modern NAS techniques struggle to deal with ever increasing search space complexity and compute cost constraints. Existing approaches can be categorized into two buckets: fine-grained computational expensive NAS and coarse-grained low cost NAS. Our objective is to craft an algorithm with the capability to perform fine-grain NAS at a low cost. We propose projecting the problem to a lower dimensional space through predicting the difference in accuracy of a pair of similar networks. This paradigm shift allows for reducing computational complexity from exponential down to linear with respect to the size of the search space. We present a strong mathematical foundation for our algorithm in addition to extensive experimental results across a host of common NAS Benchmarks. Our methods significantly out performs existing works achieving better performance coupled with a significantly higher sample efficiency.
VLTP: Vision-Language Guided Token Pruning for Task-Oriented Segmentation
Authors: Hanning Chen; Yang Ni; Wenjun Huang; Yezi Liu; SungHeon Jeong; Fei Wen; Nathaniel Bastian; Hugo Latapie; Mohsen Imani
Venue: WACV 2025 (main)

Abstract: Vision Transformers (ViTs) have emerged as the backbone of many segmentation models consistently achieving state-of-the-art (SOTA) performance. However their success comes at a significant computational cost. Image token pruning is one of the most effective strategies to address this complexity. However previous approaches fall short when applied to more complex task-oriented segmentation (TOS) where the class of each image patch is not predefined but dependent on the specific input task. This work introduces the Vision Language Guided Token Pruning (VLTP) a novel token pruning mechanism that can accelerate ViT-based segmentation models particularly for TOS guided by multi-modal large language model (MLLM). We argue that ViT does not need to process every image token through all of its layers--only the tokens related to reasoning tasks are necessary. We design a new pruning decoder to take both image tokens and vision-language guidance as input to predict the relevance of each image token to the task. Only image tokens with high relevance are passed to deeper layers of the ViT. Experiments show that the VLTP framework reduces the computational costs of ViT by approximately 25% without performance degradation and by around 40% with only a 1% performance drop. The code associated with this study can be found at this URL.