GenAI4ELab/papercli-papers huggingface dataset:
Information Extraction from Heterogeneous Documents without Ground Truth Labels using Synthetic Label Generation and Knowledge Distillation
Authors: Aniket Bhattacharyya; Anurag Tripathi
Venue: WACV 2025 (main)

Abstract: Invoices and receipts submitted by employees are visually rich documents (VRDs) with textual visual and layout information. To protect against the risk of fraud and abuse it is crucial for organizations to efficiently extract desired information from submitted receipts. This helps in the assessment of key factors such as appropriateness of the expense claim adherence to spending and transaction policies the validity of the receipt as well as downstream anomaly detection at various levels. These documents are heterogeneous with multiple formats and languages uploaded with different image qualities and often do not contain ground truth labels for the efficient training of models. In this paper we propose Task Aware Instruction-based Labelling (TAIL) a method for synthetic label generation in VRD corpuses without labels and fine-tune a multimodal Visually Rich Document Understanding Model (VRDU) on TAIL labels using response-based knowledge distillation without using the teacher model's weights or training dataset to conditionally generate annotations in the appropriate format. Using a benchmark external dataset where ground truth labels are available we demonstrate conditions under which our approach performs at par with Claude 3 Sonnet through empirical studies. We then show that the resulting model performs at par or better on the internal expense documents of a large multinational organization than state-of-the-art LMM (large multimodal model) Claude 3 Sonnet while being 85% less costly and  5X faster and outperforms layout-aware baselines by more than 10% in Average Normalized Levenshtein Similarity (ANLS) scores due to its ability to reason and extract information from rare formats. Finally we illustrate the usage of our approach in overpayment prevention.
Exo2EgoDVC: Dense Video Captioning of Egocentric Procedural Activities using Web Instructional Videos
Authors: Takehiko Ohkawa; Takuma Yagi; Taichi Nishimura; Ryosuke Furuta; Atsushi Hashimoto; Yoshitaka Ushiku; Yoichi Sato
Venue: WACV 2025 (main)

Abstract: We propose a novel benchmark for cross-view knowledge transfer of dense video captioning adapting models from web instructional videos with exocentric views to an egocentric view. While dense video captioning (predicting time segments and their captions) is primarily studied with exocentric videos (e.g. YouCook2) benchmarks with egocentric videos are restricted due to data scarcity. To overcome the limited video availability transferring knowledge from abundant exocentric web videos is demanded as a practical approach. However learning the correspondence between exocentric and egocentric views is difficult due to their dynamic view changes. The web videos contain shots showing either full-body or hand regions while the egocentric view is constantly shifting. This necessitates the in-depth study of cross-view transfer under complex view changes. To this end we first create a real-life egocentric dataset (EgoYC2) whose captions follow the definition of YouCook2 captions enabling transfer learning between these datasets with access to their ground-truth. To bridge the view gaps we propose a view-invariant learning method using adversarial training which consists of pre-training and fine-tuning stages. Our experiments confirm the effectiveness of overcoming the view change problem and knowledge transfer to egocentric views. Our benchmark pushes the study of cross-view transfer into a new task domain of dense video captioning and envisions methodologies that describe egocentric videos in natural language.
Street TryOn: Learning In-the-Wild Virtual Try-On from Unpaired Person Images
Authors: Aiyu Cui; Jay Mahajan; Viraj Shah; Preeti Gomathinayagam; Chang Liu; Svetlana Lazebnik
Venue: WACV 2025 (main)

Abstract: Most virtual try-on research is motivated to serve the fashion business by generating images to demonstrate garments on studio models at a lower cost. However virtual try-on should be a broader application that also allows customers to visualize garments on themselves using their own casual photos known as in-the-wild try-on. Unfortunately the existing methods which achieve plausible results for studio try-on settings perform poorly in the in-the-wild context. This is because these methods often require paired images (garment images paired with images of people wearing the same garment) for training. While such paired data is easy to collect from shopping websites for studio settings it is difficult to obtain for in-the-wild scenes. In this work we fill the gap by (1) introducing a StreetTryOn benchmark to support in-the-wild virtual try-on applications and (2) proposing a novel method to learn virtual try-on from a set of in-the-wild person images directly without requiring paired data. We tackle the unique challenges including warping garments to more diverse human poses and rendering more complex backgrounds faithfully by a novel DensePose warping correction method combined with diffusion-based conditional inpainting. Our experiments show competitive performance for standard studio try-on tasks and SOTA performance for street try-on and cross-domain try-on tasks.
Elemental Composite Prototypical Network: Few-Shot Object Detection on Outdoor 3D Point Cloud Scenes
Authors: Arkadipta De; Vartika Sengar; Daksh Thapar; Mahesh Chandran; Manohar Kaul
Venue: WACV 2025 (main)

Abstract: This paper introduces the Elemental Composite Prototypical Network (ECPN) a novel approach to few-shot learning (FSL) in outdoor 3D point cloud object detection. Such point clouds are inherently non-uniformly packed and show marked intra-class variations due to aberrations in lidar scanning methods. Due to the limited availability of examples in the FSL setting the intra-class variations serve as a much more formidable challenge to traditional detection algorithms. ECPN employs a novel prototypical learning method that solves the issues mentioned above. We generate and leverage multiple elemental prototypes for each class to capture essential geometric features from limited examples. These elemental prototypes are then combined in a weighted manner to arrive at composite prototypes that score relevant and irrelevant features in the elemental prototypes with respect to the query point cloud scene. Moreover we introduce a novel feature-similarity-discrimination loss to refine the model's ability to distinguish between relevant objects and their background significantly improving object detection accuracy in FSL scenarios. Our extensive testing on the nuScenes dataset demonstrates that ECPN significantly outperforms existing baselines offering a robust solution to the complexities of outdoor few-shot 3D object detection (O-FS3D) and setting a new standard for future research.
DiffMesh: A Motion-Aware Diffusion Framework for Human Mesh Recovery from Videos
Authors: Ce Zheng; Xianpeng Liu; Qucheng Peng; Tianfu Wu; Pu Wang; Chen Chen
Venue: WACV 2025 (main)

Abstract: Human mesh recovery (HMR) provides rich human body information for various real-world applications such as gaming human-computer interaction and virtual reality. While image-based HMR methods have achieved impressive results they often struggle to recover humans in dynamic scenarios leading to temporal inconsistencies and non-smooth 3D motion predictions due to the absence of human motion. In contrast video-based approaches leverage temporal information to mitigate this issue. In this paper we present DiffMesh an innovative motion-aware diffusion framework for video-based HMR. DiffMesh establishes a bridge between diffusion models and human motion efficiently generating accurate and smooth output mesh sequences by incorporating human motion within the forward process and reverse process in the diffusion model. Extensive experiments are conducted on the widely used datasets (Human3.6M and 3DPW) which demonstrate the effectiveness and efficiency of our DiffMesh. Visual comparisons in real-world scenarios further highlight DiffMesh's suitability for practical applications. The project webpage is: https://zczcwh.github.io/ diffmesh_page/
CrackStructures and CrackEnsembles: The Power of Multi-View for 2.5D Crack Detection
Authors: Christian Benz; Volker Rodehorst
Venue: WACV 2025 (main)

Abstract: While research on structural crack segmentation at the image level remains highly active progress beyond two dimensions has been limited. This stagnation is largely due to the lack of available data for crack detection in higher dimensions. To address this limitation we introduce CrackStructures a dataset tailored for real-world 2.5D crack segmentation encompassing 15 segments from five distinct structures. Additionally we present CrackEnsembles a complementary semi-synthetic dataset that combines real textures with synthetic geometry to enhance the development of learning-based algorithms. Coupled with a baseline for multi-view crack instance segmentation this work establishes a solid foundation for advancing algorithms that support real-world structural inspection.
SplatFace: Gaussian Splat Face Reconstruction Leveraging an Optimizable Surface
Authors: Jiahao Luo; Jing Liu; James Davis
Venue: WACV 2025 (main)

Abstract: We present SplatFace a novel Gaussian splatting framework designed for 3D human face reconstruction without reliance on accurate pre-determined geometry. Our method is designed to simultaneously deliver both high-quality novel view rendering and accurate 3D mesh reconstructions. We incorporate a generic 3D Morphable Model (3DMM) to provide a surface geometric structure making it possible to reconstruct faces with a limited set of input images. We introduce a joint optimization strategy that refines both the Gaussians and the morphable surface through a synergistic non-rigid alignment process. A novel distance metric splat-to-surface is proposed to improve alignment by considering both the Gaussian position and covariance. The surface information is also utilized to incorporate a world-space densification process resulting in superior reconstruction quality. Our experimental analysis demonstrates that the proposed method is competitive with both other Gaussian splatting techniques in novel view synthesis and other 3D reconstruction methods in producing 3D face meshes with high geometric precision.
GazeSearch: Radiology Findings Search Benchmark
Authors: Trong Thang Pham; Tien-Phat Nguyen; Yuki Ikebe; Akash Awasthi; Zhigang Deng; Carol C. Wu; Hien Nguyen; Ngan Le
Venue: WACV 2025 (main)

Abstract: Medical eye-tracking data is an important information source for understanding how radiologists visually interpret medical images. This information not only improves the accuracy of deep learning models for X-ray analysis but also their interpretability enhancing transparency in decision-making. However the current eye-tracking data is dispersed unprocessed and ambiguous making it difficult to derive meaningful insights. Therefore there is a need to create a new dataset with more focus and purposeful eyetracking data improving its utility for diagnostic applications. In this work we propose a refinement method inspired by the target-present visual search challenge: there is a specific finding and fixations are guided to locate it. After refining the existing eye-tracking datasets we transform them into a curated visual search dataset called GazeSearch specifically for radiology findings where each fixation sequence is purposefully aligned to the task of locating a particular finding. Subsequently we introduce a scan path prediction baseline called ChestSearch specifically tailored to GazeSearch. Finally we employ the newly introduced GazeSearch as a benchmark to evaluate the performance of current state-of-the-art methods offering a comprehensive assessment for visual search in the medical imaging domain. Code is available at https://github.com/ UARK-AICV/GazeSearch.
Pix2Poly: A Sequence Prediction Method for End-to-End Polygonal Building Footprint Extraction from Remote Sensing Imagery
Authors: Yeshwanth Kumar Adimoolam; Charalambos Poullis; Melinos Averkiou
Venue: WACV 2025 (main)

Abstract: Extraction of building footprint polygons from remotely sensed data is essential for several urban understanding tasks such as reconstruction navigation & mapping. Despite significant progress in the area extracting accurate polygonal vector building footprints remains an open problem. In this paper we introduce Pix2Poly an attention-based end-to-end trainable & differentiable deep neural network capable of directly generating explicit high-quality building footprints in a ring graph format. Pix2Poly employs a generative encoder-decoder transformer to produce a sequence of graph vertex tokens whose connectivity information is learned by an optimal matching network. Compared to previous graph learning methods ours is a truly end-to-end trainable approach that extracts high-quality building footprints & road networks without requiring complicated computationally intensive raster loss functions & intricate training pipelines. Upon evaluating Pix2Poly on several complex & challenging datasets we report that Pix2Poly outperforms state-of-the-art methods in several vector shape quality metrics while being an entirely explicit method. Our code is available at https://github.com/yeshwanth95/Pix2Poly.
Flatness Improves Backbone Generalisation in Few-Shot Classification
Authors: Rui Li; Martin Trapp; Marcus Klasson; Arno Solin
Venue: WACV 2025 (main)

Abstract: Deployment of deep neural networks in real-world settings typically requires adaptation to new tasks with few examples. Few-shot classification (FSC) provides a solution to this problem by leveraging pre-trained backbones for fast adaptation to new classes. However approaches for multi-domain FSC typically result in complex pipelines aimed at information fusion and task-specific adaptation without consideration of the importance of backbone training. In this work we introduce an effective strategy for backbone training and selection in multi-domain FSC by utilizing flatness-aware training and fine-tuning. Our work is theoretically grounded and empirically performs on par or better than state-of-the-art methods despite being simpler. Further our results indicate that backbone training is crucial for good generalisation in FSC across different adaptation methods.
ORFormer: Occlusion-Robust Transformer for Accurate Facial Landmark Detection
Authors: Jui-Che Chiang; Hou-Ning Hu; Bo-Syuan Hou; Chia-Yu Tseng; Yu-Lun Liu; Min-Hung Chen; Yen-Yu Lin
Venue: WACV 2025 (main)

Abstract: Although facial landmark detection (FLD) has gained significant progress existing FLD methods still suffer from performance drops on partially non-visible faces such as faces with occlusions or under extreme lighting conditions or poses. To address this issue we introduce ORFormer a novel transformer-based method that can detect non-visible regions and recover their missing features from visible parts. Specifically ORFormer associates each image patch token with one additional learnable token called the messenger token. The messenger token aggregates features from all but its patch. This way the consensus between a patch and other patches can be assessed by referring to the similarity between its regular and messenger embeddings enabling non-visible region identification. Our method then recovers occluded patches with features aggregated by the messenger tokens. Leveraging the recovered features ORFormer compiles high-quality heatmaps for the downstream FLD task. Extensive experiments show that our method generates heatmaps resilient to partial occlusions. By integrating the resultant heatmaps into existing FLD methods our method performs favorably against the state of the arts on challenging datasets such as WFLW and COFW.
Unleashing Potentials of Vision-Language Models for Zero-Shot HOI Detection
Authors: Moyuru Yamada; Nimish Dharamshi; Ayushi Kohli; Prasad Kasu; Ainulla Khan; Manu Ghulyani
Venue: WACV 2025 (main)

Abstract: Human-Object Interaction (HOI) detection aims to localize human-object pairs and recognize their interactions as <human action object> triplets. Recent advancements in pre-trained vision-language model (VLM) have improved zero-shot HOI detection enabling identification of unseen triplets. However existing methods leverage the VLM as an additional encoder only for interaction prediction not for human/object detection. This limitation hinders their ability to detect unseen objects. Furthermore the additional encoder increases both model size and computational cost. This paper proposes a novel HOI detection framework ECI-HOI which unleashes potentials of the pre-trained VLM for the zero-shot HOI detection by leveraging it for both of the sub-tasks. We first employ CLIP as a single image encoder reducing redundancy in the network architecture. In addition we propose an instance selector and a HO pair decoder to effectively harmonize the human/object detection and the interaction prediction in zero-shot manner. We evaluate our model under various settings on HICO-DET and our two new testsets: out-of-distribution image testset and novel object testset. Our model outperforms the state-of-the-art models while reducing the model size by more than 50% especially achieving a +10.01 mAP improvement under the unseen object setting on HICO-DET. The results on the proposed datasets highlight the zero-shot performance of our model on more challenging settings.
Memory-Efficient Pseudo-Labeling for Online Source-Free Universal Domain Adaptation using a Gaussian Mixture Model
Authors: Pascal Schlachter; Simon Wagner; Bin Yang
Venue: WACV 2025 (main)

Abstract: In practice domain shifts are likely to occur between training and test data necessitating domain adaptation (DA) to adjust the pre-trained source model to the target domain. Recently universal domain adaptation (UniDA) has gained attention for addressing the possibility of an additional category (label) shift between the source and target domain. This means new classes can appear in the target data some source classes may no longer be present or both at the same time. For practical applicability UniDA methods must handle both source-free and online scenarios enabling adaptation without access to the source data and performing batch-wise updates in parallel with prediction. In an online setting preserving knowledge across batches is crucial. However existing methods often require substantial memory which is impractical because memory is limited and valuable in particular on embedded systems. Therefore we consider memory-efficiency as an additional constraint. To achieve memory-efficient online source-free universal domain adaptation (SF-UniDA) we propose a novel method that continuously captures the distribution of known classes in the feature space using a Gaussian mixture model (GMM). This approach combined with entropy-based out-of-distribution detection allows for the generation of reliable pseudo-labels. Finally we combine a contrastive loss with a KL divergence loss to perform the adaptation. Our approach not only achieves state-of-the-art results in all experiments on the DomainNet and Office-Home datasets but also significantly outperforms the existing methods on the challenging VisDA-C dataset setting a new benchmark for online SF-UniDA. Our code is available at https://github.com/pascalschlachter/GMM.
SyncViolinist: Music-Oriented Violin Motion Generation Based on Bowing and Fingering
Authors: Hiroki Nishizawa; Keitaro Tanaka; Asuka Hirata; Shugo Yamaguchi; Qi Feng; Masatoshi Hamanaka; Shigeo Morishima
Venue: WACV 2025 (main)

Abstract: Automatically generating realistic musical performance motion can greatly enhance digital media production often involving collaboration between professionals and musicians. However capturing the intricate body hand and finger movements required for accurate musical performances is challenging. Existing methods often fall short due to the complex mapping between audio and motion typically requiring additional inputs like scores or MIDI data. In this work we present SyncViolinist a multi-stage end-to-end framework that generates synchronized violin performance motion solely from audio input. Our method overcomes the challenge of capturing both global and fine-grained performance features through two key modules: a bowing/fingering module and a motion generation module. The bowing/fingering module extracts detailed playing information from the audio which the motion generation module uses to create precise coordinated body motions reflecting the temporal granularity and nature of the violin performance. We demonstrate the effectiveness of SyncViolinist with significantly improved qualitative and quantitative results from unseen violin performance audio outperforming state-of-the-art methods. Extensive subjective evaluations involving professional violinists further validate our approach. The code and dataset are available at https://github.com/Kakanat/SyncViolinist.
Deduce and Select Evidences with Language Models for Training-Free Video Goal Inference
Authors: Yeo Keat Ee; Hao Zhang; Alexander Matyasko; Basura Fernando
Venue: WACV 2025 (main)

Abstract: We introduce ViDSE a Video framework that Deduce and Selects visual Evidence for training-free video goal inference using language models. Unlike approaches that directly apply vision-language models (VLM) or combine VLM+LLM to process dense video visuals ViDSE explicitly selects relevant visual evidence (e.g. frames) based on the hypothesis deduced by the LLM. This approach not only improves accuracy but also reveals the logical process behind the model's decisions enhancing explainability. Our experiments demonstrate that this selection process significantly reduces ambiguity in the subsequent inference reasoning stage and outperforms VLM-only and VLM+LLM models on goal inference tasks such as CrossTask and COIN. We further validate ViDSE's generalizability and robustness on action recognition tasks such as ActivityNet and UCF-101 under training-free and open-vocabulary conditions. We observe that ViDSE easily generalizes to other video tasks (e.g. action recognition) requiring filtering of redundant and irrelevant information.
DrIFT: Autonomous Drone Dataset with Integrated Real and Synthetic Data Flexible Views and Transformed Domains
Authors: Fardad Dadboud; Hamid Azad; Varun Mehta; Miodrag Bolic; Iraj Mantegh
Venue: WACV 2025 (main)

Abstract: Dependable visual drone detection is crucial for the secure integration of drones into the airspace. However drone detection accuracy is significantly affected by domain shifts due to environmental changes varied points of view and background shifts. To address these challenges we present the DrIFT dataset specifically developed for visual drone detection under domain shifts. DrIFT includes fourteen distinct domains each characterized by shifts in point of view synthetic-to-real data season and adverse weather. DrIFT uniquely emphasizes background shift by providing background segmentation maps to enable background-wise metrics and evaluation. Our new uncertainty estimation metric MCDO-map features lower postprocessing complexity surpassing traditional methods. We use the MCDO-map in our uncertainty-aware unsupervised domain adaptation method demonstrating superior performance to SOTA unsupervised domain adaptation techniques. The dataset is available at: https://github.com/CARG-uOttawa/DrIFT.git.