GenAI4ELab/papercli-papers huggingface dataset:
FastVideoEdit: Leveraging Consistency Models for Efficient Text-to-Video Editing
Authors: Youyuan Zhang; Xuan Ju; James J. Clark
Venue: WACV 2025 (main)

Abstract: Diffusion models have demonstrated remarkable capabilities in text-to-image and text-to-video generation opening up possibilities for video editing based on textual input. However the computational cost associated with sequential sampling in diffusion models poses challenges for efficient video editing. Existing approaches relying on image generation models for video editing suffer from time-consuming one-shot fine-tuning additional condition extraction or DDIM inversion making real-time applications impractical. In this work we propose FastVideoEdit an efficient zero-shot video editing approach inspired by Consistency Models (CMs). By leveraging the self-consistency property of CMs we eliminate the need for time-consuming inversion or additional condition extraction reducing editing time. Our method enables direct mapping from source video to target video with strong preservation ability through attention control. This results in improved speed advantages as fewer sampling steps can be used while maintaining comparable generation quality. Experimental results validate the state-of-the-art performance and speed advantages of FastVideoEdit across evaluation metrics encompassing editing speed temporal consistency and text-video alignment. The source code is available at github.com/youyuan-zhang/FastVideoEdit.
InDistill: Information Flow-Preserving Knowledge Distillation for Model Compression
Authors: Ioannis Sarridis; Christos Koutlis; Giorgos Kordopatis-Zilos; Yiannis Kompatsiaris; Symeon Papadopoulos
Venue: WACV 2025 (main)

Abstract: In this paper we introduce InDistill a method that serves as a warmup stage for enhancing Knowledge Distillation (KD) effectiveness. InDistill focuses on transferring critical information flow paths from a heavyweight teacher to a lightweight student. This is achieved via a training scheme based on curriculum learning that considers the distillation difficulty of each layer and the critical learning periods when the information flow paths are established. This procedure can lead to a student model that is better prepared to learn from the teacher. To ensure the applicability of InDistill across a wide range of teacher-student pairs we also incorporate a pruning operation when there is a discrepancy in the width of the teacher and student layers. This pruning operation reduces the width of the teacher's intermediate layers to match those of the student allowing direct distillation without the need for an encoding stage. The proposed method is extensively evaluated using various pairs of teacher-student architectures on CIFAR-10 CIFAR-100 and ImageNet datasets demonstrating that preserving the information flow paths consistently increases the performance of the baseline KD approaches on both classification and retrieval settings. The code is available at https://github.com/gsarridis/InDistill.
Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis
Authors: Po-Hsuan Huang; Jeng-Lin Li; Chin-Po Chen; Ming-Ching Chang; Wei-Chao Chen
Venue: WACV 2025 (main)

Abstract: Recent advancements in large vision-language models (LVLM) have significantly enhanced their ability to comprehend visual inputs alongside natural language. However a major challenge in their real-world application is hallucination where LVLMs generate non-existent visual elements eroding user trust. The underlying mechanism driving this multimodal hallucination is poorly understood. Minimal research has illuminated whether contexts such as sky tree or grass field involve the LVLM in hallucinating a frisbee. We hypothesize that hidden factors such as objects contexts and semantic foreground-background structures induce hallucination. This study proposes a novel causal approach: a hallucination probing system to identify these hidden factors. By analyzing the causality between images text prompts and network saliency we systematically explore interventions to block these factors. Our experimental findings show that a straightforward technique based on our analysis can significantly reduce hallucinations. Additionally our analyses indicate the potential to edit network internals to minimize hallucinated outputs.
SV-data2vec: Guiding Video Representation Learning with Latent Skeleton Targets
Authors: Zorana Doždor; Tomislav Hrkac; Zoran Kalafatic
Venue: WACV 2025 (main)

Abstract: Recent advancements in action recognition leverage both skeleton and video modalities to achieve state-of-the-art performance. However due to the challenges of early fusion which tends to underutilize the strengths of each modality existing methods often resort to late fusion consequently leading to more complex designs. Additionally self-supervised learning approaches utilizing both modalities remain underexplored. In this paper we introduce SV-data2vec a novel self-supervised framework for learning from skeleton and video data. Our approach employs a student-teacher architecture where the teacher network generates contextualized targets based on skeleton data. The student network performs a masked prediction task using both skeleton and visual data. Remarkably after pretraining with both modalities our method allows for fine-tuning with RGB data alone achieving results on par with multimodal approaches by effectively learning video representations through skeleton data guidance. Extensive experiments on benchmark datasets NTU RGB+D 60 NTU RGB+D 120 and Toyota Smarthome confirm that our method outperforms existing RGB based state-of-the-art techniques. The code is available at github.com/zoranadozdor/SVdata2vec.
DocTTT: Test-Time Training for Handwritten Document Recognition using Meta-Auxiliary Learning
Authors: Wenhao Gu; Li Gu; Ziqiang Wang; Ching Y Suen; Yang Wang
Venue: WACV 2025 (main)

Abstract: Despite recent significant advancements in Handwritten Document Recognition (HDR) the efficient and accurate recognition of text against complex backgrounds diverse handwriting styles and varying document layouts remains a practical challenge. Moreover this issue is seldom addressed in academic research particularly in scenarios with minimal annotated data available. In this paper we introduce the DocTTT framework to address these challenges. The key innovation of our approach is that it uses test-time training to adapt the model to each specific input during testing. We propose a novel Meta-Auxiliary learning approach that combines Meta-learning and self-supervised Masked Autoencoder(MAE). During testing we adapt the visual representation parameters using a self-supervised MAE loss. During training we learn the model parameters using a meta-learning framework so that the model parameters are learned to adapt to a new input effectively. Experimental results show that our proposed method significantly outperforms existing state-of-the-art approaches on benchmark datasets.
LLaVA-SpaceSGG: Visual Instruct Tuning for Open-Vocabulary Scene Graph Generation with Enhanced Spatial Relations
Authors: Mingjie Xu; Mengyang Wu; Yuzhi Zhao; Jason Chun Lok Li; Weifeng Ou
Venue: WACV 2025 (main)

Abstract: Scene Graph Generation (SGG) converts visual scenes into structured graph representations providing deeper scene understanding for complex vision tasks. However existing SGG models often overlook essential spatial relationships and struggle with generalization in open-vocabulary contexts. To address these limitations we propose LLaVASpaceSGG a multimodal large language model (MLLM) designed for open-vocabulary SGG with enhanced spatial relation modeling. To train it we collect the SGG instruction-tuning dataset named SpaceSGG. This dataset is constructed by combining publicly available datasets and synthesizing data using open-source models within our data construction pipeline. It combines object locations object relations and depth information resulting in three data formats: spatial SGG description question-answering and conversation. To enhance the transfer of MLLMs' inherent capabilities to the SGG task we introduce a two-stage training paradigm. Experiments show that LLaVASpaceSGG outperforms other open-vocabulary SGG methods boosting recall by 8.6% and mean recall by 28.4% compared to the baseline. Our codebase dataset and trained models are publicly accessible on GitHub at the following URL: https://github.com/Endlinc/LLaVA-SpaceSGG.
BASED: Bundle-Adjusting Surgical Endoscopic Dynamic Video Reconstruction using Neural Radiance Fields
Authors: Shreya Saha; Zekai Liang; Shan Lin; Jingpei Lu; Michael Yip; Sainan Liu
Venue: WACV 2025 (main)

Abstract: Reconstruction of deformable scenes from endoscopic videos is important for many applications such as intraoperative navigation surgical visual perception and robotic surgery. It is a foundational requirement for realizing autonomous robotic interventions for minimally invasive surgery. However previous approaches in this domain have been limited by their modular nature and are confined to specific camera and scene settings. Our work adopts the Neural Radiance Fields (NeRF) approach to learning 3D implicit representations of scenes that are both dynamic and deformable over time and furthermore with unknown camera poses. This work removes the constraints of known camera poses and overcomes the drawbacks of the state-of-the-art unstructured dynamic scene reconstruction technique which relies on the static part of the scene for accurate reconstruction. Through several experimental datasets we demonstrate the versatility of our proposed model to adapt to diverse camera and scene settings and show its promise for both current and future robotic surgical systems.
Make VLM Recognize Visual Hallucination on Cartoon Character Image with Pose Information
Authors: Bumsoo Kim; Wonseop Shin; Kyuchul Lee; Yonghoon Jung; Sanghyun Seo
Venue: WACV 2025 (main)

Abstract: Leveraging large-scale Text-to-Image (TTI) models have become a common technique to generate training or reference data in the field of image synthesis video editing 3D reconstruction. However semantic structural visual hallucinations which contain perceptually critical defects remain a concern especially in non-photorealistic rendering domains such as cartoon pixelization-style character. We propose a novel semantic structural hallucination detection system in cartoon-style images generated by TTI models collecting a new cartoon-hallucination dataset. Our approach leverages pose-aware in-context visual learning (PA-ICVL) with public Vision-Language Models (VLMs) utilizing both RGB images and pose information. By incorporating pose guidance from a fine-tuned pose estimator we enable VLMs to make more accurate decisions. Experimental results demonstrate significant improvements in identifying visual hallucinations compared to baseline methods relying solely on RGB images. Within selected two VLMs GPT-4v Gemini pro vision our proposed PA-ICVL improves the hallucination detection with 50% to 78% 57% to 80% respectively. This research advances a capability of TTI models toward real-world applications by mitigating visual hallucinations via in-context visual learning expanding their potential in non-photorealistic domains. Besides when VLM confront ambiguous tasks this results showcase thought-provoking insights how users boost the domain-adaptive capability of VLM by harnessing additional conditions. The dataset and demo VLMs are provided in the corresponding Git repository: https://github.com/gh-BumsooKim/Cartoon-Hallucinations-Detection.
SODA: Spectral Orthogonal Decomposition Adaptation for Diffusion Models
Authors: Xinxi Zhang; Song Wen; Ligong Han; Felix Juefei-Xu; Akash Srivastava; Junzhou Huang; Vladimir Pavlovic; Hao Wang; Molei Tao; Dimitris Metaxas
Venue: WACV 2025 (main)

Abstract: Adapting large-scale pre-trained generative models in a parameter-efficient manner is gaining traction. Traditional methods like low rank adaptation achieve parameter efficiency by imposing constraints but may not be optimal for tasks requiring high representation capacity. We propose a novel spectrum-aware adaptation framework for generative models. Our method adjusts both singular values and their basis vectors of pretrained weights. Using the Kronecker product and efficient Stiefel optimizers we achieve parameter-efficient adaptation of orthogonal matrices. Specifically we introduce Spectral Orthogonal Decomposition Adaptation (SODA) which balances computational efficiency and representation capacity. Extensive evaluations on text-to-image diffusion models demonstrate SODA's effectiveness offering a spectrum-aware alternative to existing fine-tuning methods.
Retaining and Enhancing Pre-Trained Knowledge in Vision-Language Models with Prompt Ensembling
Authors: Donggeun Kim; Yujin Jo; Myungjoo Lee; Taesup Kim
Venue: WACV 2025 (main)

Abstract: The advancement of vision-language models particularly the Contrastive Language-Image Pre-training (CLIP) model has revolutionized the field of machine learning by enabling robust zero-shot learning capabilities. These capabilities allow models to understand and respond to previously unseen data without task-specific training. However adapting CLIP to integrate specialized knowledge from various domains while retaining its zero-shot capabilities remains a significant challenge. To address this we introduce a novel prompt ensemble learning approach called Group-wise Prompt Ensemble (GPE). This method aims to enhance CLIP's zero-shot capabilities by incorporating new domain knowledge while improving its adaptability and robustness against data distribution shifts. Our approach hinges on three main strategies: prompt grouping with masked attention to optimize CLIP's adaptability while safeguarding its zero-shot capabilities; the incorporation of auxiliary prompts for the seamless integration of new domain insights without disrupting the original model's representation; and an ensemble learning strategy that effectively merges original and new knowledge. Through rigorous experimentation including more challenging cross-dataset transfer evaluations our GPE method redefines the benchmarks for the adaptability and efficiency of vision-language models surpassing existing models across various scenarios.
DisCo: Discovering Common Affordance from Large Models for Actionable Part Perception
Authors: Youpeng Wen; Yi Zhu; Zhihao Zhan; Pengzhen Ren; Jianhua Han; Hang Xu; Shen Zhao; Xiaodan Liang
Venue: WACV 2025 (main)

Abstract: Actionable part perception for robotic object manipulation needs to perceive parts over open-world object categories within 3D space which is challenging as the appearance of the same part on different objects varies greatly. It is frequently observed that despite the huge intra-class difference in appearance the parts share common interactive functions over different objects i.e. common affordance. According to this observation we propose DisCo a novel technique that Discovers Common affordance information from powerful large models for guiding the actionable part perception across open-world objects. Specifically we first use a large language model to identify the object names that each part potentially belongs to and a text-to-image generative model to generate image examples for the queried objects constructing image-text paired data that indicate visual and semantic information of common affordance. Then our model encodes the common affordance information by learning to pair the object-part images with their text descriptions. Subsequently the 2D-pixel features are distilled into 3D space thus the 3D point features are enriched with not only the semantic information of open-set objects but also the common affordance information which is highly generalizable. Finally a segmentation head and a pose regression network are developed to predict more accurate results of part segmentation and pose estimation improving the success rate of robotic object manipulation. Extensive experiments show that our method outperforms existing methods on the part instance and semantic segmentation by significant margins of 4.8% mAP 5.4% AP50 and 3.9% mIoU on the unseen object categories.
Endoscopic Scoring and Localization in Unconstrained Clinical Trial Videos
Authors: Jinlin Xiang; Hillol Sarker; Bozhao Qi; Ruisu Zhang; Roger Trullo; Salvatore Badalamenti; Maria Wiekowski; Annie Kruger; Etienne Pochet; Qi Tang; Wei Zhao
Venue: WACV 2025 (main)

Abstract: Endoscopic assessment using the Mayo clinic score (or Mayo score 4 categories) is currently the standard for diagnosing and evaluating mucosal disease activities. However annotating Mayo scores is time-consuming and often relies on weakly labeled evaluations from central and local readers (doctors) leading to a large number of unlabeled or mislabeled video clips. Additionally such labels also suffer from data imbalance due to patient distributions and varying disease severity levels. This gap underscores the need for more customizable and refined methods for endoscopic scoring and localization. To address these challenges we introduce an end-to-end pipeline with a new dataset for endoscopic scoring and localization in unconstrained clinical trial videos. Specifically we propose an automated scoring system that includes an active learning-based pre-processing stage for cleaning the raw videos which are then weakly labeled by doctors. Therefore we obtain a comprehensive dataset for endoscopic Mayo scoring comprising approximately 49.9 hours of video and  86423 clips providing a solid foundation for developing endoscopic scoring and localization models. Then we propose a video classification model for Mayo score classifications. For personalized disease quantification in localization we introduce a novel 1D trajectory model with a novel cumulative disease score that addresses the limitations of previous 3D trajectory projection methods. Our dataset and end-to-end pipeline offer a valuable foundation for advancing endoscopic clinical trial research.
Just Shift It: Test-Time Prototype Shifting for Zero-Shot Generalization with Vision-Language Models
Authors: Elaine Sui; Xiaohan Wang; Serena Yeung-Levy
Venue: WACV 2025 (main)

Abstract: Advancements in vision-language models (VLMs) have propelled the field of computer vision particularly in the zero-shot learning setting. Despite their promise the effectiveness of these models often diminishes due to domain shifts in test environments. To address this we introduce the Test-Time Prototype Shifting (TPS) framework a pioneering approach designed to adapt VLMs to test datasets using unlabeled test inputs. Our method is based on the notion of modulating per-class prototypes in the shared embedding space. By pre-computing and caching prototypes generated with the pre-trained text encoder TPS not only facilitates optimization-free prototype reuse for subsequent predictions but also enables seamless integration with current advancements in prompt engineering. At test-time TPS dynamically learns shift vectors for each prototype based solely on the given test sample effectively bridging the domain gap and enhancing classification accuracy. A notable aspect of our framework is its significantly reduced memory and computational demands when compared to conventional text-prompt tuning methods. Extensive evaluations across 15 image classification datasets involving natural distribution shifts and cross-dataset generalization as well as in context-dependent visual reasoning demonstrate TPS's superior performance achieving state-of-the-art results while reducing resource requirements. Code is available at https://github.com/elaine-sui/TPS.
Ad^2mix: Adversarial and Adaptive Mixup for Unsupervised Domain Adaptation
Authors: Lei Zhu; Yanyu Xu; Yong Liu; Rick Siow Mong Goh; Xinxing Xu
Venue: WACV 2025 (main)

Abstract: Transformer has recently gained tremendous popularity in unsupervised domain adaptation tasks due to its superior generalization ability. State-of-the-art methods leverage mixup to build an intermediate domain to reduce domain gap. However such strategy becomes less effective when the domain gap becomes large as the domain gap between intermediate domain and source domain is not minimized and the constructed intermediate domain is non-informative. How to address the adaptation problem when domain gap becomes large is an important research problem in domain adaptation. In this paper we propose an adversarial and adaptive mixup (Ad^2mix) framework which gradually aligns the intermediate domain towards source domain to fully unleash the potential of both the transformer architecture and mixup to address the large domain gap problem. Specifically we formulate a general framework for intermediate domain learning with mixup. We propose adversarial mixup with a specially designed mixup alike adversarial adaptation operation to reduce the domain gap between the intermediate domain and source domain. To construct an informative intermediate domain unlike existing methods which utilize a Beta distribution to generate mixup coefficients to interpolate source and target data we adaptively assign mixup coefficient for each target data instance based on their transferability and discriminativity information. Our framework creates a natural curriculum of intermediate domains from near source domain to near target domain for gradual adaptation. Extensive experimental studies and evaluations on three public domain adaptation benchmark datasets and one medical domain adaptation task demonstrate the superiority of our framework.
WARLearn: Weather-Adaptive Representation Learning
Authors: Shubham Agarwal; Raz Birman; Ofer Hadar
Venue: WACV 2025 (main)

Abstract: This paper introduces WARLearn a novel framework designed for adaptive representation learning in challenging and adversarial weather conditions. Leveraging the in-variance principal used in Barlow Twins we demonstrate the capability to port the existing models initially trained on clear weather data to effectively handle adverse weather conditions. With minimal additional training our method exhibits remarkable performance gains in scenarios characterized by fog and low-light conditions. This adaptive framework extends its applicability beyond adverse weather settings offering a versatile solution for domains exhibiting variations in data distributions. Furthermore WARLearn is invaluable in scenarios where data distributions undergo significant shifts over time enabling models to remain updated and accurate. Our experimental findings reveal a remarkable performance with a mean average precision (mAP) of 52.6% on unseen real-world foggy dataset (RTTS). Similarly in low light conditions our framework achieves a mAP of 55.7% on unseen real-world low light dataset (ExDark). Notably WARLearn surpasses the performance of state-of-the-art frameworks including FeatEnHancer Image Adaptive YOLO DENet C2PNet PairLIE and ZeroDCE by a substantial margin in adverse weather improving the baseline performance in both foggy and low light conditions. The WARLearn code is available at https://github.com/ShubhamAgarwal12/WARLearn