GenAI4ELab/papercli-papers huggingface dataset:
Generative Model-Based Fusion for Improved Few-Shot Semantic Segmentation of Infrared Images
Authors: Junno Yun; Mehmet Akçakaya
Venue: WACV 2025 (main)

Abstract: Infrared (IR) imaging is commonly used in various scenarios including autonomous driving fire safety and defense applications. Thus semantic segmentation of such images is of great interest. However this task faces several challenges including data scarcity differing contrast and input channel number compared to natural images and emergence of classes not represented in databases in certain scenarios such as defense applications. Few-shot segmentation (FSS) provides a framework to overcome these issues by segmenting query images using a few labeled support samples. However existing FSS models for IR images require paired visible RGB images which is a major limitation since acquiring such paired data is difficult or impossible in some applications. In this work we develop new strategies for FSS of IR images by using generative modeling and fusion techniques. To this end we propose to synthesize auxiliary data to provide additional channel information to complement the limited contrast in the IR images as well as IR data synthesis for data augmentation. Here the former helps the FSS model to better capture the relationship between the support and query sets while the latter addresses the issue of data scarcity. Finally to further improve the former aspect we propose a novel fusion ensemble module for integrating the two different modalities. Our methods are evaluated on different IR datasets and improve upon the state-of-the-art (SOTA) FSS models.
Similarity over Factuality: Are we Making Progress on Multimodal Out-of-Context Misinformation Detection?
Authors: Stefanos-Iordanis Papadopoulos; Christos Koutlis; Symeon Papadopoulos; Panagiotis C. Petrantonakis
Venue: WACV 2025 (main)

Abstract: Out-of-context (OOC) misinformation poses a significant challenge in multimodal fact-checking where images are paired with texts that misrepresent their original context to support false narratives. Recent research in evidence-based OOC detection has seen a trend towards increasingly complex architectures incorporating Transformers foundation models and large language models. In this study we introduce a simple yet robust baseline which assesses MUltimodal SimilaritiEs (MUSE) specifically the similarity between image-text pairs and external image and text evidence. Our results demonstrate that MUSE when used with conventional classifiers like Decision Tree Random Forest and Multilayer Perceptron can compete with and even surpass the state-of-the-art on the NewsCLIPpings and VERITE datasets at less than 1% of their computational complexity. Furthermore integrating MUSE in our proposed "Attentive Intermediate Transformer Representations" (AITR) significantly improved performance by 3.3% and 7.5% on NewsCLIPpings and VERITE respectively. Nevertheless the success of MUSE relying on surface-level patterns and shortcuts without examining factuality and logical inconsistencies raises critical questions about how we define the task construct datasets collect external evidence and overall how we assess progress in the field. We release our code at: https://github.com/stevejpapad/outcontext-misinfo-progress.
CRAFT: Designing Creative and Functional 3D Objects
Authors: Michelle Guo; Mia Tang; Hannah Cha; Ruohan Zhang; C. Karen Liu; Jiajun Wu
Venue: WACV 2025 (main)

Abstract: For designing a wide range of everyday objects the design process should be aware of both the human body and the underlying semantics of the design specification. However these two objectives present significant challenges to the current AI-based designing tools. In this work we present a method to synthesize body-aware 3D objects from a base mesh given an input body geometry and either text or image as guidance. The generated objects can be simulated on virtual characters or fabricated for real-world use. We propose to use a mesh deformation procedure that optimizes for both semantic alignment as well as contact and penetration losses. Using our method users can generate both virtual or real-world objects from text image or sketch without the need for manual artist intervention. We present both qualitative and quantitative results on various object categories demonstrating the effectiveness of our approach.
EfficientCrackNet: A Lightweight Model for Crack Segmentation
Authors: Abid Hasan Zim; Aquib Iqbal; Zaid Al-Huda; Asad Malik; Minoru Kuribayashi
Venue: WACV 2025 (main)

Abstract: Crack detection particularly from pavement images presents a formidable challenge in computer vision due to inherent complexities such as intensity inhomogeneity intricate topologies low contrast and noisy backgrounds. Automated crack detection is crucial for maintaining the structural integrity of essential infrastructures including buildings pavements and bridges. Existing lightweight methods often face challenges including computational inefficiency complex crack patterns and difficult backgrounds leading to inaccurate detection and impracticality for real-world applications. We propose EfficientCrackNet a lightweight hybrid model combining Convolutional Neural Networks (CNNs) and transformers for precise crack segmentation to address these limitations. EfficientCrackNet integrates depthwise separable convolutions (DSC) layers and MobileViT block to capture global and local features. The model employs an Edge Extraction Method (EEM) for efficient crack edge detection without pretraining and an Ultra-Lightweight Subspace Attention Module (ULSAM) to enhance feature extraction. Extensive experiments on three benchmark datasets Crack500 DeepCrack and GAPs384 demonstrate that EfficientCrackNet achieves superior performance compared to existing lightweight models requiring only 0.26M parameters and 0.483 GFLOPs. The proposed model offers an optimal balance between accuracy and computational efficiency outperforming state-of-the-art lightweight models and providing a robust and adaptable solution for real-world crack segmentation.
Multi-Level Feature Distillation of Joint Teachers Trained on Distinct Image Datasets
Authors: Adrian Iordache; Bogdan Alexe; Radu Tudor Ionescu
Venue: WACV 2025 (main)

Abstract: We propose a novel teacher-student framework to distill knowledge from multiple teachers trained on distinct datasets. Each teacher is first trained from scratch on its own dataset. Then the teachers are combined into a joint architecture which fuses the features of all teachers at multiple representation levels. The joint teacher architecture is fine-tuned on samples from all datasets thus gathering useful generic information from all data samples. Finally we employ a multi-level feature distillation procedure to transfer the knowledge to a student model for each of the considered datasets. We conduct image classification experiments on seven benchmarks and action recognition experiments on three benchmarks. To illustrate the power of our feature distillation procedure the student architectures are chosen to be identical to those of the individual teachers. To demonstrate the flexibility of our approach we combine teachers with distinct architectures. We show that our novel Multi-Level Feature Distillation (MLFD) can significantly surpass equivalent architectures that are either trained on individual datasets or jointly trained on all datasets at once. Furthermore we confirm that each step of the proposed training procedure is well motivated by a comprehensive ablation study. We publicly release our code at https://github.com/AdrianIordache/MLFD.
Learning Multiple Object States from Actions via Large Language Models
Authors: Masatoshi Tateno; Takuma Yagi; Ryosuke Furuta; Yoichi Sato
Venue: WACV 2025 (main)

Abstract: Recognizing the states of objects in a video is crucial in understanding the scene beyond actions and objects. For instance an egg can be raw cracked and whisked while cooking an omelet and these states can coexist simultaneously (an egg can be both raw and whisked). However most existing research assumes a single object state change (e.g. uncracked - cracked) overlooking the coexisting nature of multiple object states and the influence of past states on the current state. We formulate object state recognition as a multi-label classification task that explicitly handles multiple states. We then propose to learn multiple object states from narrated videos by leveraging large language models (LLMs) to generate pseudo-labels from the transcribed narrations capturing the influence of past states. The challenge is that narrations mostly describe human actions in the video but rarely explain object states. Therefore we use the LLM's knowledge of the relationship between actions and states to derive the missing object states. We further accumulate the derived object states to consider past state contexts to infer current object state pseudo-labels. We newly collect a dataset called the Multiple Object States Transition (MOST) dataset which includes manual multi-label annotation for evaluation purposes covering 60 object states across six object categories. Experimental results show that our model trained on LLM-generated pseudo-labels significantly outperforms strong vision-language models demonstrating the effectiveness of our pseudo-labeling framework that considers past context via LLMs.
Balancing Shared and Task-Specific Representations: A Hybrid Approach to Depth-Aware Video Panoptic Segmentation
Authors: Kurt H.W. Stolle
Venue: WACV 2025 (main)

Abstract: In this work we present Multiformer a novel approach to depth-aware video panoptic segmentation (DVPS) based on the mask transformer paradigm. Our method learns object representations that are shared across segmentation monocular depth estimation and object tracking subtasks. In contrast to recent unified approaches that progressively refine a common object representation we propose a hybrid method using task-specific branches within each decoder block ultimately fusing them into a shared representation at the block interfaces. Extensive experiments on the Cityscapes-DVPS and SemKITTI-DVPS datasets demonstrate that Multiformer achieves state-of-the-art performance across all DVPS metrics outperforming previous methods by substantial margins. With a ResNet-50 backbone Multiformer surpasses the previous best result by 3.0 DVPQ points while also improving depth estimation accuracy. Using a Swin-B backbone Multiformer further improves performance by 4.0 DVPQ points. Multiformer also provides valuable insights into the design of multi-task decoder architectures.
Cross-Aligned Fusion for Multimodal Understanding
Authors: Abhishek Rajora; Shubham Gupta; Suman Kundu
Venue: WACV 2025 (main)

Abstract: Recent multimodal frameworks often grapple with semantic misalignment and noise impeding effective integration of diverse modalities. In order to solve this problem this study presents CaMN (Cross-aligned Multimodal Network) a framework designed to enhance multimodal understanding through a robust cross-alignment mechanism. Unlike conventional fusion methods our framework aligns features extracted from images text and graphs via a tailored loss function enabling seamless integration and exploitation of complementary information. Leveraging Abstract Meaning Representation (AMR) we extract intricate semantic structures from textual data enriching the multimodal representation with contextual depth. Furthermore to enhance robustness we employ a masked autoencoder to simulate noise-independent feature space. Through comprehensive evaluation on the crisisMMD dataset CaMN demonstrates superior performance in crisis event classification tasks highlighting its potential in advancing multimodal understanding across diverse domains. Our code is available at https://github.com/brillard1/CaMN.
CorrFill: Enhancing Faithfulness in Reference-Based Inpainting with Correspondence Guidance in Diffusion Models
Authors: Kuan-Hung Liu; Cheng-Kun Yang; Min-Hung Chen; Yu-Lun Liu; Yen-Yu Lin
Venue: WACV 2025 (main)

Abstract: In the task of reference-based image inpainting an additional reference image is provided to restore a damaged target image to its original state. The advancement of diffusion models particularly Stable Diffusion allows for simple formulations in this task. However existing diffusion-based methods often lack explicit constraints on the correlation between the reference and damaged images resulting in lower faithfulness to the reference images in the inpainting results. In this work we propose CorrFill a training-free module designed to enhance the awareness of geometric correlations between the reference and target images. This enhancement is achieved by guiding the inpainting process with correspondence constraints estimated during inpainting utilizing attention masking in self-attention layers and an objective function to update the input tensor according to the constraints. Experimental results demonstrate that CorrFill significantly enhances the performance of multiple baseline diffusion-based methods including state-of-the-art approaches by emphasizing faithfulness to the reference images.
VIIS: Visible and Infrared Information Synthesis for Severe Low-Light Image Enhancement
Authors: Chen Zhao; Mengyuan Yu; Fan Yang; Peiguang Jing
Venue: WACV 2025 (main)

Abstract: Images captured in severe low-light circumstances often suffer from significant information absence. Existing singular modality image enhancement methods struggle to restore image regions lacking valid information. By leveraging light-impervious infrared images visible and infrared image fusion methods have the potential to reveal information hidden in darkness. However they primarily emphasize inter-modal complementation but neglect intra-modal enhancement limiting the perceptual quality of output images. To address these limitations we propose a novel task dubbed visible and infrared information synthesis (VIIS) which aims to achieve both information enhancement and fusion of the two modalities. Given the difficulty in obtaining ground truth in the VIIS task we design an information synthesis pretext task (ISPT) based on image augmentation. We employ a diffusion model as the framework and design a sparse attention-based dual-modalities residual (SADMR) conditioning mechanism to enhance information interaction between the two modalities. This mechanism enables features with prior knowledge from both modalities to adaptively and iteratively attend to each modality's information during the denoising process. Our extensive experiments demonstrate that our model qualitatively and quantitatively outperforms not only the state-of-the-art methods in relevant fields but also the newly designed baselines capable of both information enhancement and fusion. The code is available at https://github.com/Chenz418/VIIS.
Continual Learning of Personalized Generative Face Models with Experience Replay
Authors: Annie N. Wang; Luchao Qi; Roni Sengupta
Venue: WACV 2025 (main)

Abstract: We introduce a novel continual learning problem: how to sequentially update the weights of a personalized 2D and 3D generative face model as new batches of photos in different appearances styles poses and lighting are captured regularly. We observe that naive sequential fine-tuning of the model leads to catastrophic forgetting of past representations of the individual's face. We then demonstrate that a simple random sampling-based experience replay method is effective at mitigating catastrophic forgetting when a relatively large number of images can be stored and replayed. However for long-term deployment of these models with relatively smaller storage this simple random sampling-based replay technique also forgets past representations. Thus we introduce a novel experience replay algorithm that combines random sampling with StyleGAN's latent space to represent the buffer as an optimal convex hull. We observe that our proposed convex hull-based experience replay is more effective in preventing forgetting than a random sampling baseline and the lower bound. We introduce continual learning datasets for five celebrities along with the evaluation framework metrics and visualizations to examine this problem. See our project page for more details.
Decoupled PROB: Decoupled Query Initialization Tasks and Objectness-Class Learning for Open World Object Detection
Authors: Riku Inoue; Masamitsu Tsuchiya; Yuji Yasui
Venue: WACV 2025 (main)

Abstract: Open World Object Detection (OWOD) is a challenging computer vision task that extends standard object detection by (1) detecting and classifying unknown objects without supervision and (2) incrementally learning new object classes without forgetting previously learned ones. The absence of ground truths for unknown objects makes OWOD tasks particularly challenging. Many methods have addressed this by using pseudo-labels for unknown objects. The recently proposed Probabilistic Objectness transformer-based open-world detector (PROB) is a state-of-the-art model that does not require pseudo-labels for unknown objects as it predicts probabilistic objectness. However this method faces issues with learning conflicts between objectness and class predictions. To address this issue and further enhance performance we propose a novel model Decoupled PROB. Decoupled PROB introduces Early Termination of Objectness Prediction (ETOP) to stop objectness predictions at appropriate layers in the decoder resolving the learning conflicts between class and objectness predictions in PROB. Additionally we introduce Task-Decoupled Query Initialization (TDQI) which efficiently extracts features of known and unknown objects thereby improving performance. TDQI is a query initialization method that combines query selection and learnable queries and it is a module that can be easily integrated into existing DETR-based OWOD models. Extensive experiments on OWOD benchmarks demonstrate that Decoupled PROB surpasses all existing methods across several metrics significantly improving performance.
RT-DETRv3: Real-Time End-to-End Object Detection with Hierarchical Dense Positive Supervision
Authors: Shuo Wang; Chunlong Xia; Feng Lv; Yifeng Shi
Venue: WACV 2025 (main)

Abstract: RT-DETR is the first real-time end-to-end transformer-based object detector. Its efficiency comes from the framework design and the Hungarian matching. However compared to dense supervision detectors like the YOLO series the Hungarian matching provides much sparser supervision leading to insufficient model training and difficult to achieve optimal results. To address these issues we proposed a hierarchical dense positive supervision method based on RT-DETR named RT-DETRv3. Firstly we introduce a CNN-based auxiliary branch that provides dense supervision that collaborates with the original decoder to enhance the encoder's feature representation. Secondly to address insufficient decoder training we propose a novel learning strategy involving self-attention perturbation. This strategy diversifies label assignment for positive samples across multiple query groups thereby enriching positive supervisions. Additionally we introduce a shared-weight decoder branch for dense positive supervision to ensure more high-quality queries matching each ground truth. Notably all aforementioned modules are training-only. We conduct extensive experiments to demonstrate the effectiveness of our approach on COCO val2017. RT-DETRv3 significantly outperforms existing real-time detectors including the RT-DETR series and the YOLO series. For example RT-DETRv3-R18 achieves 48.1% AP (+1.6%/+1.4%) compared to RT-DETR-R18/RT-DETRv2-R18 while maintaining the same latency. Furthermore RT-DETRv3-R101 can attain an impressive 54.6% AP outperforming YOLOv10-X. The code will be released at https://github.com/clxia12/RT-DETRv3.
Exploiting VLM Localizability and Semantics for Open Vocabulary Action Detection
Authors: Wentao Bao; Kai Li; Yuxiao Chen; Deep A Patel; Martin Renqiang Min; Yu Kong
Venue: WACV 2025 (main)

Abstract: Action detection aims to detect (recognize and localize) human actions spatially and temporally in videos. Existing approaches focus on the closed-set setting where an action detector is trained and tested on videos from a fixed set of action categories. However this constrained setting is not viable in an open world where test videos inevitably come beyond the trained action categories. In this paper we address the practical yet challenging Open-Vocabulary Action Detection (OVAD) problem. It aims to detect any action in test videos while training a model on a fixed set of action categories. To achieve such an open-vocabulary capability we propose a novel method OpenMixer that exploits the inherent semantics and localizability of large vision-language models (VLM) within the family of query-based detection transformers (DETR). Specifically the OpenMixer is developed by spatial and temporal OpenMixer blocks (S-OMB and T-OMB) and a dynamically fused alignment (DFA) module. The three components collectively enjoy the merits of strong generalization from pre-trained VLMs and end-to-end learning from DETR design. Moreover we established OVAD benchmarks under various settings and the experimental results show that the OpenMixer performs the best over baselines for detecting seen and unseen actions. We release the codes models and dataset splits at: https://github.com/Cogito2012/OpenMixer.
Dance Any Beat: Blending Beats with Visuals in Dance Video Generation
Authors: Xuanchen Wang; Heng Wang; Dongnan Liu; Weidong Cai
Venue: WACV 2025 (main)

Abstract: Generating dance from music is crucial for advancing automated choreography. Current methods typically produce skeleton keypoint sequences instead of dance videos and lack the capability to make specific individuals dance which reduces their real-world applicability. These methods also require precise keypoint annotations complicating data collection and limiting the use of self-collected video datasets. To overcome these challenges we introduce a novel task: generating dance videos directly from images of individuals guided by music. This task enables the dance generation of specific individuals without requiring keypoint annotations making it more versatile and applicable to various situations. Our solution the Dance Any Beat Diffusion model (DabFusion) utilizes a reference image and a music piece to generate dance videos featuring various dance types and choreographies. The music is analyzed by our specially designed music encoder which identifies essential features including dance style movement and rhythm. DabFusion excels in generating dance videos not only for individuals in the training dataset but also for any previously unseen person. This versatility stems from its approach of generating latent optical flow which contains all necessary motion information to animate any person in the image. We evaluate DabFusion's performance using the AIST++ dataset focusing on video quality audio-video synchronization and motion-music alignment. We propose a 2D Motion-Music Alignment Score (2D-MM Align) which builds on the Beat Alignment Score to more effectively evaluate motion-music alignment for this new task. Experiments show that our DabFusion establishes a solid baseline for this innovative task. Video results can be found on our project page: https://DabFusion.github.io.