GenAI4ELab/papercli-papers huggingface dataset:
Shapley Consensus Deep Learning for Ensemble Pruning
Authors: Youcef Djenouri; Ahmed Nabil Belbachir; Asma Belhadi; Nassim Belmecheri; Tomasz Michalak
Venue: WACV 2025 (main)

Abstract: This paper targets a new foundation for designing general-purpose learning systems by establishing a consensus method that facilitates self-adaptation and flexibility to deal with different learning tasks and different data distribution. We present the Shapely Consensus Deep Learning (SCDL) as a consensus method for general-purpose intelligence without the help of a domain expert. SCDL is two-level based learning process. In the first level several deep learning models have been trained for each historical observation. The Shapley Value is determined to compute the contribution of each subset of models in the training. The models are pruned according to their contribution in the learning process. In the second level the loss information of each data distribution is saved in the knowledge base. Both levels are explored to prune the models for each new observation. We present the evaluation of the generality of SCDL using different datasets with different shapes and complexities. The results reveal the effectiveness of SCDL for weakly classification. Concretely SCDL achieved 90% of AUC with less than 86% for the baseline solutions.
Personalized Mixture of Experts for Multi-Site Medical Image Segmentation
Authors: Md Motiur Rahman; Mohamed Trabelsi; Huseyin Uzunalioglu; Aidan Boyd
Venue: WACV 2025 (main)

Abstract: The sharing of sensitive medical data among institutions presents a significant challenge due to strict privacy regulations the need for robust de-identification processes and the ethical imperative to protect patient confidentiality. Federated Learning (FL) addresses these challenges by enabling institutions to collaboratively train AI models on decentralized data thereby enhancing privacy and security without directly sharing sensitive patient information. However FL requires complex synchronization implementations has costly communication overheads and may fail to capture data heterogeneity across institutions. In this work we propose Personalized Mixture of Local Experts (P-MoLE) a Personalized Federated Learning (PFL) approach that effectively combines predictions from multiple locally trained models in a sample-specific manner. Leveraging both the individuality of each local model and variation across the ensemble P-MoLE learns the profile of each institution's local model and strategically weighs their prediction's contributions to the final segmentation. This approach harnesses the heterogeneity of each institution's unique data to increase the generalization capabilities across all institutions. By each institution sharing only the final models trained locally on the sensitive data no private patient data is exposed and the need for expensive communication infrastructure is removed. Results across two popular multi-institutional medical imaging datasets show P-MoLE achieves state-of-the-art performance without the extensive cooperative effort requirement of previous works. Additionally ablation study results show that P-MoLE is flexible to the number of local models in the ensemble increasing performance over the local models alone in each case.
Learning Semantic Part-Based Graph Structure for 3D Point Cloud Domain Generalization
Authors: G Ujwal Sai; Arkadipta De; Vartika Sengar; Anuj Rathore; Daksh Thapar; Manohar Kaul
Venue: WACV 2025 (main)

Abstract: In 3D data analysis point clouds provide detailed geometric insights for applications like computer vision and geospatial analysis. However their irregularity and diversity make classification challenging especially in domain generalization where models must generalize to new data distributions. Our research introduces a novel 3D Domain Generalization (3DDG) method using Unsupervised Part Decomposition (UPD) and Graph Structure Induction (GSI). The UPD module employs spectral clustering and a modified Shannon entropy method to segment point clouds into meaningful parts. The GSI module constructs a graph of these parts' spatial relationships processed by a Graph Neural Network (GNN) to understand complex geometries. Our approach enhances part-based analysis improving classification accuracy on the PointDA-10 and GraspNetPC-10 datasets by 1.25% and 2.6% respectively. These results highlight our advancements in 3D domain generalization enabling more robust classification models for diverse point cloud data.
MENTOR: Human Perception-Guided Pretraining for Increased Generalization
Authors: Colton R. Crum; Adam Czajka
Venue: WACV 2025 (main)

Abstract: Leveraging human perception into training of convolutional neural networks (CNN) has boosted generalization capabilities of such models in open-set recognition tasks. One of the active research questions is where (in the model architecture or training pipeline) and how to efficiently incorporate always-limited human perceptual data into training strategies of models. In this paper we introduce MENTOR (huMan pErceptioN-guided preTraining fOr increased geneRalization) which addresses this question through two unique rounds of training CNNs tasked with open-set anomaly detection. First we train an autoencoder to learn human saliency maps given an input image without any class labels. The autoencoder is thus tasked with discovering domain-specific salient features which mimic human perception. Second we remove the decoder part add a classification layer on top of the encoder and train this new model conventionally now using class labels. We show that MENTOR successfully raises the generalization performance across three different CNN backbones in a variety of anomaly detection tasks (demonstrated for detection of unknown iris presentation attacks synthetically-generated faces and anomalies in chest X-ray images) compared to traditional pretraining methods (e.g. sourcing the weights from ImageNet) and as well as state-of-the-art methods that incorporate human perception guidance into training. In addition we demonstrate that MENTOR can be flexibly applied to existing human perception-guided methods and subsequently increasing their generalization with no architectural modifications.
Multi-Modal Large Language Model with RAG Strategies in Soccer Commentary Generation
Authors: Xiang Li; Yangfan He; Shuaishuai Zu; Zhengyang Li; Tianyu Shi; Yiting Xie; Kevin Zhang
Venue: WACV 2025 (main)

Abstract: As a globally celebrated sport soccer has seen its appeal greatly amplified by engaging and vivid commentary. Recently Multi-Modal Large Language Models (MLLMs) have attracted attention in generating soccer commentaries due to their remarkable capacities of understanding different modalities of the input videos. Most of these methods have shown that the use of multiple modalities can enhance the commentary quality which includes video audio and structured meta-data. However delivering precise and rich commentary requires the ability to accurately discern subtle differences in similar backgrounds events and players. This presents a significant challenge for existing MLLMs. So we propose SoccerComment a framework for generating soccer commentary that integrates MLLMs with Retrieval-Augmented Generation (RAG) strategies. This framework enhances inference efficiency and reduces the need for continuous training through a multi-modal clustering memory unit and retrieval-augmented in-context learning mechanisms ultimately improving the accuracy and diversity of the commentary. Based on similar retrieved scenarios SoccerComment demonstrates outstanding zero-shot performance offering a new direction and scalable solution for future research in soccer commentary generation.
UAL-Bench: The First Comprehensive Unusual Activity Localization Benchmark
Authors: Hasnat Md Abdullah; Tian Liu; Kangda Wei; Shu Kong; Ruihong Huang
Venue: WACV 2025 (main)

Abstract: Localizing unusual activities in videos such as abnormal behaviors or traffic incidents holds practical significance. However pretrained foundation models struggle with localizing diverse unusual events likely because of their insufficient representation in the models' pretraining datasets. To explore foundation models' capability in localizing unusual activities we introduce UAL-Bench a comprehensive benchmark for unusual activity localization featuring three video datasets (UAG-OOPS UAG-SSBD and UAG-FunQA) and an instruction-tuning dataset (OOPS-UAG-Instruct) to improve model capabilities. We also introduce a new metric R@1 TD <= p as an auxiliary metric to reasonably consider detections as true positive if their starting and ending timestamps are within a threshold. On UAL-Bench we evaluate three approaches: Video-Language Models (Vid-LLMs) instruction-tuned Vid-LLMs and a novel integration of Vision-Language Models and Large Language Models (VLM-LLM). Our results show the VLM-LLM approach excels in localizing short-span unusual events and predicting their onset (start time) more accurately than Vid-LLMs. Our findings highlight the challenges posed by long-duration videos particularly in autism diagnosis scenarios and the need for further advancements in localization techniques. Our work not only provides a benchmark for unusual activity localization but also outlines the key challenges for existing foundation models suggesting future research directions on this important task.
FOR: Finetuning for Object Level Open Vocabulary Image Retrieval
Authors: Hila Levi; Guy Heller; Dan Levi
Venue: WACV 2025 (main)

Abstract: As working with large datasets becomes standard the task of accurately retrieving images containing objects of interest by an open set textual query gains practical importance. The current leading approach utilizes a pre-trained CLIP model without any adaptation to the target domain balancing accuracy and efficiency through additional post-processing. In this work we propose FOR: Finetuning for Object-centric Open-vocabulary Image Retrieval which allows finetuning on a target dataset using closed-set labels while keeping the visual-language association crucial for open vocabulary retrieval. FOR is based on two design elements: a specialized decoder variant of the CLIP head customized for the intended task and its coupling within a multi-objective training framework. Together these design choices result in a significant increase in accuracy show-casing improvements of up to 8 mAP@50 points over SoTA across three datasets. Additionally we demonstrate that FOR is also effective in a semi-supervised setting achieving impressive results even when only a small portion of the dataset is labeled.
FRAUD-Net: Fraud News Detection using Sample Uncertainty & Domain Aware Generalized Network
Authors: Devendra Patel; Vikas Verma; Shreyas Kumar Tah; Shwetabh Biswas; Soma Biswas
Venue: WACV 2025 (main)

Abstract: Due to the widespread impact of social media efficiently detecting out-of-context misinformation where a real image is paired with a fake caption has become imperative. Towards this goal we propose a novel framework FRAUD-Net which incorporates several unexplored aspects of this task in the model formulation. Keeping in mind that the image and textual evidences collected using the input image-text pair by web search play a crucial role in this task we build a generalized model which can handle missing or variable number of evidences as expected in real-world scenarios. We achieve this by effectively utilizing the common semantic latent space of Visual Language Models and transformer attention blocks. In addition we observe that explicit domain information not only allows the model to learn better but also dynamically adjusts to the diverse domains (like politics healthcare etc.) during testing. We also propose a mechanism to handle noisy training data by analyzing prediction consistency which helps in model training. Extensive experiments on the large-scale NewsClippings and Verite benchmark datasets showcase the effectiveness of the proposed framework compared to the state-of-the-art techniques for this challenging task.
All-in-One Image Compression and Restoration
Authors: Huimin Zeng; Jiacheng Li; Ziqiang Zheng; Zhiwei Xiong
Venue: WACV 2025 (main)

Abstract: Visual images corrupted by various types and levels of degradations are commonly encountered in practical image compression. However most existing image compression methods are tailored for clean images therefore struggling to achieve satisfying results on these images. Joint compression and restoration methods typically focus on a single type of degradation and fail to address a variety of degradations in practice. To this end we propose a unified framework for all-in-one image compression and restoration which incorporates the image restoration capability against various degradations into the process of image compression. The key challenges involve distinguishing authentic image content from degradations and flexibly eliminating various degradations without prior knowledge. Specifically the proposed framework approaches these challenges from two perspectives: ie content information aggregation and degradation representation aggregation. Extensive experiments demonstrate the following merits of our model: 1) superior rate-distortion (RD) performance on various degraded inputs while preserving the performance on clean data; 2) strong generalization ability to real-world and unseen scenarios; 3) higher computing efficiency over compared methods. Our code is available at https://github.com/ZeldaM1/All-in-one.
PLReMix: Combating Noisy Labels with Pseudo-Label Relaxed Contrastive Representation Learning
Authors: Xiaoyu Liu; Beitong Zhou; Zuogong Yue; Cheng Cheng
Venue: WACV 2025 (main)

Abstract: Recently the usage of Contrastive Representation Learning (CRL) as a pre-training technique improves the performance of learning with noisy labels (LNL) methods. However instead of pre-training when trivially combining CRL loss with LNL methods as an end-to-end framework the empirical experiments show severe degeneration of the performance. We verify through experiments that this issue is caused by optimization conflicts of losses and propose an end-to-end PLReMix framework by introducing a Pseudo-Label Relaxed (PLR) contrastive loss. This PLR loss constructs a reliable negative set of each sample by filtering out its inappropriate negative pairs alleviating the loss conflicts by trivially combining these losses. The proposed PLR loss is pluggable and we have integrated it into other LNL methods observing their improved performance. Furthermore a two-dimensional Gaussian Mixture Model is adopted to distinguish clean and noisy samples by leveraging semantic information and model outputs simultaneously. Experiments on multiple benchmark datasets demonstrate the effectiveness of the proposed method. Codes will be available.
Data Generation for Hardware-Friendly Post-Training Quantization
Authors: Lior Dikstein; Ariel Lapid; Arnon Netzer; Hai Victor Habi
Venue: WACV 2025 (main)

Abstract: Zero-shot quantization (ZSQ) using synthetic data is a key approach for post-training quantization (PTQ) under privacy and security constraints. However existing data generation methods often struggle to effectively generate data suitable for hardware-friendly quantization where all model layers are quantized. We analyze existing data generation methods based on batch normalization (BN) matching and identify several gaps between synthetic and real data: 1) Current generation algorithms do not optimize the entire synthetic dataset simultaneously; 2) Data augmentations applied during training are often overlooked; and 3) A distribution shift occurs in the final model layers due to the absence of BN in those layers. These gaps negatively impact ZSQ performance particularly in hardware-friendly quantization scenarios. In this work we propose Data Generation for Hardware-Friendly Quantization (DGH) a novel method that addresses these gaps. DGH jointly optimizes all generated images regardless of the image set size or GPU memory constraints. To address data augmentation mismatches DGH includes a preprocessing stage that mimics the augmentation process and enhances image quality by incorporating natural image priors. Finally we propose a new distribution-stretching loss that aligns the support of the feature map distribution between real and synthetic data. This loss is applied to the model's output and can be adapted to various tasks. DGH demonstrates significant improvements in quantization performance across multiple tasks achieving up to a 30% increase in accuracy for hardware-friendly ZSQ in both classification and object detection often performing on par with real data.
An Investigation on LLMs' Visual Understanding Ability using SVG for Image-Text Bridging
Authors: Mu Cai; Zeyi Huang; Yuheng Li; Utkarsh Ojha; Haohan Wang; Yong Jae Lee
Venue: WACV 2025 (main)

Abstract: Large language models (LLMs) have made significant advancements in natural language understanding. However through that enormous semantic representation that the LLM has learnt is it somehow possible for it to understand images as well? This work investigates this question. To enable the LLM to process images we convert them into a representation given by Scalable Vector Graphics (SVG). To study what the LLM can do with this XML-based textual description of images we test the LLM on three broad computer vision tasks: (i) visual reasoning and question answering (ii) image classification under distribution shift few-shot learning and (iii) generating new images using visual prompting. Even though we do not naturally associate LLMs with any visual understanding capabilities our results indicate that the LLM can often do a decent job in many of these tasks potentially opening new avenues for research into LLMs' ability to understand image data. Our code data and models can be found here https://github.com/mu-cai/svg-llm.
RiemStega: Covariance-Based Loss for Print-Proof Transmission of Data in Images
Authors: Aniana Cruz; Guilherme Schardong; Luiz Schirmer; João Marcos; Farhad Shadmand; Nuno Gonçalves
Venue: WACV 2025 (main)

Abstract: Covariance matrices outperform first-order features in many tasks attracting considerable attention from the computer vision research community. Covariance matrices encode second-order statistics between features at the same time it is robust to noise. Based on this we propose representing images by covariance matrices and defining a loss function that measures the distance between them through the Riemannian distance. Motivated by the robustness and invariance properties of the affine invariant Riemannian metric the proposed method was validated in printer-proof data transmission which is a challenging task due to the trade-off between image quality and message recovery capabilities after printing and digitization procedures. The effectiveness of this approach was systematically assessed using MS COCO and IMM Face datasets. The results demonstrated that the proposed approach outperforms conventional methods that use Euclidean distance generating encoded images with better quality and achieving higher recovery accuracy in printed images. Additionally a broader application of the proposed loss was successfully tested in image generation tasks using generative adversarial networks (GANs).
Rethinking Low-Rank Adaptation in Vision: Exploring Head-Level Responsiveness Across Diverse Tasks
Authors: Yibo Zhong; Yao Zhou
Venue: WACV 2025 (main)

Abstract: Low-rank adaptation (LoRA) has shifted the paradigm of adapting pre-trained Vision Transformers (ViT) achieving great efficiency by updating only a subset of tailored parameters to approximate weight updates. However the multi-head design of the self-attention mechanism with the heads working in parallel in the computation flow exhibiting similar visual patterns and requiring update over all of them incurs unnecessary storage and computational overhead. In this paper we propose Head-level responsiveness tuning for low-rank adaptation (Heart-LoRA). The proposed method explores redundancy among the heads and selectively activates task-responsive heads thus enabling fine-grained head-level tuning. Additionally given the different responsiveness of heads to diverse visual tasks our proposed method dynamically activates a subset of the approximated heads that are tailored to the current task. Experimental results show that Heart-LoRA yields superior performance over state-of-the-art PETL approaches on visual adaptation benchmark datasets.
Dynamic Adapter Tuning for Long-Tailed Class-Incremental Learning
Authors: Yanan Gu; Muli Yang; Xu Yang; Kun Wei; Hongyuan Zhu; Gabriel James Goenawan; Cheng Deng
Venue: WACV 2025 (main)

Abstract: Long-tailed class-incremental learning (LT-CIL) aims to learn new classes continuously from a long-tailed data stream while simultaneously dealing with challenges such as imbalanced learning of tail classes and catastrophic forgetting. To address these challenges most existing methods employ a two-stage strategy by initializing model training from scratch with further balanced knowledge driven calibration. This strategy faces challenges in deriving discriminative features from cold-started backbones for the long-tailed distribution of data consequently leading to relatively diminished performance. In this paper with the powerful feature extraction capability of pre-trained foundation models we have achieved a one-stage approach that delivers superior performance. Specifically we propose Dynamic Adapter Tuning (DAT) which employs a dynamic adapter cache mechanism to adapt a pre-trained model to learn tasks sequentially. The adapter in the cache is either dynamically selected or created according to task similarity and further compactified with the new task's adapter to mitigate cross-task and cross-class gaps in LT-CIL significantly alleviating catastrophic forgetting and imbalance learning issues respectively. With extensive experimental validation our method consistently achieves state-of-the-art performance under the challenging LT-CIL setting.
Distilling Aggregated Knowledge for Weakly-Supervised Video Anomaly Detection
Authors: Jash Dalvi; Ali Dabouei; Gunjan Dhanuka; Min Xu
Venue: WACV 2025 (main)

Abstract: Video anomaly detection aims to develop automated models capable of identifying abnormal events in surveillance videos. The benchmark setup for this task is extremely challenging due to: i) the limited size of the training sets ii) weak supervision provided in terms of video-level labels and iii) intrinsic class imbalance induced by the scarcity of abnormal events. In this work we show that distilling knowledge from aggregated representations of multiple backbones into a single-backbone Student model achieves state-of-the-art performance. In particular we develop a bi-level distillation approach along with a novel disentangled cross-attention-based feature aggregation network. Our proposed approach DAKD (Distilling Aggregated Knowledge with Disentangled Attention) demonstrates superior performance compared to existing methods across multiple benchmark datasets. Notably we achieve significant improvements of 1.36% 0.78% and 7.02% on the UCF-Crime ShanghaiTech and XD-Violence datasets respectively.