GenAI4ELab/papercli-papers huggingface dataset:
Achieving Byzantine-Resilient Federated Learning via Layer-Adaptive Sparsified Model Aggregation
Authors: Jiahao Xu; Zikai Zhang; Rui Hu
Venue: WACV 2025 (main)

Abstract: Federated Learning (FL) enables multiple clients to train a model collaboratively without sharing their local data. Yet the FL system is vulnerable to well-designed Byzantine attacks which aim to disrupt the model training process by uploading malicious model updates. Existing robust aggregation rule-based defense methods overlook the diversity of magnitude and direction across different layers of the model updates resulting in limited robustness performance particularly in non-IID settings. To address these challenges we propose the Layer-Adaptive Sparsified Model Aggregation (LASA) approach which combines pre-aggregation sparsification with layer-wise adaptive aggregation to improve robustness. Specifically LASA includes a pre-aggregation sparsification module that sparsifies updates from each client before aggregation reducing the impact of malicious parameters and minimizing the interference from less important parameters for the subsequent filtering process. Based on sparsified updates a layer-wise adaptive filter then adaptively selects benign layers using both magnitude and direction metrics across all clients for aggregation. We provide a detailed theoretical robustness analysis of LASA and a resilience analysis of the FL integrated with LASA. Extensive experiments are conducted on various IID and non-IID datasets. The numerical results demonstrate the effectiveness of LASA. Code is available at https://github.com/JiiahaoXU/LASA.
Disentangling Spatio-Temporal Knowledge for Weakly Supervised Object Detection and Segmentation in Surgical Video
Authors: Guiqiu Liao; Matjaz Jogan; Sai Koushik; Eric Eaton; Daniel A. Hashimoto
Venue: WACV 2025 (main)

Abstract: Weakly supervised video object segmentation (WSVOS) enables the identification of segmentation maps without requiring extensive annotations of object masks relying instead on coarse video labels indicating object presence. Weakly supervised semantic segmentation of objects in surgical videos is however more challenging due to a complex interaction of multiple transient objects such as surgical tools moving in and out of the surgical field. In this scenario state-of-the-art WSVOS methods struggle to learn accurate segmentation maps. We address this problem by introducing ViDeo Spatio-Temporal disentanglement Networks (VDST-Net) a framework to disentangle complex spatiotemporal object interactions using semi-decoupled knowledge distillation to predict high-quality class activation maps (CAMs). A teacher network is designed to help a temporal-reasoning student network resolve activation conflicts as the student leverages temporal dependencies when specifics about object location and timing in the video are not provided. We demonstrate the efficacy of our framework on a challenging surgical video dataset where objects are on average present in less than 60% of annotated frames and compare our method to state-of-the-art methods on surgical data and on a public dataset commonly used to benchmark WSVOS. Our method outperforms state-of-the-art techniques and generates accurate segmentation masks under video-level weak supervision. Our code is available at: https://github.com/PCASOlab/VDST-net.
Calib3D: Calibrating Model Preferences for Reliable 3D Scene Understanding
Authors: Lingdong Kong; Xiang Xu; Jun Cen; Wenwei Zhang; Liang Pan; Kai Chen; Ziwei Liu
Venue: WACV 2025 (main)

Abstract: Safety-critical 3D scene understanding tasks necessitate not only accurate but also confident predictions from 3D perception models. This study introduces Calib3D a pioneering effort to benchmark and scrutinize the reliability of 3D scene understanding models from an uncertainty estimation viewpoint. We comprehensively evaluate 28 state-of-the-art models across 10 diverse 3D datasets uncovering insightful phenomena that cope with both the aleatoric and epistemic uncertainties in 3D scene understanding. We discover that despite achieving impressive levels of accuracy existing models frequently fail to provide reliable uncertainty estimates - a pitfall that critically undermines their applicability in safety-sensitive contexts. Through extensive analysis of key factors such as network capacity LiDAR representations rasterization resolutions and 3D data augmentation techniques we correlate these aspects directly with the model calibration efficacy. Furthermore we introduce DeptS a novel depth-aware scaling approach aimed at enhancing 3D model calibration. Extensive experiments across a wide range of configurations validate the superiority of our method. We hope this work could serve as a cornerstone for fostering reliable 3D scene understanding. Code and benchmark toolkit are publicly available.
TACLE: Task and Class-Aware Exemplar-Free Semi-Supervised Class Incremental Learning
Authors: Jayateja Kalla; Rohit Kumar; Soma Biswas
Venue: WACV 2025 (main)

Abstract: We propose a novel TACLE (TAsk and CLass-awarE) framework for the relatively unexplored and challenging problem of exemplar-free semi-supervised class incremental learning. In this scenario at each new task the model has to learn new classes from both (few) labeled and unlabeled data without access to exemplars from previous classes. In addition to leveraging the capabilities of pre-trained models TACLE proposes a novel task-adaptive threshold thereby maximizing the utilization of the available unlabeled data as incremental learning progresses. Additionally to enhance the performance of the under-represented classes within each task we propose a class-aware weighted cross-entropy loss. We also exploit the unlabeled data for classifier alignment which further enhances the model performance. Extensive experiments on benchmark datasets namely CIFAR10 CIFAR100 and ImageNet-Subset100 demonstrate the effectiveness of the proposed TACLE framework. We further showcase its effectiveness when the unlabeled data is imbalanced and also for the extreme case of one labeled example per class. code: https://github.com/rokmr/TACLE
Background-Aware Moment Detection for Video Moment Retrieval
Authors: Minjoon Jung; Youwon Jang; Seongho Choi; Joochan Kim; Jin-Hwa Kim; Byoung-Tak Zhang
Venue: WACV 2025 (main)

Abstract: Video moment retrieval (VMR) identifies a specific moment in an untrimmed video for a given natural language query. This task is prone to suffer the weak alignment problem innate in video datasets. Due to the ambiguity a query does not fully cover the relevant details of the corresponding moment or the moment may contain misaligned and irrelevant frames potentially limiting further performance gains. To tackle this problem we propose a background-aware moment detection transformer (BM-DETR). Our model adopts a contrastive approach carefully utilizing the negative queries matched to other moments in the video. Specifically our model learns to predict the target moment from the joint probability of each frame given the positive query and the complement of negative queries. This leads to effective use of the surrounding background improving moment sensitivity and enhancing overall alignments in videos. Extensive experiments on four benchmarks demonstrate the effectiveness of our approach. Our code is available at https://github.com/minjoong507/BM-DETR.
MFNeRF: Memory Efficient NeRF with Mixed-Feature Hash Table
Authors: Yongjae Lee; Li Yang; Deliang Fan
Venue: WACV 2025 (main)

Abstract: Recently neural radiance fields (NeRFs) have shown remarkable performance in generating photorealistic novel views in 3D modeling. The traditional NeRF typically requires extensive training and long rendering times inspiring many recent works to utilize efficient data structures such as feature grids to ease the computational burden of multilayer perceptron networks. However Those approaches require storing features in dense grids that demands a substantial amount of memory space resulting in a notable memory bottleneck. Consequently it leads to a significant increase in training time. To address this issue in this work we propose MFNeRF a memory-efficient NeRF framework that employs a mixed-feature hash table to improve memory efficiency and reduce training time while maintaining reconstruction quality. Specifically we first design a mixed-feature hash encoding method to adaptively mix parts of multi-level feature grids and map them to a single hash table. Following that in order to obtain the correct index of a grid point we further develop an index transformation method that transforms indices of an arbitrary-level grid to those of a canonical grid. Extensive benchmarking against the state-of-the-art methods including InstantNGP TensoRF and DVGO indicates that our MFNeRF achieves faster training and rendering times on the same GPU hardware with a significantly smaller memory while maintaining similar or even higher reconstruction quality. Compared to the InstantNGP-Big model our method could achieve 89% improvement in the figure of merit defined in terms of PSNR*FPS/MB.
PETALface: Parameter Efficient Transfer Learning for Low-Resolution Face Recognition
Authors: Kartik Narayan; Nithin Gopalakrishnan Nair; Jennifer Xu; Rama Chellappa; Vishal M. Patel
Venue: WACV 2025 (main)

Abstract: Pre-training on large-scale datasets and utilizing margin-based loss functions have been highly successful in training models for high-resolution face recognition. However these models struggle with low-resolution face datasets in which the faces lack the facial attributes necessary for distinguishing different faces. Full fine-tuning on low-resolution datasets a naive method for adapting the model yields inferior performance due to catastrophic forgetting of pre-trained knowledge. Additionally the domain difference between high-resolution (HR) gallery images and low-resolution (LR) probe images in low resolution datasets leads to poor convergence for a single model to adapt to both gallery and probe after fine-tuning. To this end we propose PETALface a Parameter-Efficient Transfer Learning approach for low-resolution face recognition. Through PETALface we attempt to solve both the aforementioned problems. (1) We solve catastrophic forgetting by leveraging the power of parameter efficient fine-tuning(PEFT). (2) We introduce two low-rank adaptation modules to the backbone with weights adjusted based on the input image quality to account for the difference in quality for the gallery and probe images. To the best of our knowledge PETALface is the first work leveraging the powers of PEFT for low resolution face recognition. Extensive experiments demonstrate that the proposed method outperforms full fine-tuning on low-resolution datasets while preserving performance on high-resolution and mixed-quality datasets all while using only 0.48% of the parameters.
MissionGNN: Hierarchical Multimodal GNN-Based Weakly Supervised Video Anomaly Recognition with Mission-Specific Knowledge Graph Generation
Authors: Sanggeon Yun; Ryozo Masukawa; Minhyoung Na; Mohsen Imani
Venue: WACV 2025 (main)

Abstract: In the context of escalating safety concerns across various domains the tasks of Video Anomaly Detection (VAD) and Video Anomaly Recognition (VAR) have emerged as critically important for applications in intelligent surveillance evidence investigation violence alerting etc. These tasks aimed at identifying and classifying deviations from normal behavior in video data face significant challenges due to the rarity of anomalies which leads to extremely imbalanced data and the impracticality of extensive frame-level data annotation for supervised learning. This paper introduces a novel hierarchical graph neural network (GNN) based model MissionGNN that addresses these challenges by leveraging a state-of-the-art large language model and a comprehensive knowledge graph for efficient weakly supervised learning in VAR. Our approach circumvents the limitations of previous methods by avoiding heavy gradient computations on large multimodal models and enabling fully frame-level training without fixed video segmentation. Utilizing automated mission-specific knowledge graph generation our model provides a practical and efficient solution for real-time video analysis without the constraints of previous segmentation-based or multimodal approaches. Experimental validation on benchmark datasets demonstrates our model's performance in VAD and VAR highlighting its potential to redefine the landscape of anomaly detection and recognition in video surveillance systems.
Cascaded Dual Vision Transformer for Accurate Facial Landmark Detection
Authors: Ziqiang Dang; Jianfang Li; Lin Liu
Venue: WACV 2025 (main)

Abstract: Facial landmark detection is a fundamental problem in computer vision for many downstream applications. This paper introduces a new facial landmark detector based on vision transformers which consists of two unique designs: Dual Vision Transformer (D-ViT) and Long Skip Connections (LSC). Based on the observation that the channel dimension of feature maps essentially represents the linear bases of the heatmap space we propose learning the interconnections between these linear bases to model the inherent geometric relations among landmarks via Channel-split ViT. We integrate such channel-split ViT into the standard vision transformer (i.e. spatial-split ViT) forming our Dual Vision Transformer to constitute the prediction blocks. We also suggest using long skip connections to deliver low-level image features to all prediction blocks thereby preventing useful information from being discarded by intermediate supervision. Extensive experiments are conducted to evaluate the performance of our proposal on the widely used benchmarks i.e. WFLW COFW and 300W demonstrating that our model outperforms the previous SOTAs across all three benchmarks.
QuantAttack: Exploiting Quantization Techniques to Attack Vision Transformers
Authors: Amit Baras; Alon Zolfi; Yuval Elovici; Asaf Shabtai
Venue: WACV 2025 (main)

Abstract: In recent years there has been a significant trend in deep neural networks (DNNs) particularly transformer-based models of developing ever-larger and more capable models. While they demonstrate state-of-the-art performance their growing scale requires increased computational resources (e.g. GPUs with greater memory capacity). To address this problem quantization techniques (i.e. low-bit-precision representation and matrix multiplication) have been proposed. Most quantization techniques employ a static strategy in which the model parameters are quantized either during training or inference without considering the test-time sample. In contrast dynamic quantization techniques which have become increasingly popular adapt during inference based on the input provided while maintaining full-precision performance. However their dynamic behavior and average-case performance assumption makes them vulnerable to a novel threat vector - adversarial attacks that target the model's efficiency and availability. In this paper we present QuantAttack a novel attack that targets the availability of quantized vision transformers slowing down the inference and increasing memory usage and energy consumption. The source code is available online.
RendBEV: Semantic Novel View Synthesis for Self-Supervised Bird's Eye View Segmentation
Authors: Henrique Piñeiro Monteagudo; Leonardo Taccari; Aurel Pjetri; Francesco Sambo; Samuele Salti
Venue: WACV 2025 (main)

Abstract: Bird's Eye View (BEV) semantic maps have recently garnered a lot of attention as a useful representation of the environment to tackle assisted and autonomous driving tasks. However most of the existing work focuses on the fully supervised setting training networks on large annotated datasets. In this work we present RendBEV a new method for the self-supervised training of BEV semantic segmentation networks leveraging differentiable volumetric rendering to receive supervision from semantic perspective views computed by a 2D semantic segmentation model. Our method enables zero-shot BEV semantic segmentation and already delivers competitive results in this challenging setting. When used as pretraining to then fine-tune on labeled BEV ground truth our method significantly boosts performance in low-annotation regimes and sets a new state of the art when fine-tuning on all available labels.
NeuroViG - Integrating Event Cameras for Resource-Efficient Video Grounding
Authors: Dulanga Weerakoon; Vigneshwaran Subbaraju; Joo Hwee Lim; Archan Misra
Venue: WACV 2025 (main)

Abstract: Spatio-Temporal Video Grounding (STVG) - the task of identifying the target object in the field-of-view that the language instruction refers to - is a fundamental vision-language task. Current STVG approaches typically utilize feeds from an RGB camera that is assumed to be always-on and process the video frames using complex neural network pipelines. As a result they often impose prohibitive system overheads (energy latency) on pervasive devices. To address this we propose NeuroViG with two key innovations: (a) leveraging on event streams from a low-power neuromorphic event camera sensor to perform selective triggering of the more energy-hungry RGB camera for STVG and (b) augmenting the STVG model with a lightweight Adaptive Frame Selector (AFS) that bypasses complex transformer-based operations for a majority of video frames thereby enabling its execution on a pervasive Jetson AGX device. We have also introduced modifications to the neural network processing pipeline such that the system can offer tunable tradeoffs between accuracy and energy/latency. Our proposed NeuroViG system allows us to reduce the STVG energy overhead and latency by   4x and   3.8x respectively for less than 1% loss in accuracy.
Aggregated Attributions for Explanatory Analysis of 3D Segmentation Models
Authors: Maciej Chrabaszcz; Hubert Baniecki; Piotr Komorowski; Szymon Plotka; Przemyslaw Biecek
Venue: WACV 2025 (main)

Abstract: Analysis of 3D segmentation models especially in the context of medical imaging is often limited to segmentation performance metrics that overlook the crucial aspect of explainability and bias. Currently effectively explaining these models with saliency maps is challenging due to the high dimensions of input images multiplied by the ever-growing number of segmented class labels. To this end we introduce Agg^2Exp a methodology for aggregating fine-grained voxel attributions of the segmentation model's predictions. Unlike classical explanation methods that primarily focus on the local feature attribution Agg^2Exp enables a more comprehensive global view on the importance of predicted segments in 3D images. Our benchmarking experiments show that gradient-based voxel attributions are more faithful to the model's predictions than perturbation-based explanations. As a concrete use-case we apply Agg^2Exp to discover knowledge acquired by the Swin UNEt TRansformer model trained on the TotalSegmentator v2 dataset for segmenting anatomical structures in computed tomography medical images. Agg^2Exp facilitates the explanatory analysis of large segmentation models beyond their predictive performance. The source code is publicly available at https://github.com/MI2DataLab/agg2exp.
GANESH: Generalizable NeRF for Lensless Imaging
Authors: Rakesh Raj Madhavan; Akshat Kaimal; Badhrinarayanan K.V; Vinayak Gupta; Rohit Choudhary; Chandrakala Shanmuganathan; Kaushik Mitra
Venue: WACV 2025 (main)

Abstract: Lensless imaging offers a significant opportunity to develop ultra-compact cameras by removing the conventional bulky lens system. However without a focusing element the sensor's output is no longer a direct image but a complex multiplexed scene representation. Traditional methods have attempted to address this challenge by employing learnable inversions and refinement models but these methods are primarily designed for 2D reconstruction and do not generalize well to 3D reconstruction. We introduce GANESH a novel framework designed to enable simultaneous refinement and novel view synthesis from multi-view lensless images. Unlike existing methods that require scene-specific training our approach supports on-the-fly inference without retraining on each scene. Moreover our framework allows us to tune our model to specific scenes enhancing the rendering and refinement quality. To facilitate research in this area we also present the first multi-view lens- less dataset LenslessScenes. Extensive experiments demonstrate that our method outperforms current approaches in reconstruction accuracy and refinement quality. Code and video results are available here.
Looking at Model Debiasing through the Lens of Anomaly Detection
Authors: Vito Paolo Pastore; Massimiliano Ciranni; Davide Marinelli; Francesca Odone; Vittorio Murino
Venue: WACV 2025 (main)

Abstract: Deep neural networks are likely to learn unintended spurious correlations between training data and labels when dealing with biased data potentially limiting the generalization to unseen samples not presenting the same bias. In this context model debiasing approaches can be devised aiming at reducing the model's dependency on such unwanted correlations either leveraging the knowledge of bias information or not. In this work we focus on the latter and more realistic scenario showing the importance of accurately predicting the bias-conflicting and bias-aligned samples to obtain compelling performance in bias mitigation. On this ground we propose to conceive the problem of model bias from an out-of-distribution perspective introducing a new bias identification method based on anomaly detection. We claim that when data is mostly biased bias-conflicting samples can be regarded as outliers with respect to the bias-aligned distribution in the feature space of a biased model thus allowing for precisely detecting them with an anomaly detection method. Coupling the proposed bias identification approach with bias-conflicting data upsampling and augmentation in a two-step strategy we reach state-of-the-art performance on synthetic and real benchmark datasets. Ultimately our proposed approach shows that the data bias issue does not necessarily require complex debiasing methods given that an accurate bias identification procedure is defined. Source code is available at https://github.com/Malga-Vision/MoDAD
On Neural BRDFs: A Thorough Comparison of State-of-the-Art Approaches
Authors: Florian Hofherr; Bjoern Haefner; Daniel Cremers
Venue: WACV 2025 (main)

Abstract: The bidirectional reflectance distribution function (BRDF) is an essential tool to capture the complex interaction of light and matter. Recently several works have employed neural methods for BRDF modeling following various strategies ranging from utilizing existing parametric models to purely neural parametrizations. While all methods yield impressive results a comprehensive comparison of the different approaches is missing in the literature. In this work we present a thorough evaluation of several approaches including results for qualitative and quantitative reconstruction quality and an analysis of reciprocity and energy conservation. Moreover we propose two extensions that can be added to existing approaches: A novel additive combination strategy for neural BRDFs that split the reflectance into a diffuse and a specular part and an input mapping that ensures reciprocity exactly by construction while previous approaches only ensure it by soft constraints.