GenAI4ELab/papercli-papers huggingface dataset:
TPP-Gaze: Modelling Gaze Dynamics in Space and Time with Neural Temporal Point Processes
Authors: Alessandro D'Amelio; Giuseppe Cartella; Vittorio Cuculo; Manuele Lucchi; Marcella Cornia; Rita Cucchiara; Giuseppe Boccignone
Venue: WACV 2025 (main)

Abstract: Attention guides our gaze to fixate the proper location of the scene and holds it in that location for the deserved amount of time given current processing demands before shifting to the next one. As such gaze deployment crucially is a temporal process. Existing computational models have made significant strides in predicting spatial aspects of observer's visual scanpaths (where to look) while often putting on the background the temporal facet of attention dynamics (when). In this paper we present TPP-Gaze a novel and principled approach to model scanpath dynamics based on Neural Temporal Point Process (TPP) that jointly learns the temporal dynamics of fixations position and duration integrating deep learning methodologies with point process theory. We conduct extensive experiments across five publicly available datasets. Our results show the overall superior performance of the proposed model compared to state-of-the-art approaches. Source code and trained models are publicly available at: https://github.com/phuselab/tppgaze.
Towards Unsupervised Blind Face Restoration using Diffusion Prior
Authors: Tianshu Kuai; Sina Honari; Igor Gilitschenski; Alex Levinshtein
Venue: WACV 2025 (main)

Abstract: Blind face restoration methods have shown remarkable performance particularly when trained on large-scale synthetic datasets with supervised learning. These datasets are often generated by simulating low-quality face images with a handcrafted image degradation pipeline. The models trained on such synthetic degradations however cannot deal with inputs of unseen degradations. In this paper we address this issue by using only a set of input images with unknown degradations and without ground truth targets to fine-tune a restoration model that learns to map them to clean and contextually consistent outputs. We utilize a pre-trained diffusion model as a generative prior through which we generate high quality images from the natural image distribution while maintaining the input image content through consistency constraints. These generated images are then used as pseudo targets to fine-tune a pre-trained restoration model. Unlike many recent approaches that employ diffusion models at test time we only do so during training and thus maintain an efficient inference-time performance. Extensive experiments show that the proposed approach can consistently improve the perceptual quality of pre-trained blind face restoration models while maintaining great consistency with the input contents. Our best model also achieves the state-of-the-art results on both synthetic and real-world datasets.
Long-Term Ad Memorability: Understanding & Generating Memorable Ads
Authors: Harini Si; Somesh Singh; Yaman Kumar Singla; Aanisha Bhattacharyya; Veeky Baths; Changyou Chen; Rajiv Ratn Shah; Balaji Krishnamurthy
Venue: WACV 2025 (main)

Abstract: Despite the importance of long-term memory in marketing and brand building until now there has been no large-scale study on the memorability of ads. All previous memorability studies have been conducted on short-term recall on specific content types like action videos. On the other hand long-term memorability is crucial for advertising industry and ads are almost always highly multimodal. Therefore we release the first memorability dataset LAMBDA consisting of 1749 participants and 2205 ads covering 276 brands. Running statistical tests over different participant subpopulations and ad types we find many interesting insights into what makes an ad memorable e.g. fast-moving ads are more memorable than those with slower scenes; people who use ad-blockers remember a lower number of ads than those who don't. Next we present a model Henry to predict the memorability of a content. Henry achieves state-of-the-art performance across all prominent literature memorability datasets. It shows strong generalization performance with better results in 0-shot on unseen datasets. Finally with the intent of memorable ad generation we present a scalable method to build a high-quality memorable ad generation model by leveraging automatically annotated data. Our approach SEED (Self rEwarding mEmorability Modeling) starts with a language model trained on LAMBDA as seed data and progressively trains an LLM to generate more memorable ads. We show that the generated advertisements have 44% higher memorability scores than the original ads. We release this large-scale ad dataset UltraLAMBDA consisting of 5 million ads. Our code and the datasets LAMBDA and UltraLAMBDA are open-sourced at https://behavior-in-the-wild.github.io/memorability.
Mamba-ST: State Space Model for Efficient Style Transfer
Authors: Filippo Botti; Alex Ergasti; Leonardo Rossi; Tomaso Fontanini; Claudio Ferrari; Massimo Bertozzi; Andrea Prati
Venue: WACV 2025 (main)

Abstract: The goal of style transfer is given a content image and a style source generating a new image preserving the content but with the artistic representation of the style source. Most of the state-of-the-art architectures use transformers or diffusion-based models to perform this task despite the heavy computational burden that they require. In particular transformers use self- and cross-attention layers which have large memory footprint while diffusion models require high inference time. To overcome the above this paper explores a novel design of Mamba an emergent State-Space Model (SSM) called Mamba-ST to perform style transfer. To do so we adapt Mamba linear equation to simulate the behavior of cross-attention layers which are able to combine two separate embeddings into a single output but drastically reducing memory usage and time complexity. We modified the Mamba's inner equations so to accept inputs from and combine two separate data streams. To the best of our knowledge this is the first attempt to adapt the equations of SSMs to a vision task like style transfer without requiring any other module like cross-attention or custom normalization layers. An extensive set of experiments demonstrates the superiority and efficiency of our method in performing style transfer compared to transformers and diffusion models. Results show improved quality in terms of both ArtFID and FID metrics. Code is available at https://github.com/FilippoBotti/MambaST.
Through the Curved Cover: Synthesizing Cover Aberrated Scenes with Refractive Field
Authors: Liuyue Xie; Jiancong Guo; László A. Jeni; Zhiheng Jia; Mingyang Li; Yunwen Zhou; Chao Guo
Venue: WACV 2025 (main)

Abstract: Recent extended reality headsets and field robots have adopted covers to protect the front-facing cameras from environmental hazards and falls. The surface irregularities on the cover can lead to optical aberrations like blurring and non-parametric distortions. Novel view synthesis methods like NeRF and 3D Gaussian Splatting are ill-equipped to synthesize from sequences with optical aberrations. To address this challenge we introduce SynthCover to enable novel view synthesis through protective covers for downstream extended reality applications. SynthCover employs a Refractive Field that estimates the cover's geometry enabling precise analytical calculation of refracted rays. Experiments on synthetic and real-world scenes demonstrate our method's ability to accurately model scenes viewed through protective covers achieving a significant improvement in rendering quality compared to prior methods. We also show that the model can adjust well to various cover geometries with synthetic sequences captured with covers of different surface curvatures. To motivate further studies on this problem we provide the benchmarked dataset containing real and synthetic walkable scenes captured with protective cover optical aberrations.
Crossroads of Continents: Automated Artifact Extraction for Cultural Adaptation with Large Multimodal Models
Authors: Anjishnu Mukherjee; Ziwei Zhu; Antonios Anastasopoulos
Venue: WACV 2025 (main)

Abstract: We present a comprehensive three-phase study to examine (1) the cultural understanding of Large Multimodal Models (LMMs) by introducing DALLE STREET a large-scale dataset generated by DALL-E 3 and validated by humans containing 9935 images of 67 countries and 10 concept classes; (2) the underlying implicit and potentially stereotypical cultural associations with a cultural artifact extraction task; and (3) an approach to adapt cultural representation in an image based on extracted associations using a modular pipeline CULTUREADAPT. We find disparities in cultural understanding at geographic sub-region levels with both open-source (LLaVA) and closed-source (GPT-4V) models on DALLE STREET and other existing benchmarks which we try to understand using over 18000 artifacts that we identify in association to different countries. Our findings reveal a nuanced picture of the cultural competence of LMMs highlighting the need to develop culture-aware systems.
SoundSil-DS: Deep Denoising and Segmentation of Sound-Field Images with Silhouettes
Authors: Risako Tanigawa; Kenji Ishikawa; Noboru Harada; Yasuhiro Oikawa
Venue: WACV 2025 (main)

Abstract: Development of optical technology has enabled imaging of two-dimensional (2D) sound fields. This acousto-optic sensing enables understanding of the interaction between sound and objects such as reflection and diffraction. Moreover it is expected to be used an advanced measurement technology for sonars in self-driving vehicles and assistive robots. However the low sound-pressure sensitivity of the acousto-optic sensing results in high intensity of noise on images. Therefore denoising is an essential task to visualize and analyze the sound fields. In addition to denoising segmentation of sound and object silhouette is also required to analyze interactions between them. In this paper we propose sound-field-images-with-object-silhouette denoising and segmentation (SoundSil-DS) that jointly perform denoising and segmentation for sound fields and object silhouettes on a visualized image. We developed a new model based on the current state-of-the-art denoising network. We also created a dataset to train and evaluate the proposed method through acoustic simulation. The proposed method was evaluated using both simulated and measured data. We confirmed that our method can applied to experimentally measured data. These results suggest that the proposed method may improve the post-processing for sound fields such as physical model-based three-dimensional reconstruction since it can remove unwanted noise and separate sound fields and other object silhouettes. Our code is available at https://github.com/nttcslab/soundsil-ds.
OpenCapBench: A Benchmark to Bridge Pose Estimation and Biomechanics
Authors: Yoni Gozlan; Antoine Falisse; Scott Uhlrich; Anthony Gatti; Michael Black; Jennifer Hicks; Scott Delp; Akshay Chaudhari
Venue: WACV 2025 (main)

Abstract: Pose estimation has promised to impact healthcare by enabling more practical methods to quantify nuances of human movement and biomechanics. However despite the inherent connection between pose estimation and biomechanics these disciplines have largely remained disparate. For example most current pose estimation benchmarks use metrics such as Mean Per Joint Position Error Percentage of Correct Keypoints or mean Average Precision to assess performance without quantifying kinematic and physiological correctness - key aspects for biomechanics. To alleviate this challenge we develop OpenCapBench to offer an easy-to-use unified benchmark to assess common tasks in human pose estimation evaluated under physiological constraints. OpenCapBench computes consistent kinematic metrics through joints angles provided by an open-source musculoskeletal modeling software (OpenSim). Through OpenCapBench we demonstrate that current pose estimation models use keypoints that are too sparse for accurate biomechanics analysis. To mitigate this challenge we introduce SynthPose a new approach that enables finetuning of pre-trained 2D human pose models to predict an arbitrarily denser set of keypoints for accurate kinematic analysis through the use of synthetic data. Incorporating such finetuning on synthetic data of prior models leads to twofold reduced joint angle errors. Moreover OpenCapBench allows users to benchmark their own developed models on our clinically relevant cohort. Overall OpenCapBench bridges the computer vision and biomechanics communities aiming to drive simultaneous advances in both areas.
Neural Graph Map: Dense Mapping with Efficient Loop Closure Integration
Authors: Leonard Bruns; Jun Zhang; Patric Jensfelt
Venue: WACV 2025 (main)

Abstract: Neural field-based SLAM methods typically employ a single monolithic field as their scene representation. This prevents efficient incorporation of loop closure constraints and limits scalability. To address these shortcomings we propose a novel RGB-D neural mapping framework in which the scene is represented by a collection of lightweight neural fields which are dynamically anchored to the pose graph of a sparse visual SLAM system. Our approach shows the ability to integrate large-scale loop closures while requiring only minimal reintegration. Furthermore we verify the scalability of our approach by demonstrating successful building-scale mapping taking multiple loop closures into account during the optimization and show that our method outperforms existing state-of-the-art approaches on large scenes in terms of quality and runtime. Our code is available open-source at https://github.com/KTH-RPL/neural_graph_mapping.
Label Convergence: Defining an Upper Performance Bound in Object Recognition through Contradictory Annotations
Authors: David Eike Tschirschwitz; Volker Rodehorst
Venue: WACV 2025 (main)

Abstract: Annotation errors are a challenge not only during training of machine learning models but also during their evaluation. Label variations and inaccuracies in datasets often manifest as contradictory examples that deviate from established labeling conventions. Such inconsistencies when significant prevent models from achieving optimal performance on metrics such as mean Average Precision (mAP). We introduce the notion of "label convergence" to describe the highest achievable performance under the constraint of contradictory test annotations essentially defining an upper bound on model accuracy. Recognizing that noise is an inherent characteristic of all data our study analyzes five real-world datasets including the LVIS dataset to investigate the phenomenon of label convergence. We approximate that label convergence is between 62.63-67.52 mAP@[0.5:0.95:0.05] for LVIS with 95% confidence attributing these bounds to the presence of real annotation errors. With current state-of-the-art (SOTA) models at the upper end of the label convergence interval for the well-studied LVIS dataset we conclude that model capacity is sufficient to solve current object detection problems. Therefore future efforts should focus on three key aspects: (1) updating the problem specification and adjusting evaluation practices to account for unavoidable label noise (2) creating cleaner data especially test data and (3) including multi-annotated data to investigate annotation variation and make these issues visible from the outset.
A Two-Head Loss Function for Deep Average-K Classification
Authors: Camille Garcin; Maximilien Servajean; Alexis Joly; Joseph Salmon
Venue: WACV 2025 (main)

Abstract: Average-K classification is an alternative to top-K classification in which the number of labels returned varies with the ambiguity of the input image but must average to K over all the samples. A simple method to solve this task is to threshold the softmax output of a model trained with the cross-entropy loss. This approach is theoretically proven to be asymptotically consistent but it is not guaranteed to be optimal for a finite set of samples. In this paper we propose a new loss function based on a multi-label classification head in addition to the classical softmax. This second head is trained using pseudo-labels generated by thresholding the softmax head while guaranteeing that K classes are returned on average. We show that this approach allows the model to better capture ambiguities between classes and as a result to return more consistent sets of possible classes. Experiments on two datasets from the literature demonstrate that our approach outperforms the softmax baseline as well as several other loss functions more generally designed for weakly supervised multi-label classification. The gains are larger the higher the uncertainty especially for classes with few samples.
DyRoNet: Dynamic Routing and Low-Rank Adapters for Autonomous Driving Streaming Perception
Authors: Xiang Huang; Zhi-Qi Cheng; Jun-Yan He; Chenyang Li; Wangmeng Xiang; Baigui Sun
Venue: WACV 2025 (main)

Abstract: The advancement of autonomous driving systems hinges on the ability to achieve low-latency and high-accuracy perception. To address this critical need this paper introduces Dynamic Routering Network (DyRoNet) a low-rank enhanced dynamic routing framework designed for streaming perception in autonomous driving systems. DyRoNet integrates a suite of pre-trained branch networks each meticulously fine-tuned to function under distinct environmental conditions. At its core the framework offers a speed router module developed to assess and route input data to the most suitable branch for processing. This approach not only addresses the inherent limitations of conventional models in adapting to diverse driving conditions but also ensures the balance between performance and efficiency. Extensive experimental evaluations demonstrating the adaptability of DyRoNet to diverse branch selection strategies resulting in significant performance enhancements across different scenarios. This work not only establishes a new benchmark for streaming perception but also provides valuable engineering insights for future work.
PrivateEye: In-Sensor Privacy Preservation Through Optical Feature Separation
Authors: Adith Boloor; Weikai Lin; Tianrui Ma; Yu Feng; Yuhao Zhu; Xuan Zhang
Venue: WACV 2025 (main)

Abstract: We address privacy issues in applications where images captured by an edge device (camera) are sent to the cloud for inference on utility tasks such as classification. Sending raw images to the cloud exposes them to data sniffing attacks and misuse by untrusted third-party service providers beyond the user's intended tasks. We propose an encoding scheme that not only evades direct visual inspection to the images or image reconstruction but also prevents sensitive information from being ascertained. Unlike commonly used adversarial learning approaches the proposed method is two-fold: first it uses a diffractive optical neural network to spatially separate features corresponding to different tasks on the sensor plane in the optical domain. Then only the pixels corresponding to the utility task region are read. This encoding ensures that private features are never digitally stored on the edge device thereby preventing privacy leakage. The proposed method successfully reduces the privacy retrieval in binary tasks with minimal accuracy loss (  2%) of the utility task while reducing private task accuracy by   35% and defending against reconstruction attacks with SSIM score of 0.43.
Learning under Noisy Labels Spurious Points and Diverse Structures: TS40K a 3D Point Cloud Dataset of Rural Terrain and Electrical Transmission Systems
Authors: Diogo Lavado; Ricardo Santos; André Coelho; João Santos; Alessandra Micheletti; Cláudia Soares
Venue: WACV 2025 (main)

Abstract: Research in 3D scene understanding particularly in autonomous driving and indoor segmentation has made significant strides. However most available datasets focus on urban settings. We introduce TS40K a 3D point cloud dataset spanning 40000 km of electrical transmission systems in rural terrain addressing power-grid inspections to prevent outages damages and fires. TS40K offers high point density and no occlusion presenting challenges like noisy labels diverse structures and sensor noise causing spurious points. We evaluate state-of-the-art methods on 3D semantic segmentation and object detection revealing limitations in power grid inspection. TS40K invites further research to tackle these challenges. Resources available in: https://github.com/dlavado/TS40K
DT-LSD: Deformable Transformer-Based Line Segment Detection
Authors: Sebastian Janampa; Marios Pattichis
Venue: WACV 2025 (main)

Abstract: Line segment detection is a fundamental low-level task in computer vision and improvements in this task can impact more advanced methods that depend on it. Most new methods developed for line segment detection are based on Convolutional Neural Networks (CNNs). Our paper seeks to address challenges that prevent the wider adoption of transformer-based methods for line segment detection. More specifically we introduce a new model called Deformable Transformer-based Line Segment Detection (DT-LSD) that supports cross-scale interactions and can be trained quickly. This work proposes a novel Deformable Transformer-based Line Segment Detector (DT-LSD) that addresses LETR's drawbacks. For faster training we introduce Line Contrastive DeNoising (LCDN) a technique that stabilizes the one-to-one matching process and speeds up training by 34X. We show that DT-LSD is faster and more accurate than its predecessor transformer-based model (LETR) and outperforms all CNN-based models in terms of accuracy. In the Wireframe dataset DT-LSD achieves 71.7 for sAP^10 and 73.9 for sAP^15; while 33.2 for sAP^10 and 35.1 for sAP^15 in the YorkUrban dataset. Code available at https://github.com/SebastianJanampa/DT-LSD.
Unconstrained Open Vocabulary Image Classification: Zero-Shot Transfer from Text to Image via CLIP Inversion
Authors: Philipp Allgeuer; Kyra Ahrens; Stefan Wermter
Venue: WACV 2025 (main)

Abstract: We introduce NOVIC an innovative real-time uNconstrained Open Vocabulary Image Classifier that uses an autoregressive transformer to generatively output classification labels as language. Leveraging the extensive knowledge of CLIP models NOVIC harnesses the embedding space to enable zero-shot transfer from pure text to images. Traditional CLIP models despite their ability for open vocabulary classification require an exhaustive prompt of potential class labels restricting their application to images of known content or context. To address this we propose an "object decoder" model that is trained on a large-scale 92M-target dataset of templated object noun sets and LLM-generated captions to always output the object noun in question. This effectively inverts the CLIP text encoder and allows textual object labels from essentially the entire English language to be generated directly from image-derived embedding vectors without requiring any a priori knowledge of the potential content of an image and without any label biases. The trained decoders are tested on a mix of manually and web-curated datasets as well as standard image classification benchmarks and achieve fine-grained prompt-free prediction scores of up to 87.5% a strong result considering the model must work for any conceivable image and without any contextual clues.