GenAI4ELab/papercli-papers huggingface dataset:
CrowdMAC: Masked Crowd Density Completion for Robust Crowd Density Forecasting
Authors: Ryo Fujii; Ryo Hachiuma; Hideo Saito
Venue: WACV 2025 (main)

Abstract: A crowd density forecasting task aims to predict how the crowd density map will change in the future from observed past crowd density maps. However the past crowd density maps are often incomplete due to the miss-detection of pedestrians and it is crucial to develop a robust crowd density forecasting model against the miss-detection. This paper presents a MAsked crowd density Completion framework for crowd density forecasting (CrowdMAC) which is simultaneously trained to forecast future crowd density maps from partially masked past crowd density maps (i.e. forecasting maps from past maps with miss-detection) while reconstructing the masked observation maps (i.e. imputing past maps with miss-detection). Additionally we propose Temporal-Density-aware Masking (TDM) which non-uniformly masks tokens in the observed crowd density map considering the sparsity of the crowd density maps and the informativeness of the subsequent frames for the forecasting task. Moreover we introduce multi-task masking to enhance training efficiency. In the experiments CrowdMAC achieves state-of-the-art performance on seven large-scale datasets including SDD ETH-UCY inD JRDB VSCrowd FDST and croHD. We also demonstrate the robustness of the proposed method against both synthetic and realistic miss-detections. The code is released at https://fujiry0.github.io/CrowdMAC-project-page.
SEED4D: A Synthetic Ego-Exo Dynamic 4D Data Generator Driving Dataset and Benchmark
Authors: Marius Kästingschäfer; Théo Gieruc; Sebastian Bernhard; Dylan Campbell; Eldar Insafutdinov; Eyvaz Najafli; Thomas Brox
Venue: WACV 2025 (main)

Abstract: Models for egocentric 3D and 4D reconstruction including few-shot interpolation and extrapolation settings can benefit from having images from exocentric viewpoints as supervision signals. No existing dataset provides the necessary mixture of complex dynamic and multi-view data. To facilitate the development of 3D and 4D reconstruction methods in the autonomous driving context we propose a Synthetic Ego-Exo Dynamic 4D (SEED4D) data generator and dataset. We present a customizable easy-to-use data generator for spatio-temporal multi-view data creation. Our open-source data generator allows the creation of synthetic data for camera setups commonly used in the NuScenes KITTI360 and Waymo datasets. Additionally SEED4D encompasses two large-scale multi-view synthetic urban scene datasets. Our static (3D) dataset encompasses 212k inward- and outward-facing vehicle images from 2k scenes while our dynamic (4D) dataset contains 16.8M images from 10k trajectories each sampled at 100 points in time with egocentric images exocentric images and LiDAR data. The datasets and the data generator can be found here: https://seed4d.github.io
The FineView Dataset:A 3D Scanned Multi-View Object Dataset of Fine-Grained Category Instances
Authors: Suguru Onda; Ryan Farrell
Venue: WACV 2025 (main)

Abstract: In the past decade state-of-the-art deep learning models have shown impressive performance in many computer vision tasks by learning from large and diverse image datasets. Most of these datasets consist of web-scraped image collections. This approach however makes it very challenging to obtain desirable data such as multiple views of the same object 3D geometric information or camera parameters for a large-scale image dataset. In this paper we propose a 3D-scanned multi-view 2D image dataset of fine-grained category instances with accurate camera calibration parameters. We describe our bi-directional multi-camera and 3D scanning system and the data collection pipeline. Our target objects are relatively small highly-detailed fine-grained category instances such as insects. We present this dataset as a contribution to fine-grained visual categorization 3D representation learning and for use in other computer vision tasks. The final version of the FineView dataset is available at: https://github.com/byu-vision/fineview
Incorporating Task Progress Knowledge for Subgoal Generation in Robotic Manipulation through Image Edits
Authors: Xuhui Kang; Yen-Ling Kuo
Venue: WACV 2025 (main)

Abstract: Understanding the progress of a task allows humans to not only track what has been done but also to better plan for future goals. We demonstrate TaKSIE a novel framework that incorporates task progress knowledge into visual subgoal generation for robotic manipulation tasks. We jointly train a recurrent network with a latent diffusion model to generate the next visual subgoal based on the robot's current observation and the input language command. At execution time the robot leverages a visual progress representation to monitor the task progress and adaptively samples the next visual subgoal from the model to guide the manipulation policy. We train and validate our model in simulated and real-world robotic tasks achieving state-of-the-art performance on the CALVIN manipulation benchmark. We find that the inclusion of task progress knowledge can improve the robustness of trained policy for different initial robot poses or various movement speeds during demonstrations. The project page is available at https://live-robotics-uva.github.io/TaKSIE/.
Enhancing Zero-Shot Facial Expression Recognition by LLM Knowledge Transfer
Authors: Zengqun Zhao; Yu Cao; Shaogang Gong; Ioannis Patras
Venue: WACV 2025 (main)

Abstract: Current facial expression recognition (FER) models are often designed in a supervised learning manner and thus are constrained by the lack of large-scale facial expression images with high-quality annotations. Consequently these models often fail to generalize well performing poorly on unseen images in inference. Vision-language-based zero-shot models demonstrate a promising potential for addressing such challenges. However these models lack task-specific knowledge and therefore are not optimized for the nuances of recognizing facial expressions. To bridge this gap this work proposes a novel method Exp-CLIP to enhance zero-shot FER by transferring the task knowledge from large language models (LLMs). Specifically based on the pre-trained vision-language encoders we incorporate a projection head designed to map the initial joint vision-language space into a space that captures representations of facial actions. To train this projection head for subsequent zero-shot predictions we propose to align the projected visual representations with task-specific semantic meanings derived from the LLM encoder and the text instruction-based strategy is employed to customize the LLM knowledge. Given unlabelled facial data and efficient training of the projection head Exp-CLIP achieves superior zero-shot results to the CLIP models and several other large vision-language models (LVLMs) on seven in-the-wild FER datasets.
Non-Cross Diffusion for Semantic Consistency
Authors: Ziyang Zheng; Ruiyuan Gao; Qiang Xu
Venue: WACV 2025 (main)

Abstract: In diffusion models deviations from a straight generative flow are a common issue resulting in semantic inconsistencies and suboptimal generations. To address this challenge we introduce `Non-Cross Diffusion' an innovative approach in generative modeling for learning ordinary differential equation (ODE) models. Our methodology strategically incorporates an ascending dimension of input to effectively connect points sampled from two distributions with uncrossed paths. This design is pivotal in ensuring enhanced semantic consistency throughout the inference process which is especially critical for applications reliant on consistent generative flows including various distillation methods and deterministic sampling which are fundamental in image editing and interpolation tasks. Our empirical results demonstrate the effectiveness of Non-Cross Diffusion showing a substantial improvements in semantic consistencies at various inference steps and enhancing the overall performance of diffusion models.
Semantic Clustering of Image Retrieval Databases used for Visual Localization
Authors: Henry Hölzemann; Torsten Fiolka
Venue: WACV 2025 (main)

Abstract: Accurate self-localization of unmanned aerial systems (UAS) is needed to reduce their dependency on global navigation satellite systems (GNSS). Image retrieval techniques comparing aerial images with a reference database can be used for visual localization (VL). But the search space may be vast and a full search not feasible on a small UAS. In this work we propose a novel solution that divides the reference database into smaller clusters based on the semantic content of images. To this end we generate and make use of a dataset for semantic segmentation of aerial image captures. By characterizing scenes and objects in images semantically retrieval-based systems are able to differentiate images and scenes efficiently. Using a divide-and-conquer approach images with similar semantics are matched within smaller partial databases. This technique leads to reduced search times and approaches VL as a feasible solution for UAS localization in large-scale outdoor environments.
FDS: Feedback-Guided Domain Synthesis with Multi-Source Conditional Diffusion Models for Domain Generalization
Authors: Mehrdad Noori; Milad Cheraghalikhani; Ali Bahri; Gustavo A Vargas Hakim; David Osowiechi; Moslem Yazdanpanah; Ismail Ben Ayed; Christian Desrosiers
Venue: WACV 2025 (main)

Abstract: Domain Generalization techniques aim to enhance model robustness by simulating novel data distributions during training typically through various augmentation or stylization strategies. However these methods frequently suffer from limited control over the diversity of generated images and lack assurance that these images span distinct distributions. To address these challenges we propose FDS Feedback-guided Domain Synthesis a novel strategy that employs diffusion models to synthesize novel pseudo-domains by training a single model on all source domains and performing domain mixing based on learned features. By incorporating images that pose classification challenges to models trained on original samples alongside the original dataset we ensure the generation of a training set that spans a broad distribution spectrum. Our comprehensive evaluations demonstrate that this methodology sets new benchmarks in domain generalization performance across a range of challenging datasets effectively managing diverse types of domain shifts. The code can be found at https://github.com/Mehrdad-Noori/FDS.
LatteCLIP: Unsupervised CLIP Fine-Tuning via LMM-Synthetic Texts
Authors: Anh-Quan Cao; Maximilian Jaritz; Matthieu Guillaumin; Raoul de Charette; Loris Bazzani
Venue: WACV 2025 (main)

Abstract: Large-scale vision-language pre-trained (VLP) models (e.g. CLIP) are renowned for their versatility as they can be applied to diverse applications in a zero-shot setup. However when these models are used in specific domains their performance often falls short due to domain gaps or the under-representation of these domains in the training data. While fine-tuning VLP models on custom datasets with human-annotated labels can address this issue annotating even a small-scale dataset (e.g. 100k samples) can be an expensive endeavor often requiring expert annotators if the task is complex. To address these challenges we propose LatteCLIP an unsupervised method for fine-tuning CLIP models on classification with known class names in custom domains without relying on human annotations. Our method leverages Large Multimodal Models (LMMs) to generate expressive textual descriptions for both individual images and groups of images. These provide additional contextual information to guide the fine-tuning process in the custom domains. Since LMM-generated descriptions are prone to hallucination or missing details we introduce a novel strategy to distill only the useful information and stabilize the training. Specifically we learn rich per-class prototype representations from noisy generated texts and dual pseudo-labels. Our experiments on 10 domain-specific datasets show that LatteCLIP outperforms pre-trained zero-shot methods by an average improvement of +4.74 points in top-1 accuracy and other state-of-the-art unsupervised methods by +3.45 points.
GANFusion: Feed-Forward Text-to-3D with Diffusion in GAN Space
Authors: Souhaib Attaiki; Paul Guerrero; Duygu Ceylan; Niloy Mitra; Maks Ovsjanikov
Venue: WACV 2025 (main)

Abstract: We train a feed-forward text-to-3D diffusion generator for human characters using only single-view 2D data for supervision. Existing 3D generative models cannot yet match the fidelity of image and/or video generative models. State-of-the-art 3D generators are either limited by the volume and diversity of existing 3D data available for supervision or those that can be trained with only 2D data as supervision produce coarser results cannot be text-conditioned and/or must revert to test-time optimization. We observe that GAN- and diffusion-based generators have complementary qualities: GANs can be trained efficiently with 2D supervision to produce high-quality 3D objects but are hard to condition on text. In contrast diffusion models can be conditioned efficiently but tend to be hard to train with only 2D supervision. We introduce GANFusion that starts by generating unconditional triplane features for 3D data using a GAN architecture trained with only single-view 2D data. We then generate random samples from the GAN caption them and train a text-conditioned diffusion model that directly learns to sample from the space of good triplane features that can be decoded into 3D objects. We evaluate the proposed method in the context of text-conditioned full-body human generation and show improvements over possible alternatives.
Zero-Shot Detection of Out-of-Context Objects using Foundation Models
Authors: Anirban Roy; Adam Cobb; Ramneet Kaur; Sumit Jha; Nathaniel Bastian; Alexander Berenbeim; Robert Thomson; Iain Cruickshank; Alvaro Velasquez; Susmit Jha
Venue: WACV 2025 (main)

Abstract: We address the problem of detecting out-of-context (OOC) objects in a scene. Given an image we aim to detect whether the image has objects that are not present in their usual context and localize such OOC objects. Existing approaches for OOC detection rely on defining the common context in terms of the manually constructed features such as the co-occurrence of objects spatial relations between objects and shape and size of the objects and then learning such context for a given dataset. But context is often nuanced ranging from very common to very surprising. Further learned context from specific datasets may not be generalized as datasets may not truly represent the human notion of what is in context. Motivated by the success of large language models and more generally foundation models (FMs) in common sense reasoning we investigate the FM's ability to capture a more generalized notion of context. We find that a pre-trained FM such as GPT-4 provides a more nuanced notion of OOC and enables zero-shot OOC detection when coupled with other pre-trained FMs for caption generation such as BLIP-2 and image in-painting with Stable Diffusion 2.0. Our approach does not need any dataset-specific training. We demonstrate the efficacy of our approach on two OOC object detection datasets achieving 90.8% zero-shot accuracy on the MIT-OOC dataset and 87.26% on the IJCAI22-COCO-OOC dataset.
RefVSR++: Exploiting Reference Inputs for Reference-Based Video Super-Resolution
Authors: Han Zou; Masanori Suganuma; Takayuki Okatani
Venue: WACV 2025 (main)

Abstract: Smartphones with multi-camera systems featuring cameras with varying field-of-views (FoVs) are increasingly common. This variation in FoVs results in content differences across videos paving the way for an innovative approach to video super-resolution (VSR). This method enhances the VSR performance of lower resolution (LR) videos by leveraging higher resolution reference (Ref) videos. Previous works which operate on this principle generally expand on traditional VSR models by combining LR and Ref inputs over time into a unified stream. However we can expect that better results are obtained by independently aggregating these Ref image sequences temporally. Therefore we introduce an improved method RefVSR++ which performs the parallel aggregation of LR and Ref images in the temporal direction aiming to optimize the use of the available data. RefVSR++ also incorporates improved mechanisms for aligning image features over time crucial for effective VSR. Our experiments demonstrate that RefVSR++ outperforms previous works by over 1dB in PSNR setting a new benchmark in the field.
LLM-Generated Rewrite and Context Modulation for Enhanced Vision Language Models in Digital Pathology
Authors: Cagla Deniz Bahadir; Gozde B. Akar; Mert R. Sabuncu
Venue: WACV 2025 (main)

Abstract: Recent advancements in vision-language models (VLMs) have found important applications in medical imaging particularly in digital pathology. VLMs demand large-scale datasets of image-caption pairs which is often hard to obtain in medical domains. State-of-the-art VLMs in digital pathology have been pre-trained on datasets that are significantly smaller than their computer vision counterparts. Furthermore the caption of a pathology slide often refers to a small sub-set of features in the image--an important point that is ignored in existing VLM pre-training schemes. Another important issue that is under-appericated is that the performance of state-of-the-art VLMs in zero-shot classification tasks can be sensitive to the choice of the prompts. In this paper we first employ language rewrites using a large language model (LLM) to enrich a public pathology image-caption dataset and make it publicly available. Our extensive experiments demonstrate that by training with language rewrites we can boost the performance of a state-of-the-art digital pathology VLM on downstream tasks such as zero-shot classification and text-to-image and image-to-text retrieval. We further leverage LLMs to demonstrate the sensitivity of zero-shot classification results to the choice of prompts and propose a scalable approach to characterize this when comparing models. Finally we present a novel context modulation layer that adjusts the image embeddings for better aligning with the paired text and use context-specific language rewrites for training this layer. In our results we show that the proposed context modulation framework can further yield substantial performance gains.
HEX: Hierarchical Emergence Exploitation in Self-Supervised Algorithms
Authors: Kiran Kokilepersaud; Seulgi Kim; Mohit Prabhushankar; Ghassan AlRegib
Venue: WACV 2025 (main)

Abstract: In this paper we propose an algorithm that can be used on top of a wide variety of self-supervised (SSL) approaches to take advantage of hierarchical structures that emerge during training. SSL approaches typically work through some invariance term to ensure consistency between similar samples and a regularization term to prevent global dimensional collapse. Dimensional collapse refers to data representations spanning a lower-dimensional subspace. Recent work has demonstrated that the representation space of these algorithms gradually reflects a semantic hierarchical structure as training progresses. Data samples of the same hierarchical grouping tend to exhibit greater dimensional collapse locally compared to the dataset as a whole due to sharing features in common with each other. Ideally SSL algorithms would take advantage of this hierarchical emergence to have an additional regularization term to account for this local dimensional collapse effect. However the construction of existing SSL algorithms does not account for this property. To address this we propose an adaptive algorithm that performs a weighted decomposition of the denominator of the InfoNCE loss into two terms: local hierarchical and global collapse regularization respectively. This decomposition is based on an adaptive threshold that gradually lowers to reflect the emerging hierarchical structure of the representation space throughout training. It is based on an analysis of the cosine similarity distribution of samples in a batch. We demonstrate that this hierarchical emergence exploitation (HEX) approach can be integrated across a wide variety of SSL algorithms. Empirically we show performance improvements of up to 5.6% relative improvement over baseline SSL approaches on classification accuracy on Imagenet with 100 epochs of training.
ElasticLaneNet: An Efficient Geometry-Flexible Lane Detection Framework
Authors: Yaxin Feng; Yuan Lan; Luchan Zhang; Yang Xiang
Venue: WACV 2025 (main)

Abstract: The task of lane detection involves identifying the boundaries of driving areas in real-time. Recognizing lanes with variable and complex geometric structures remains a challenge. In this paper we explore a novel and flexible way of implicit lanes representation named Elastic Lane map (ELM) and introduce an efficient physics-informed end-to-end lane detection framework namely ElasticLaneNet (Elastic interaction energy-informed Lane detection Network). The approach considers predicted lanes as moving zero-contours on the flexibly shaped ELM that are attracted to the ground truth guided by an elastic interaction energy-loss function (EIE loss). Our framework well integrates the global information and low-level features. The method performs well in complex lane scenarios including those with large curvature turns intersections various crossing lanes Y-shapes lanes dense lanes etc. We apply our approach on three datasets: SDLane TuSimple and CULane. The results demonstrate exceptional performance of our method with the state-of-the-art results on the structurally diverse SDLane achieving F1-score of 89.51 Recall of 87.50 and Precision of 91.61 with fast inference speed.
Federated-Continual Dynamic Segmentation of Histopathology Guided by Barlow Continuity
Authors: Niklas Babendererde; Haozhe Zhu; Moritz Fuchs; Jonathan Stieber; Anirban Mukhopadhyay
Venue: WACV 2025 (main)

Abstract: Federated- and Continual Learning have been established as approaches to enable privacy-aware learning on continuously changing data as required for deploying AI systems in histopathology images. However data shifts can occur in a dynamic world spatially between institutions and temporally due to changing data over time. This leads to two issues: Client Drift where the central model degrades from aggregating data from clients trained on shifted data and Catastrophic Forgetting from temporal shifts such as changes in patient populations. Both tend to degrade the model's performance of previously seen data or spatially distributed training. Despite both problems arising from the same underlying problem of data shifts existing research addresses them only individually. In this work we introduce a method that can jointly alleviate Client Drift and Catastrophic Forgetting by using our proposed Dynamic Barlow Continuity that evaluates client updates on a public reference dataset and uses this to guide the training process to a spatially and temporally shift-invariant model. We evaluate our approach on the histopathology datasets BCSS and Semicol and prove our method to be highly effective by jointly improving the dice score as much as from 15.8% to 71.6% in Client Drift and from 42.5% to 62.8% in Catastrophic Forgetting. This enables Dynamic Learning by establishing spatio-temporal shift-invariance.