GenAI4ELab/papercli-papers huggingface dataset:
Segment Anything Meets Point Tracking
Authors: Frano Rajič; Lei Ke; Yu-Wing Tai; Chi-Keung Tang; Martin Danelljan; Fisher Yu
Venue: WACV 2025 (main)

Abstract: Foundation models have marked a significant stride toward addressing generalization challenges in deep learning. While the Segment Anything Model (SAM) has established a strong foothold in image segmentation existing video segmentation methods still require extensive mask labeling for fine-tuning or face performance drops on unseen data domains otherwise. In this paper we show how foundation models for image segmentation make a step toward enhancing domain generalizability in video segmentation. We discover that combined with long-term point tracking image segmentation models yield state-of-the-art results in zero-shot video segmentation across multiple benchmarks. Surprisingly point trackers exhibit generalization to domains beyond their synthetic pre-training sequences which we attribute to the trackers' ability to harness the rich local information in the vicinity of each tracked point. Thus we introduce SAM-PT an innovative method for point-centric video segmentation leveraging the capabilities of SAM alongside long-term point tracking. SAM-PT extends SAM's capability to tracking and segmenting anything in dynamic videos. Unlike traditional video segmentation methods that focus on object-centric mask propagation our approach uniquely exploits point propagation to utilize local structure information independent of object semantics. The effectiveness of point-based tracking is underscored by direct evaluation on the zero-shot open-world UVO benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks including DAVIS YouTube-VOS and BDD100K suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions. We release our code at https://github.com/SysCV/sam-pt.
Cross-Domain and Cross-Dimension Learning for Image-to-Graph Transformers
Authors: Alexander H. Berger; Laurin Lux; Suprosanna Shit; Ivan Ezhof; Georgios Kaissis; Martin J. Menten; Daniel Rueckert; Johannes C. Paetzold
Venue: WACV 2025 (main)

Abstract: Direct image-to-graph transformation is a challenging task solving object detection and relationship prediction in a single model. Due to this task's complexity large training datasets are rare in many domains making the training of deep-learning methods challenging. This data sparsity necessitates transfer learning strategies akin to the state-of-the-art in general computer vision. In this work we introduce a set of methods enabling cross-domain and cross-dimension learning for image-to-graph transformers. We propose (1) a regularized edge sampling loss to effectively learn object relations in multiple domains with different numbers of edges (2) a domain adaptation framework for image-to-graph transformers aligning image- and graph-level features from different domains and (3) a projection function that allows using 2D data for training 3D transformers. We demonstrate our method's utility in cross-domain and cross-dimension experiments where we utilize labeled data from 2D road networks for simultaneous learning in vastly different target domains. Our method consistently outperforms standard transfer learning and self-supervised pretraining on challenging benchmarks such as retinal or whole-brain vessel graph extraction.
Beyond Boxes: Mask-Guided Spatio-Temporal Feature Aggregation for Video Object Detection
Authors: Khurram Azeem Hashmi; Talha Uddin Sheikh; Didier Stricker; Muhammad Zeshan Afzal
Venue: WACV 2025 (main)

Abstract: The primary challenge in Video Object Detection (VOD) is effectively exploiting temporal information to enhance object representations. Traditional strategies such as aggregating region proposals often suffer from feature variance due to the inclusion of background information. We introduce a novel instance mask-based feature aggregation approach significantly refining this process and deepening the understanding of object dynamics across video frames. We present FAIM a new VOD method that enhances temporal Feature Aggregation by leveraging Instance Mask features. In particular we propose the lightweight Instance Feature Extraction Module (IFEM) to learn instance mask features and the Temporal Instance Classification Aggregation Module (TICAM) to aggregate instance mask and classification features across video frames. Using YOLOX as a base detector FAIM achieves 87.9% mAP on the ImageNet VID dataset at 33 FPS on a single 2080Ti GPU setting a new benchmark for the speed-accuracy trade-off. Additional experiments on multiple datasets validate that our approach is robust method-agnostic and effective in multi-object tracking demonstrating its broader applicability to video understanding tasks.
SpiralMLP: A Lightweight Vision MLP Architecture
Authors: Haojie Mu; Burhan Ul Tayyab; Nicholas Chua
Venue: WACV 2025 (main)

Abstract: We present SpiralMLP a novel architecture introduces a Spiral FC layer as a replacement for the conventional Token Mixing approach. Differing from several existing MLP-based models that primarily emphasize axes our Spiral FC layer is designed as a deformable convolution layer with spiral-like offsets. We further adapt Spiral FC into two variants: Self-Spiral FC and Cross-Spiral FC enabling both local and global feature integration seamlessly eliminating the need for additional processing steps. To thoroughly investigate the effectiveness of the spiral-like offsets and validate our design we conduct ablation studies and explore optimal configurations. In empirical tests SpiralMLP reaches state-of-the-art performance similar to Transformers CNNs and other MLPs benchmarking on ImageNet-1k COCO and ADE20K. SpiralMLP still maintains linear computational complexity O(HW) and is compatible with varying input image resolutions. Our study reveals that targeting the full receptive field is not essential for achieving high performance instead adopting a refined approach offers better results.
On-the-Fly Object-aware Representative Point Selection in Point Cloud
Authors: Xiaoyu Zhang; Ziwei Wang; Hai Dong; Zhifeng Bao; Jiajun Liu
Venue: WACV 2025 (main)

Abstract: Point clouds are essential for object modeling and play a critical role in assisting driving tasks for autonomous vehicles (AVs). However the significant volume of data generated by AVs creates challenges for storage bandwidth and processing cost. To tackle these challenges we propose a representative point selection framework for point cloud downsampling which preserves critical object-related information while effectively filtering out irrelevant background points. Our method involves two steps: (1) Object Presence Detection where we introduce an unsupervised density peak-based classifier and a supervised Naive Bayes classifier to handle diverse scenarios and (2) Sampling Budget Allocation where we propose a strategy that selects object-relevant points while maintaining a high retention rate of object information. Extensive experiments on the KITTI and nuScenes datasets demonstrate that our method consistently outperforms state-of-the-art baselines in both efficiency and effectiveness across varying sampling rates. As a model-agnostic solution our approach integrates seamlessly with diverse downstream models making it a valuable and scalable addition to the 3D point cloud downsampling toolkit for AV applications.
Socially-Informed Reconstruction for Pedestrian Trajectory Forecasting
Authors: Haleh Damirchi; Ali Etemad; Michael Greenspan
Venue: WACV 2025 (main)

Abstract: Pedestrian trajectory prediction remains a challenge for autonomous systems particularly due to the intricate dynamics of social interactions. Accurate forecasting requires a comprehensive understanding not only of each pedestrian's previous trajectory but also of their interaction with the surrounding environment an important part of which are other pedestrians moving dynamically in the scene. To learn effective socially-informed representations we propose a model that uses a reconstructor alongside a conditional variational autoencoder-based trajectory forecasting module. This module generates pseudo-trajectories which we use as augmentations throughout the training process. To further guide the model towards social awareness we propose a novel social loss that aids in forecasting of more stable trajectories. We validate our approach through extensive experiments demonstrating strong performances in comparison to state of-the-art methods on the ETH/UCY and SDD benchmarks.
Continuous Spatio-Temporal Memory Networks for 4D Cardiac Cine MRI Segmentation
Authors: Meng Ye; Bingyu Xin; Leon Axel; Dimitris Metaxas
Venue: WACV 2025 (main)

Abstract: Current cardiac cine magnetic resonance image (cMR) studies focus on the end diastole (ED) and end systole (ES) phases while ignoring the abundant temporal information in the whole image sequence. This is because whole sequence segmentation is currently a tedious process and inaccurate. Conventional whole sequence segmentation approaches first estimate the motion field between frames which is then used to propagate the mask along the temporal axis. However the mask propagation results could be prone to error especially for the basal and apex slices where through-plane motion leads to significant morphology and structural change during the cardiac cycle. Inspired by recent advances in video object segmentation (VOS) based on spatio-temporal memory (STM) networks we propose a continuous STM (CSTM) network for semi-supervised whole heart and whole sequence cMR segmentation. Our CSTM network takes full advantage of the spatial scale temporal and through-plane continuity prior of the underlying heart anatomy structures to achieve accurate and fast 4D segmentation. Results of extensive experiments across multiple cMR datasets show that our method can improve the 4D cMR segmentation performance especially for the hard-to-segment regions. Project page is at https://github.com/DeepTag/CSTM.
Foundation X: Integrating Classification Localization and Segmentation through Lock-Release Pretraining Strategy for Chest X-ray Analysis
Authors: Nahid Ul Islam; DongAo Ma; Jiaxuan Pang; Shivasakthi Senthil Velan; Michael Gotway; Jianming Liang
Venue: WACV 2025 (main)

Abstract: Developing robust and versatile deep-learning models is essential for enhancing diagnostic accuracy and guiding clinical interventions in medical imaging but it requires a large amount of annotated data. The advancement of deep learning has facilitated the creation of numerous medical datasets with diverse expert-level annotations. Aggregating these datasets can maximize data utilization and address the inadequacy of labeled data. However the heterogeneity of expert-level annotations across tasks such as classification localization and segmentation presents a significant challenge for learning from these datasets. To this end we introduce Foundation X an end-to-end framework that utilizes diverse expert-level annotations from numerous public datasets to train a foundation model capable of multiple tasks including classification localization and segmentation. To address the challenges of annotation and task heterogeneity we propose a Lock-Release pretraining strategy to enhance the cyclic learning from multiple datasets combined with the student-teacher learning paradigm ensuring the model retains general knowledge for all tasks while preventing overfitting to any single task. To demonstrate the effectiveness of Foundation X we trained a model using 11 chest X-ray datasets covering annotations for classification localization and segmentation tasks. Our experimental results show that Foundation X achieves notable performance gains through extensive annotation utilization excels in cross-dataset and cross-task learning and further enhances performance in organ localization and segmentation tasks. All code and pretrained models are publicly accessible at GitHub.com/JLiangLab/Foundation_X.
MVMD: A Multi-View Approach for Enhanced Mirror Detection
Authors: Yidan Shen; Yu Wen; Chen Zhang; Xin Fu; Renjie Hu
Venue: WACV 2025 (main)

Abstract: In 3D reconstruction mirrors introduce significant challenges by creating distorted and fragmented spaces resulting in inaccurate and unreliable 3D models. As 3D reconstruction typically relies on multi-view images to capture different perspectives of a scene detecting and labeling mirrors in multi-view images before reconstruction can effectively address this issue. However existing methods focus solely on single-image detection overlooking the rich information provided by multi-view setups. To overcome this limitation we propose MVMD a novel Multi-View Mirror Detection method along with the first database specifically designed for mirror detection in multi-view scenes. The design of MVMD is grounded in the inherent associations between objects seen from different views and those reflected inside and outside of mirrors. These relationships are learned through cross- and self-attention mechanisms. MVMD consists of three key blocks: the Inter-Views Block tracks the shifts of objects within mirrors caused by changes in viewpoint; the Intra-View Block detects object reflections inside mirrors; and the Refinement Block sharpens mirror boundaries and enhances detected details. Experimental results show that our method improves accuracy by up to 2.6% and IoU by up to 11.1% compared to single-image mirror detection techniques. This substantial improvement makes MVMD particularly effective for computer vision tasks especially in enhancing the accuracy of 3D reconstruction in mirror-dense environments. Code and data are available at: https://github.com/mvmdwacv25.
Mind the Map! Accounting for Existing Maps When Estimating Online HDMaps from Sensors
Authors: Rémy Sun; Li Yang; Diane Lingrand; Frederic Precioso
Venue: WACV 2025 (main)

Abstract: While HDMaps are a crucial component of autonomous driving they are expensive to acquire and maintain. Estimating these maps from sensors therefore promises to significantly lighten costs. These estimations however overlook existing HDMaps with current methods at most geolocalizing low quality maps or considering a general database of known maps. In this paper we propose to account for existing maps of the precise situation studied when estimating HDMaps. To prove this we identify 3 reasonable types of useful existing maps (minimalist noisy and outdated). We then introduce MapEX a novel online HDMap estimation framework that accounts for existing maps. MapEX achieves this by encoding map elements into query tokens and by refining the matching algorithm used to train classic query based map estimation models. We demonstrate that MapEX brings significant improvements on the nuScenes dataset. For instance MapEX - given noisy maps - improves by 38% over the MapTRv2 detector it is based on and by 8% over the current SOTA.
Treading Towards Privacy-Preserving Table Structure Recognition
Authors: Sachin Raja; Ajoy Mondal; C.V. Jawahar
Venue: WACV 2025 (main)

Abstract: We present TabGuard a privacy-preserving framework for an end-to-end secure Table Structure Recognition. TabGuard masks all the contents of the table locally and utilizes the masked table image for structure recognition. Our method is simple yet effective for detecting table cells while preserving the inherent table alignment characteristics to reconstruct tables. Our approach benefits from inductive bias expressed through an approximated table grid which helps alleviate challenges in the detection of cells that are small or have extreme aspect ratios. Experimental results demonstrate that our solution not only establishes a new state-of-the-art on several benchmark datasets but also effectively addresses long-standing challenges associated with dense tables having complex layouts. We make our code publically available at https://github.com/sachinraja13/TabGuard.
TreeFormer: Single-View Plant Skeleton Estimation via Tree-Constrained Graph Generation
Authors: Xinpeng Liu; Hiroaki Santo; Yosuke Toda; Fumio Okura
Venue: WACV 2025 (main)

Abstract: Accurate estimation of plant skeletal structure (e.g. branching structure) from images is essential for smart agriculture and plant science. Unlike human skeletons with fixed topology plant skeleton estimation presents a unique challenge i.e. estimating arbitrary tree graphs from images. While recent graph generation methods successfully infer thin structures from images it is challenging to constrain the output graph strictly to a tree structure. To this problem we present TreeFormer a plant skeleton estimator via tree-constrained graph generation. Our approach combines learning-based graph generation with traditional graph algorithms to impose the constraints during the training loop. Specifically our method projects an unconstrained graph onto a minimum spanning tree (MST) during the training loop and incorporates this prior knowledge into the gradient descent optimization by suppressing unwanted feature values. Experiments show that our method accurately estimates target plant skeletal structures for multiple domains: Synthetic tree patterns real botanical roots and grapevine branches. Our implementations are available at https://github.com/huntorochi/TreeFormer/.
Rethinking Cluster-Conditioned Diffusion Models for Label-Free Image Synthesis
Authors: Nikolaos Adaloglou; Tim Kaiser; Felix Michels; Markus Kollmann
Venue: WACV 2025 (main)

Abstract: Diffusion-based image generation models can enhance image quality when conditioned on ground truth labels. Here we conduct a comprehensive experimental study on image-level conditioning for diffusion models using cluster assignments. We investigate how individual clustering determinants such as the number of clusters and the clustering method impact image synthesis across three different datasets. Given the optimal number of clusters with respect to image synthesis we show that cluster-conditioning can achieve state-of-the-art performance with an FID of 1.67 for CIFAR10 and 2.17 for CIFAR100 along with a strong increase in training sample efficiency. We further propose a novel empirical method to estimate an upper bound for the optimal number of clusters. Unlike existing approaches we find no significant association between clustering performance and the corresponding cluster-conditional FID scores. Code is available at https://github.com/ HHU-MMBS/cedm-official-wavc2025
Robust Portrait Image Matting and Depth-of-Field Synthesis via Multiplane Images
Authors: Zhefan Rao; Tianjia Zhang; Yuen Fui Lau; Qifeng Chen
Venue: WACV 2025 (main)

Abstract: High-quality portrait photography has become an essential function in our daily lives. However due to the limited aperture and focal length of a smartphone camera images captured by a smartphone cannot match the same level of bokeh effect by a digital single-lens reflex camera. A typical solution on a smartphone is to simulate out-of-focus effects from an all-in-focus image where the key is robust depth estimation and portrait matting. To achieve this we propose a multi-stage multi-branch matting network to estimate a strand-level portrait alpha mask which is then used to refine the coarse depth map from the pre-trained model. Combining the input portrait image with the estimated depth map and alpha mask we propose a learning-free optimization mechanism to construct a multi-plane image (MPI) representation for depth-of-field synthesis. The MPI consists of multiple layers of disk-blurred images with kernel size proportional to the absolute depth distance to the focus layer. Then a depth-aware blurring process is applied to enforce the bokeh effect. Besides each MPI layer has an alpha channel controlling the visibility according to the corresponding depth. Finally an image with bokeh is rendered by compositing all MPI layers. We conduct comprehensive experiments to evaluate our method which demonstrates that our method can generate more accurate alpha masks and more realistic images with bokeh compared to prior work.
Context-Aware Outlier Rejection for Robust Multi-View 3D Tracking of Similar Small Birds in An Outdoor Aviary
Authors: Keon Moradi; Ethan Haque; Jasmeen Kaur; Alexandra B. Bentz; Eli S. Bridge; Golnaz Habibi
Venue: WACV 2025 (main)

Abstract: This paper presents a novel approach for robust 3D tracking of multiple birds in an outdoor aviary using a multi-camera system. Our method addresses the significant challenges posed by the visual similarity of birds and the complexities of their rapid movements in three-dimensional space. We leverage environmental context called landmarks to enhance feature matching between cameras leading to more accurate 3D reconstructions and tracking of their positions. Removing outliers in multi-view settings is especially challenging because the objects of interest are visually similar. In our approach outliers are rejected based on their nearest landmark. This leads to constructing precise 3D models of individual birds; additionally it applies these models to track multiple birds simultaneously. By utilizing environmental context our approach significantly improves the differentiation between visually similar birds a key obstacle in existing tracking systems. Experimental results demonstrate the effectiveness of our method showing a 20% elimination of outliers in the 3D reconstruction process with a 97% accuracy in matching. This remarkable accuracy in 3D modeling translates to robust and reliable tracking of multiple birds even in challenging outdoor conditions. Our work not only advances the field of computer vision but also provides a valuable tool for studying bird behavior and movement patterns in natural settings. We also provide a large annotated dataset of 80 birds residing in four enclosures for 20 hours of footage which provides a rich testbed for researchers in computer vision ornithologists and ecologists. Code and the link to the dataset is available at https://github.com/airou-lab/3D_Multi_Bird_Tracking.
MLLM-LLaVA-FL: Multimodal Large Language Model Assisted Federated Learning
Authors: Jianyi Zhang; Hao Yang; Ang Li; Xin Guo; Pu Wang; Haiming Wang; Yiran Chen; Hai Li
Venue: WACV 2025 (main)

Abstract: Previous studies on federated learning (FL) often encounter performance degradation due to data heterogeneity among different clients. In light of the recent advances in multimodal large language models (MLLMs) such as GPT-4v and LLaVA which demonstrate their exceptional proficiency in multimodal tasks such as image captioning and multimodal question answering. We introduce a novel federated learning framework named Multimodal Large Language Model Assisted Federated Learning (MLLM-LLaVA-FL) which employs powerful MLLMs at the server end to address the heterogeneous and long-tailed challenges. Owing to the advanced cross-modality representation capabilities and the extensive open-vocabulary prior knowledge of MLLMs our framework is adept at harnessing the extensive yet previously underexploited open-source data accessible from websites and powerful server-side computational resources. Hence the MLLM-LLaVA-FL not only enhances the performance but also avoids increasing the risk of privacy leakage and the computational burden on local devices distinguishing it from prior methodologies. Our framework has three key stages. Initially we conduct global visual-text pretraining of the model. This pretraining is facilitated by utilizing the extensive open-source data available online with the assistance of MLLMs. Subsequently the pretrained model is distributed among various clients for local training. Finally once the locally trained models are transmitted back to the server a global alignment is carried out under the supervision of MLLMs to further enhance the performance. Experimental evaluations on established benchmarks show that our framework delivers promising performance in the typical scenarios with data heterogeneity and long-tail distribution across different clients in FL.