GenAI4ELab/papercli-papers huggingface dataset:
Situational Scene Graph for Structured Human-Centric Situation Understanding
Authors: Chinthani Sugandhika; Chen Li; Deepu Rajan; Basura Fernando
Venue: WACV 2025 (main)

Abstract: Graph based representation has been widely used in modelling spatio-temporal relationships in video understanding. Although effective existing graph-based approaches focus on capturing the human-object relationships while ignoring fine-grained semantic properties of the action components. These semantic properties are crucial for understanding the current situation such as where does the action takes place what tools are used and functional properties of the objects. In this work we propose a graph-based representation called Situational Scene Graph (SSG) to encode both human-object relationships and the corresponding semantic properties. The semantic details are represented as predefined roles and values inspired by situation frame which is originally designed to represent a single action. Based on our proposed representation we introduce the task of situational scene graph generation and propose a multi-stage pipeline Inter active and Complementary Network (InComNet) to address the task. Given that the existing datasets are not applicable to the task we further introduce a SSG dataset whose annotations consist of semantic role-value frames for human objects and verb predicates of human-object relations. Finally we demonstrate the effectiveness of our proposed SSG representation by testing on different downstream tasks. Experimental results show that the unified representation can not only benefit predicate classification and semantic role-value classification but also benefit reasoning tasks on human centric situation understanding. Code and data are available at https://github.com/LUNAProject22/SSG.
Continual Learning in 3D Point Clouds: Employing Spectral Techniques for Exemplar Selection
Authors: Hossein Resani; Behrooz Nasihatkon; Mohammadreza Alimoradi Jazi
Venue: WACV 2025 (main)

Abstract: We introduce a novel framework for Continual learning in 3D object classification. Our approach is based on the selection of prototypes from each class using spectral clustering. For non-Euclidean data such as point clouds spectral clustering can be employed as long as one can define a distance measure between pairs of samples. Choosing the appropriate distance measure enables us to leverage 3D geometric characteristics to identify representative prototypes for each class. We explore the effectiveness of clustering in the input space (3D points) local feature space (1024-dimensional points) and global feature space. We conduct experiments on the ModelNet40 ShapeNet and ScanNet datasets achieving state-of-the-art accuracy exclusively through the use of input space features. By leveraging the combined input local and global features we have improved the state-of-the-art on ModelNet40 and ShapeNet utilizing nearly half the memory used by competing approaches. For the challenging ScanNet dataset our method enhances accuracy by 4.1% while consuming just 28% of the memory used by our competitors demonstrating the scalability of our approach.
Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Models
Authors: Katherine Xu; Lingzhi Zhang; Jianbo Shi
Venue: WACV 2025 (main)

Abstract: Recent advances in text-to-image (T2I) diffusion models have facilitated creative and photorealistic image synthesis. By varying the random seeds we can generate many images for a fixed text prompt. Technically the seed controls the initial noise and in multi-step diffusion inference the noise used for reparameterization at intermediate timesteps in the reverse diffusion process. However the specific impact of the random seed on the generated images remains relatively unexplored. In this work we conduct a large-scale scientific study into the impact of random seeds during diffusion inference. Remarkably we reveal that the best 'golden' seed achieved an impressive FID of 21.60 compared to the worst 'inferior' seed's FID of 31.97. Additionally a classifier can predict the seed number used to generate an image with over 99.9% accuracy in just a few epochs establishing that seeds are highly distinguishable based on generated images. Encouraged by these findings we examined the influence of seeds on interpretable visual dimensions. We find that certain seeds consistently produce grayscale images prominent sky regions or image borders. Seeds also affect image composition including object location size and depth. Moreover by leveraging these 'golden' seeds we demonstrate improved image generation such as high-fidelity inference and diversified sampling. Our investigation extends to inpainting tasks where we uncover some seeds that tend to insert unwanted text artifacts. Overall our extensive analyses highlight the importance of selecting good seeds and offer practical utility for image generation.
OmniGS: Fast Radiance Field Reconstruction using Omnidirectional Gaussian Splatting
Authors: Longwei Li; Huajian Huang; Sai-Kit Yeung; Hui Cheng
Venue: WACV 2025 (main)

Abstract: Photorealistic reconstruction relying on 3D Gaussian Splatting has shown promising potential in various domains. However the current 3D Gaussian Splatting system only supports radiance field reconstruction using undistorted perspective images. In this paper we present OmniGS a novel omnidirectional Gaussian splatting system to take advantage of omnidirectional images for fast radiance field reconstruction. Specifically we conduct a theoretical analysis of spherical camera model derivatives in 3D Gaussian Splatting. According to the derivatives we then implement a new GPU-accelerated omnidirectional rasterizer that directly splats 3D Gaussians onto the equirectangular screen space for omnidirectional image rendering. We realize differentiable optimization of the omnidirectional radiance field without the requirement of cube-map rectification or tangent-plane approximation. Extensive experiments conducted in egocentric and roaming scenarios demonstrate that our method achieves state-of-the-art reconstruction quality and high rendering speed using omnidirectional images. The code will be publicly available at https://github.com/liquorleaf/OmniGS.
CusConcept: Customized Visual Concept Decomposition with Diffusion Models
Authors: Zhi Xu; Shaozhe Hao; Kai Han
Venue: WACV 2025 (main)

Abstract: Enabling generative models to decompose visual concepts from a single image is a complex and challenging problem. In this paper we study a new and challenging task customized concept decomposition wherein the objective is to leverage diffusion models to decompose a single image and generate visual concepts from various perspectives. To address this challenge we propose a two-stage framework CusConcept (short for Customized Visual Concept Decomposition) to extract customized visual concept embedding vectors that can be embedded into prompts for text-to-image generation. In the first stage CusConcept employs a vocabulary-guided concept decomposition mechanism to build vocabularies along human-specified conceptual axes. The decomposed concepts are obtained by retrieving corresponding vocabularies and learning anchor weights. In the second stage joint concept refinement is performed to enhance the fidelity and quality of generated images. We further curate an evaluation benchmark for assessing the performance of the open-world concept decomposition task. Our approach can effectively generate high-quality images of the decomposed concepts and produce related lexical predictions as secondary outcomes. Extensive qualitative and quantitative experiments demonstrate the effectiveness of CusConcept. Our code and data are available at https://github.com/xzLcan/CusConcept.
ALPI: Auto-Labeller with Proxy Injection for 3D Object Detection using 2D Labels Only
Authors: Saad Lahlali; Nicolas Granger; Herve Le Borgne; Quoc-Cuong Pham
Venue: WACV 2025 (main)

Abstract: 3D object detection plays a crucial role in various applications such as autonomous vehicles robotics and augmented reality. However training 3D detectors requires a costly precise annotation which is a hindrance to scaling annotation to large datasets. To address this challenge we propose a weakly supervised 3D annotator that relies solely on 2D bounding box annotations from images along with size priors. One major problem is that supervising a 3D detection model using only 2D boxes is not reliable due to ambiguities between different 3D poses and their identical 2D projection. We introduce a simple yet effective and generic solution: we build 3D proxy objects with annotations by construction and add them to the training dataset. Our method requires only size priors to adapt to new classes. To better align 2D supervision with 3D detection our method ensures depth invariance with a novel expression of the 2D losses. Finally to detect more challenging instances our annotator follows an offline pseudo-labelling scheme which gradually improves its 3D pseudo-labels. Extensive experiments on the KITTI dataset demonstrate that our method not only performs on-par or above previous works on the Car category but also achieves performance close to fully supervised methods on more challenging classes. We further demonstrate the effectiveness and robustness of our method by being the first to experiment on the more challenging nuScenes dataset. We additionally propose a setting where weak labels are obtained from a 2D detector pre-trained on MS-COCO instead of human annotations. The code is available at https://github.com/CEA-LIST/ALPI
One VLM to Keep it Learning: Generation and Balancing for Data-Free Continual Visual Question Answering
Authors: Deepayan Das; Davide Talon; Massimiliano Mancini; Yiming Wang; Elisa Ricci
Venue: WACV 2025 (main)

Abstract: Vision-Language Models (VLMs) have shown significant promise in Visual Question Answering (VQA) tasks by leveraging web-scale multimodal datasets. However these models often struggle with continual learning due to catastrophic forgetting when adapting to new tasks. As an effective remedy to mitigate catastrophic forgetting rehearsal strategy uses the data of past tasks upon learning new task. However such strategy incurs the need of storing past data which might not be feasible due to hardware constraints or privacy concerns. In this work we propose the first data-free method that leverages the language generation capability of a VLM instead of relying on external models to produce pseudo-rehearsal data for addressing continual VQA. Our proposal named as GaB generates pseudo-rehearsal data by posing previous task questions on new task data. Yet despite being effective the distribution of generated questions skews towards the most frequently posed questions due to the limited and task-specific training data. To mitigate this issue we introduce a pseudo-rehearsal balancing module that aligns the generated data towards the ground-truth data distribution using either the question meta-statistics or an unsupervised clustering method. We evaluate our proposed method on two recent benchmarks i.e. VQACL-VQAv2 and CLOVE-function benchmarks. GaB outperforms all the data-free baselines with substantial improvement in maintaining VQA performance across evolving tasks while being on-par with methods with access to the past data. Code will be made public soon.
Improving Accuracy and Generalization for Efficient Visual Tracking
Authors: Ram Zaveri; Shivang Patel; Yu Gu; Gianfranco Doretto
Venue: WACV 2025 (main)

Abstract: Efficient visual trackers overfit to their training distributions and lack generalization abilities resulting in them performing well on their respective in-distribution (ID) test sets and not as well on out-of-distribution (OOD) sequences imposing limitations to their deployment in-the-wild under constrained resources. We introduce SiamABC a highly efficient Siamese tracker that significantly improves tracking performance even on OOD sequences. SiamABC takes advantage of new architectural designs in the way it bridges the dynamic variability of the target and of new losses for training. Also it directly addresses OOD tracking generalization by including a fast backward-free dynamic test-time adaptation method that continuously adapts the model according to the dynamic visual changes of the target. Our extensive experiments suggest that SiamABC shows remarkable performance gains in OOD sets while maintaining accurate performance on the ID benchmarks. SiamABC outperforms MixFormerV2-S by 7.6% on the OOD AVisT benchmark while being 3x faster (100 FPS) on a CPU. Our code and models are available at https://wvuvl.github.io/SiamABC/.
Diffusion-Based Generative Regularization for Supervised Discriminative Learning
Authors: Takuya Asakura; Nakamasa Inoue; Koichi Shinoda
Venue: WACV 2025 (main)

Abstract: Ensuring the quality and quantity of labeled training data has long been a challenge in training deep neural networks for discriminative tasks. One solution to this problem is to use a generative model to augment training data and learn a discriminative model with it. For image classification with the recent development of diffusion models it has become possible to generate a variety synthetic images and there are high expectations for their use as training data. However to obtain high-quality labeled synthetic images the hyperparameters and prompts often need to be manually tuned and the accuracy of the trained image classification model is highly dependent on them. To address this issue this paper proposes diffusion-based generative regularization a supervised discriminative learning framework that utilizes a diffusion-based image generation model as a regularizer to robustly learn discriminative representations without the need to synthesize images. Our experiments using vision transformers and stable diffusion models on ImageNet-1k demonstrate that the proposed framework improves classification accuracy on both in-distribution and distribution-shifted data.
DDS: Decoupled Dynamic Scene-Graph Generation Network
Authors: A S M Iftekhar; Raphael Ruschel; Satish Kumar; Suya You; B. S. Manjunath
Venue: WACV 2025 (main)

Abstract: Scene-graph generation involves creating a structural representation of the relationships between objects in a scene by predicting subject-object-relation triplets from input data. Existing methods show poor performance in detecting triplets outside of a predefined set primarily due to their reliance on dependent feature learning. To address this issue we propose DDS- a decoupled dynamic scene-graph generation network- that consists of two independent branches that can disentangle extracted features. The key innovation of the current paper is the decoupling of the features representing the relationships from those of the objects which enables the detection of novel object-relationship combinations. The DDS model is evaluated on three datasets and outperforms previous methods by a significant margin especially in detecting previously unseen triplets.
Generalizable Single-View Object Pose Estimation by Two-Side Generating and Matching
Authors: Yujing Sun; Caiyi Sun; Yuan Liu; Yuexin Ma; Siu Ming Yiu
Venue: WACV 2025 (main)

Abstract: In this paper we present a novel generalizable object pose estimation method to determine the object pose using only one RGB image. Unlike traditional approaches that rely on instance-level object pose estimation and necessitate extensive training data our method offers generalization to unseen objects without extensive training operates with a single reference image of the object and eliminates the need for 3D object models or multiple views of the object. These characteristics are achieved by utilizing a diffusion model to generate novel-view images and conducting a two-sided matching on these generated images. Quantitative experiments demonstrate the superiority of our method over existing pose estimation techniques across both synthetic and real-world datasets. Remarkably our approach maintains strong performance even in scenarios with significant viewpoint changes highlighting its robustness and versatility in challenging conditions. The code will be released at https://github.com/scy639/Gen2SM
Per-Pixel Solution of Multispectral Photometric Stereo
Authors: Shin Ishihara; Imari Sato
Venue: WACV 2025 (main)

Abstract: Photometric Stereo (PS) estimates surface normals by analyzing images lit from different angles. Enhancing PS with spectral imaging known as multispectral photometric stereo (MPS) uses varying light source colors for simultaneous image capture. As in traditional PS obtaining a unique solution is challenging in MPS when the reflectance properties of the object are unknown. This paper presents an approach utilizing the spatial arrangement and color of light sources to solve the MPS problem in the condition of spatially varying reflectance from a minimum of seven spectral images without spatial smoothness constraints. A robust optimization technique is introduced to manage real data. Experiments on synthetic and real scenes validate the method's effectiveness including for non-Lambertian surfaces. The method can contribute to advanced digital archiving that simultaneously records surface normal and spectral reflectance.
Leveraging Vision Language Models for Specialized Agricultural Tasks
Authors: Muhammad Arbab Arshad; Talukder Zaki Jubery; Tirtho Roy; Rim Nassiri; Asheesh K. Singh; Arti Singh; Chinmay Hegde; Baskar Ganapathysubramanian; Aditya Balu; Adarsh Krishnamurthy; Soumik Sarkar
Venue: WACV 2025 (main)

Abstract: As Vision Language Models (VLMs) become increasingly accessible to farmers and agricultural experts there is a growing need to evaluate their potential in specialized tasks. We present AgEval a comprehensive benchmark for assessing VLMs' capabilities in plant stress phenotyping offering a solution to the challenge of limited annotated data in agriculture. Our study explores how general-purpose VLMs can be leveraged for domain-specific tasks with only a few annotated examples providing insights into their behavior and adaptability. AgEval encompasses 12 diverse plant stress phenotyping tasks evaluating zero-shot and few-shot in-context learning performance of state-of-the-art models including Claude GPT Gemini and LLaVA. Our results demonstrate VLMs' rapid adaptability to specialized tasks with the best-performing model showing an increase in F1 scores from 46.24% to 73.37% in 8-shot identification. To quantify performance disparities across classes we introduce metrics such as the coefficient of variation (CV) revealing that VLMs' training impacts classes differently with CV ranging from 26.02% to 58.03%. We also find that strategic example selection enhances model reliability with exact category examples improving F1 scores by 15.38% on average. AgEval establishes a framework for assessing VLMs in agricultural applications offering valuable benchmarks for future evaluations. Our findings suggest that VLMs with minimal few-shot examples show promise as a viable alternative to traditional specialized models in plant stress phenotyping while also highlighting areas for further refinement. Results and benchmark details are available at: https://github.com/arbab-ml/AgEval
RopeTP: Global Human Motion Recovery via Integrating Robust Pose Estimation with Diffusion Trajectory Prior
Authors: Mingjiang Liang; Yongkang Cheng; Hualin Liang; Shaoli Huang; Wei Liu
Venue: WACV 2025 (main)

Abstract: We present RopeTP a novel framework that combines Robust pose estimation with a diffusion Trajectory Prior to reconstruct global human motion from videos. At the heart of RopeTP is a hierarchical attention mechanism that significantly improves context awareness which is essential for accurately inferring the posture of occluded body parts. This is achieved by exploiting the relationships with visible anatomical structures enhancing the accuracy of local pose estimations. The improved robustness of these local estimations allows for the reconstruction of precise and stable global trajectories. Additionally RopeTP incorporates a diffusion trajectory model that predicts realistic human motion from local pose sequences. This model ensures that the generated trajectories are not only consistent with observed local actions but also unfold naturally over time thereby improving the realism and stability of 3D human motion reconstruction. Extensive experimental validation shows that RopeTP surpasses current methods on two benchmark datasets particularly excelling in scenarios with occlusions. It also outperforms methods that rely on SLAM for initial camera estimates and extensive optimization delivering more accurate and realistic trajectories.
GeoDiffuser: Geometry-Based Image Editing with Diffusion Models
Authors: Rahul Sajnani; Jeroen Vanbaar; Jie Min; Kapil D Katyal; Srinath Sridhar
Venue: WACV 2025 (main)

Abstract: The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However these methods are bespoke imprecise require additional information or are limited to only 2D image edits. We present GeoDiffuser a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation 3D rotation and removal. We present quantitative results including a perceptual study that shows how our approach is better than existing methods. The success of image generative models has enabled us to build methods that can edit images based on text or other user input. However these methods are bespoke imprecise require additional information or are limited to only 2D image edits. We present GeoDiffuser a zero-shot optimization-based method that unifies common 2D and 3D image-based object editing capabilities into a single method. Our key insight is to view image editing operations as geometric transformations. We show that these transformations can be directly incorporated into the attention layers in diffusion models to implicitly perform editing operations. Our training-free optimization method uses an objective function that seeks to preserve object style but generate plausible images for instance with accurate lighting and shadows. It also inpaints disoccluded parts of the image where the object was originally located. Given a natural image and user input we segment the foreground object using SAM and estimate a corresponding transform which is used by our optimization approach for editing. GeoDiffuser can perform common 2D and 3D edits like object translation 3D rotation and removal. We present quantitative results including a perceptual study that shows how our approach is better than existing methods.