GenAI4ELab/papercli-papers huggingface dataset:
BioNet and NeFF: Crop Biomass Prediction from Point Clouds to Drone Imagery
Authors: Xuesong Li; Zeeshan Hayder; Ali Zia; Connor Cassidy; Shiming Liu; Warwick Stiller; Eric Stone; Warren Conaty; Lars Petersson; Vivien Rolland
Venue: WACV 2025 (main)

Abstract: Crop biomass offers crucial insights into plant health and yield making it essential for crop science farming systems and agricultural research. However current measurement methods which are labor-intensive destructive and imprecise hinder large-scale quantification of this trait. To address this limitation we present a biomass prediction network (BioNet) designed for adaptation across different data modalities including point clouds and drone imagery. Our BioNet utilizing a sparse 3D convolutional neural network (CNN) and a transformer-based prediction module processes point clouds and other 3D data representations to predict biomass. To further extend BioNet for drone imagery we integrate a neural feature field (NeFF) module enabling 3D structure reconstruction and the transformation of 2D semantic features from vision foundation models into the corresponding 3D surfaces. For the point cloud modality BioNet demonstrates superior performance on two public datasets with an approximate 6.1% relative improvement (RI) over the state-of-the-art. In the RGB image modality the combination of BioNet and NeFF achieves a 7.9% RI. Additionally the NeFF-based approach utilizes inexpensive portable drone-mounted cameras providing a scalable solution for large field applications.
AutoProSAM: Automated Prompting SAM for 3D Multi-Organ Segmentation
Authors: Chengyin Li; Rafi Ibn Sultan; Prashant Khanduri; Yao Qiang; Chetty Indrin; Dongxiao Zhu
Venue: WACV 2025 (main)

Abstract: Segment Anything Model (SAM) is one of the pioneering prompt-based foundation models for image segmentation and has been rapidly adopted for various medical imaging applications. However in clinical settings creating effective prompts is notably challenging and time-consuming requiring the expertise of domain specialists such as physicians. This requirement significantly diminishes SAM's primary advantage--its interactive capability with end users--in medical applications. Moreover recent studies have indicated that SAM originally designed for 2D natural images performs suboptimally on 3D medical image segmentation tasks. This subpar performance is attributed to the domain gaps between natural and medical images and the disparities in spatial arrangements between 2D and 3D images particularly in multi-organ segmentation applications. To overcome these challenges we present a novel technique termed AutoProSAM. This method automates 3D multi-organ CT-based segmentation by leveraging SAM's foundational model capabilities without relying on domain experts for prompts. The approach utilizes parameter-efficient adaptation techniques to adapt SAM for 3D medical imagery and incorporates an effective automatic prompt learning paradigm specific to this domain. By eliminating the need for manual prompts it enhances SAM's capabilities for 3D medical image segmentation and achieves state-of-the-art (SOTA) performance in CT-based multi-organ segmentation tasks. The code is in this link.
VideoGameBunny: Towards Vision Assistants for Video Games
Authors: Mohammad Reza Taesiri; Cor-Paul Bezemer
Venue: WACV 2025 (main)

Abstract: Large multimodal models known as LMMs hold substantial promise across various domains from personal assistance in daily tasks to sophisticated applications like medical diagnostics. However their capabilities have limitations in the video game domain including challenges with scene understanding hallucinations and inaccurate descriptions of video game content especially in open-source models. This paper describes the development of VideoGameBunny a LLaVA-style model based on Bunny specifically tailored for understanding images from video games. We release intermediate checkpoints training logs and an extensive dataset comprising 185259 video game images from 413 titles along with 389565 image-instruction pairs that include image captions question-answer pairs and a JSON representation of 16 elements of 136974 images. Our experiments show that our high-quality game-related data has the potential to make a relatively small model outperform the much larger state-of-the-art model LLaVa-1.6-34b which has more than four times the number of parameters. Our study paves the way for future research in video game understanding on tasks such as playing commentary and debugging. Code and data are available at videogamebunny.github.io
ShapeMorph: 3D Shape Completion via Blockwise Discrete Diffusion
Authors: Jiahui Li; Pourya Shamsolmoali; Yue Lu; Masoumeh Zareapoor
Venue: WACV 2025 (main)

Abstract: We introduce ShapeMorph a diffusion-based method specifically designed for generating precise and diverse 3D shape completions. By integrating an irregular discrete representation with a novel blockwise discrete diffusion model ShapeMorph can produce multiple high-quality shape completions while maintaining fidelity to the input. In particular each 3D shape is encoded into a compact sequence of irregularly distributed discrete variables ensuring an accurate capture of the object's topological details. We then propose a blockwise discrete diffusion model to precisely learn the shape completion distribution based on various incompleteness. We also introduce a Flow transformer into our diffusion process serving as a denoising network to enhance the modeling adaptability and flexibility. ShapeMorph addresses common challenges in existing methods such as poor completion limited diversity and misalignment with the input. Results show that ShapeMorph outperforms state-of-the-art methods and effectively processes a variety of input types and levels of incompleteness.
NAT: Learning to Attack Neurons for Enhanced Adversarial Transferability
Authors: Krishna Kanth Nakka; Alexandre Alahi
Venue: WACV 2025 (main)

Abstract: The generation of transferable adversarial perturbations typically involves training a generator to maximize embedding separation between clean and adversarial images at a single mid-layer of a source model. In this work we build on this approach and introduce Neuron Attack for Transferability (NAT) a method designed to target specific neuron within the embedding. Our approach is motivated by the observation that previous layer-level optimizations often disproportionately focus on a few neurons representing similar concepts leaving other neurons within the attacked layer minimally affected. NAT shifts the focus from embedding-level separation to a more fundamental neuron-specific approach. We find that targeting individual neurons effectively disrupts the core units of the neural network providing a common basis for transferability across different models. Through extensive experiments on 41 diverse ImageNet models and 9 fine-grained models NAT achieves fooling rates that surpass existing baselines by over 14% in cross-model and 4% in cross-domain settings. Furthermore by leveraging the complementary attacking capabilities of the trained generators we achieve impressive fooling rates within just 10 queries. Our code is available at: https://krishnakanthnakka.github.io/NAT/
GEXIA: Granularity Expansion and Iterative Approximation for Scalable Multi-Grained Video-Language Learning
Authors: Yicheng Wang; Zhikang Zhang; Jue Wang; David Fan; Zhenlin Xu; Linda Liu; Xiang Hao; Vimal Bhat; Xinyu Li
Venue: WACV 2025 (main)

Abstract: In various video-language learning tasks the challenge of achieving cross-modality alignment with multi-grained data persists. We propose a method to tackle this challenge from two crucial perspectives: data and modeling. Given the absence of a multi-grained video-text pretraining dataset we introduce a Granularity EXpansion (GEX) method with Integration and Compression operations to expand the granularity of a single-grained dataset. To better model multi-grained data we introduce an Iterative Approximation Module (IAM) which embeds multi-grained videos and texts into a unified low-dimensional semantic space while preserving essential information for cross-modal alignment. Furthermore GEXIA is highly scalable with no restrictions on the number of video-text granularities for alignment. We evaluate our work on three categories of video tasks across seven benchmark datasets showcasing state-of-the-art or comparable performance. Remarkably our model excels in tasks involving long-form video understanding even though the pretraining dataset only contains short video clips.
MoRAG - Multi-Fusion Retrieval Augmented Generation for Human Motion
Authors: Sai Shashank Kalakonda; Shubh Maheshwari; Ravi Kiran Sarvadevabhatla
Venue: WACV 2025 (main)

Abstract: We introduce MoRAG a novel multi-part fusion based retrieval-augmented generation strategy for text-based human motion generation. The method enhances motion diffusion models by leveraging additional knowledge obtained through an improved motion retrieval process. By effectively prompting large language models (LLMs) we address spelling errors and rephrasing issues in motion retrieval. Our approach utilizes a multi-part retrieval strategy to improve the generalizability of motion retrieval across the language space. We create diverse samples through the spatial composition of the retrieved motions. Furthermore by utilizing low-level part-specific motion information we can construct motion samples for unseen text descriptions. Our experiments demonstrate that our framework can serve as a plug-and-play module improving the performance of motion diffusion models. Code pre-trained models and sample videos are available at https://motion-rag.github.io.
PositiveCoOp: Rethinking Prompting Strategies for Multi-Label Recognition with Partial Annotations
Authors: Samyak Rawlekar; Shubhang Bhatnagar; Narendra Ahuja
Venue: WACV 2025 (main)

Abstract: Vision-language models (VLMs) like CLIP have been adapted for Multi-Label Recognition (MLR) with partial annotations by leveraging prompt-learning where positive and negative prompts are learned for each class to associate their embeddings with class presence or absence in the shared vision-text feature space. While this approach improves MLR performance by relying on VLM priors we hypothesize that learning negative prompts may be suboptimal as the datasets used to train VLMs lack image-caption pairs explicitly focusing on class absence. To analyze the impact of positive and negative prompt learning on MLR we introduce PositiveCoOp and NegativeCoOp where only one prompt is learned with VLM guidance while the other is replaced by an embedding vector learned directly in the shared feature space without relying on the text encoder. Through empirical analysis we observe that negative prompts degrade MLR performance and learning only positive prompts combined with learned negative embeddings (PositiveCoOp) outperforms dual prompt learning approaches. Moreover we quantify the performance benefits that prompt-learning offers over a simple vision-features-only baseline observing that the baseline displays strong performance comparable to dual prompt learning approach (DualCoOp) when the proportion of missing labels is low while requiring half the training compute and 16 times fewer parameters.
AIC3DOD: Advancing Indoor Class-Incremental 3D Object Detection with Point Transformer Architecture and Room Layout Constraints
Authors: Zhongyao Cheng; Fang Wu; Peisheng Qian; Ziyuan Zhao; Xulei Yang
Venue: WACV 2025 (main)

Abstract: Over the recent years there has been a growing interest in class-incremental 3D object detection based on point clouds. However the current state-of-the-art (SOTA) methods still fall short of practical adoption mainly due to two key observations. Firstly existing SOTA methods are limited by the capability of feature representation from the object detection model. Secondly these methods overlook the importance of incorporating prior information or geometry constraints which are crucial elements for 3D point cloud tasks. In this study we strive to enhance the performance of class-incremental 3D object detection for indoor scenes by proposing AIC3DOD - Advancing Indoor Class-incremental 3D Object Detection using the point transformer architecture with room layout constraints. Our approach employs a transformer architecture in our detection model and optimizes the class incremental step in the transformer architecture. Besides AIC3DOD incorporates additional prior information namely room layout to impose physical constraints on detected objects thereby enhancing overall object detection performance. Extensive experimental results on the ScanNet dataset demonstrate the effectiveness of our approach showcasing our superior performance compared to other SOTA methods in the class-incremental 3D object detection task.
PACA: Prespective-Aware Cross-Attention Representation for Zero-Shot Scene Rearrangement
Authors: Shutong Jin; Ruiyu Wang; Kuangyi Chen; Florian T. Pokorny
Venue: WACV 2025 (main)

Abstract: Scene rearrangement like table tidying is a challenging task in robotic manipulation due to the complexity of predicting diverse object arrangements. Web-scale trained generative models such as Stable Diffusion can aid by generating natural scenes as goals. To facilitate robot execution object-level representations must be extracted to match the real scenes with the generated goals and to calculate object pose transformations. Current methods typically use a multi-step design that involves separate models for generation segmentation and feature encoding which can lead to a low success rate due to error accumulation. Furthermore they lack control over the viewing perspectives of the generated goals restricting the tasks to 3-DoF settings. In this paper we propose PACA a zero-shot pipeline for scene rearrangement that leverages perspective-aware cross-attention representation derived from Stable Diffusion. Specifically we develop an object-level representation that integrates generation segmentation and feature encoding into a single step. Additionally we introduce perspective control thus enabling the matching of 6-DoF camera views and extending past approaches that were limited to 3-DoF top-down settings. The efficacy of our method is demonstrated through its zero-shot performance in real robot experiments across various scenes achieving an average matching accuracy and execution success rate of 87% and 67% respectively.
SUM: Saliency Unification through Mamba for Visual Attention Modeling
Authors: Alireza Hosseini; Amirhossein Kazerouni; Saeed Akhavan; Michael Brudno; Babak Taati
Venue: WACV 2025 (main)

Abstract: Visual attention modeling important for interpreting and prioritizing visual stimuli plays a significant role in applications such as marketing multimedia and robotics. Traditional saliency prediction models especially those based on Convolutional Neural Networks (CNNs) or Transformers achieve notable success by leveraging large-scale annotated datasets. However the current state-of-the-art (SOTA) models that use Transformers are computationally expensive. Additionally separate models are often required for each image type lacking a unified approach. In this paper we propose Saliency Unification through Mamba (SUM) a novel approach that integrates the efficient long-range dependency modeling of Mamba with U-Net to provide a unified model for diverse image types. Using a novel Conditional Visual State Space (C-VSS) block SUM dynamically adapts to various image types including natural scenes web pages and commercial imagery ensuring universal applicability across different data types. Our comprehensive evaluations across five benchmarks demonstrate that SUM seamlessly adapts to different visual characteristics and consistently outperforms existing models. These results position SUM as a versatile and powerful tool for advancing visual attention modeling offering a robust solution universally applicable across different types of visual content. Our codebase and pretrained models are publicly accessible on the https://arhosseini77.github.io/sum_page/.
Adversarial Learning Based Knowledge Distillation on 3D Point Clouds
Authors: Sanjay S J; Akash J; Sreehari Rajan; Dimple A Shajahan; Charu Sharma
Venue: WACV 2025 (main)

Abstract: The significant improvements in point cloud representation learning have increased its applicability in many real-life applications resulting in the need for lightweight better-performing models. One widely proposed efficient method is knowledge distillation where a lightweight model uses knowledge from large models. Very few works exist on distilling the knowledge for point clouds. Most of the work focuses on cross-modal-based approaches that make the method expensive to train. This paper proposes PointKAD an adversarial knowledge distillation framework for point cloud-based tasks. PointKAD includes adversarial feature distillation and response distillation with the help of discriminators to extract and distill the representation of feature maps and logits. We conduct extensive experimental studies on both synthetic (ModelNet40) and real (ScanObjectNN) datasets to show that PointKAD achieves state-of-the-art results compared to the existing knowledge distillation methods for point cloud classification. Additionally we present results on the part segmentation task highlighting the efficacy of the PointKAD framework. Our experiments further reveal that PointKAD is capable of transferring knowledge across different tasks and datasets showcasing its versatility. Furthermore we demonstrate that PointKAD can be applied to a cross-modal training setup achieving competitive performance with cross-modal-based point cloud methods for classification.
Shift Equivariant Pose Network
Authors: Pengxiao Wang; Tzu-Heng Lin; Chunyu Wang; Yizhou Wang
Venue: WACV 2025 (main)

Abstract: Human pose estimation has been greatly advanced in recent years. However even the best-performing models are not shift equivariant. In particular a small change in input images often results in drastic alterations in output which are problematic especially in video applications. The prevalence of top-down approaches which typically rely on a non-equivariant object detector in the first stage exacerbates this issue. In this paper we first demonstrate that the biased keypoint representation and the non-equivariant network components are the two main obstacles to shift equivariant pose estimation. To address the limitation we propose an unbiased decoding method and redesign the necessary network components (e.g. APS-ResBlock SSP). Extensive experiments show that our method not only produces much more stable results with shifting input but also achieves better metrics with the ability to tolerate inaccurate detector output from the first stage. To our knowledge this is the first work to address the problem of shift equivariance in the field of pose estimation. Our method could be easily applied to existing CNN-based pose estimation networks.
Relaxing Binary Constraints in Contrastive Vision-Language Medical Representation Learning
Authors: Xiaoyang Wei; Camille Kurtz; Florence Cloppet
Venue: WACV 2025 (main)

Abstract: By aligning paired image and caption embeddings as input contrastive vision-language representation learning has witnessed significant advances as illustrated by CLIP allowing visual encoders to learn from textual supervision and vice versa. Benefiting from millions of image-caption pairs collected from the Internet CLIP-like models show competitive performances against fully supervised baselines. However the learned visual representations are still undermined due to the binary constraint as most contrastive learning frameworks follow strict one-to-one correspondence for the input pairs of data and optimize the models using the InfoNCE loss function. The embeddings of the paired image-text are aligned while the unpaired image-text are pushed away from each other. In fact there are naturally many "false negatives" among these negative pairs since unpaired data can also have a high similarity. In this work we aim to overcome the impact of false negatives in vision-language representation learning by introducing soft targets for estimating the similarity between unpaired images and texts using external semantic knowledge structured in the form of graphs. The interest of such a method is demonstrated in the application context of medical imaging.
Few-Shot Structure-Informed Machinery Part Segmentation with Foundation Models and Graph Neural Networks
Authors: Michael Schwingshackl; Fabio F. Oberweger; Markus Murschitz
Venue: WACV 2025 (main)

Abstract: This paper proposes a novel approach to few-shot semantic segmentation for machinery with multiple parts that exhibit spatial and hierarchical relationships. Our method integrates the foundation models CLIPSeg and Segment Anything Model (SAM) with the interest point detector SuperPoint and a graph convolutional network (GCN) to accurately segment machinery parts. By providing 1 to 25 annotated samples our model evaluated on a purely synthetic dataset depicting a truck-mounted loading crane achieves effective segmentation across various levels of detail. Training times are kept under five minutes on consumer GPUs. The model demonstrates robust generalization to real data achieving a qualitative synthetic-to-real generalization with a J&F score of 92.2 on real data using 10 synthetic support samples. When benchmarked on the DAVIS 2017 dataset it achieves a J&F score of 71.5 in semi-supervised video segmentation with three support samples. This method's fast training times and effective generalization to real data make it a valuable tool for autonomous systems interacting with machinery and infrastructure and illustrate the potential of combined and orchestrated foundation models for few-shot segmentation tasks.
EasyRet3D: Uncalibrated Multi-View Multi-Human 3D Reconstruction and Tracking
Authors: Junjie Oscar Yin; Ting Li; Jiahao Wang; Yi Zhang; Alan Yuille
Venue: WACV 2025 (main)

Abstract: Current methods performing 3D human pose estimation from multi-view still bear several key limitations. First most methods require manual intrinsic and extrinsic camera calibration which is laborious and difficult in many settings. Second more accurate models rely on further training on the same datasets they evaluate severely limiting their generalizability in real-world settings. We address these limitations with EasyRet3D (Easy REconstruction and Tracking in 3D) which simultaneously reconstructs and tracks 3D humans in a global coordinate frame across all views with uncalibrated cameras and videos in the wild. EasyRet3D is a compositional framework that composes our proposed modules (Automatic Calibration module Adaptive Stitching Module and Optimization Module) and off-the-shelf large pre-trained models at intermediate steps to avoid manual intrinsic and extrinsic calibration and task-specific training. EasyRet3D outperforms all existing multi-view 3D tracking or pose estimation methods in Panoptic EgoHumans Shelf and Human3.6M datasets. Codebase and demos will be released on the project website.