Title: SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Audio-Visual

3D Spatial Reasoning

Multi-modal LLMs

Audio-Visual QA
Abstract: 3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition yet remains largely unexplored by existing Audio-Visual Large Language Models (AV-LLMs) and benchmarks, which predominantly focus on static or 2D scenes. We introduce SAVVY-Bench, the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio. SAVVY-Bench is comprised of thousands of carefully curated question–answer pairs probing both directional and distance relationships involving static and moving objects, and requires fine-grained temporal grounding, consistent 3D localization, and multi-modal annotation. To tackle this challenge, we propose SAVVY, a novel training-free reasoning pipeline that consists of two stages: (i) Egocentric Spatial Tracks Estimation, which leverages AV-LLMs as well as other audio-visual methods to track the trajectories of key objects related to the query using both visual and spatial audio cues, and (ii) Dynamic Global Map Construction, which aggregates multi-modal queried object trajectories and converts them into a unified global dynamic map. Using the constructed map, a final QA answer is obtained through a coordinate transformation that aligns the global map with the queried viewpoint. Empirical evaluation demonstrates that SAVVY substantially enhances performance of state-of-the-art AV-LLMs, setting a new standard and stage for approaching dynamic 3D spatial reasoning in AV-LLMs.


Peer reviews:
comments: SUMMARY:
This paper introduced a benchmark called SAVVY-bench that evaluate the dynamic audio-visual reasoning capabilities of models. They also introduced a training-free pipeline that augments existing AV-LLMs to achieve state-of-the-art performance on SAVVY-bench.

STRENGTHS AND WEAKNESSES:
Strengths:
1. audio-visual spatial reasoning through time has been a missing piece in LLM evaluation, and this paper does exactly that. The task construction is based on real world dataset rather than simulated environment, which make it more reliable. 
2. The construction of the data is rigorously documented in main paper and supp. It's an impressive amount of work.
3. The proposed pipeline improve baseline Gemini 2.5 pro significantly without any training

Weakness:
1. Not really a weakness, but the approach SAVVY basically shows that compared to using LLM E2E, we get better results if we applied traditional methods in segmentation, mapping and localization to extract information (and using LLM also just as a feature extractor). It's hard to pin point the novelty in the approach, as it's a combination of existing methods.

QUESTIONS:
1. what is the bottleneck for models on this benchmark at this point?

LIMITATIONS:
no
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-08

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors design a benchmark, SAVVY-Bench, to evaluate the spatial reasoning capabilities of audio-visual large language models (AV-LLMs). This is the first work to incorporate spatial audio as a condition in spatial perception tasks for LLMs. The benchmark evaluates existing AV-LLMs from both egocentric and allocentric perspectives, across both direction and distance dimensions. The authors also propose SAVVY, a training-free multimodal spatial reasoning system built on top of a state-of-the-art AV-LLM. SAVVY integrates three distinct spatial direction and distance estimation methods: Snapshot Descriptor, Text-Guided Snapshot Segmentation, and Spatial Audio Cues. The outputs from these three methods are denoised (outlier removal) and aggregated to construct egocentric spatial tracks, which are then used to build a Dynamic Global Map.\n\nIn experiments, the authors demonstrate that SAVVY outperforms all existing state-of-the-art AV-LLMs on the SAVVY-Bench benchmark. However, all AV-LLM systems, including SAVVY, still lag behind human performance in spatial perception tasks, indicating that there remains significant room for improvement in audio-visual spatial reasoning for LLMs. Ablation studies further validate that the three proposed methods for egocentric spatial track construction each contribute uniquely to different aspects of spatial perception. Moreover, the experiments show that combining all three methods yields the best overall performance in audio-visual spatial reasoning.\n\nSTRENGTHS AND WEAKNESSES:\na) Strengths\n1. The authors are the first to introduce spatial audio into spatial reasoning for LLMs. From the perspective of multimodal spatial perception, integrating both audio and visual spatial cues aligns more closely with the way humans perceive space in real life. This task is both meaningful and novel.\n2. The SAVVY-Bench constructed by the authors has undergone meticulous data preprocessing, annotation, QA synthesis, and quality review processes. The methodology used to build the benchmark is rigorous.\n3. The proposed SAVVY system estimates spatial direction and distance from the vision perspective using SD and Seg methods, and from the audio perspective using existing spatial audio localization and distance estimation models. The system effectively removes outliers and integrates information from both vision and audio, making excellent use of both modalities and achieving performance that surpasses the state-of-the-art AV-LLMs.\n4. In terms of both writing and experimentation, the authors provide comprehensive quantitative and qualitative analysis.\n\nb) Weaknesses\n1. The question types in SAVVY-Bench are relatively monotonous, limited to only egocentric/allocentric and direction/distance, resulting in the evaluation of a rather narrow spatial reasoning task.\n2. The system is described as a training-free approach, which is a strength. However, this may come with computational trade-offs. It would be useful to discuss the computational cost of the SAVVY system in comparison to other state-of-the-art models, especially since it integrates multiple methods (e.g., Snapshot Descriptors, Segmentation, and Audio Cues). This could be particularly important for practical deployment in real-time systems.\n\nQUESTIONS:\n1. In Table 6, the spatial reasoning capabilities of the combinations SD+Seg and Audio+Seg are not compared. It would be helpful if the authors could include results for these two cases. There is a possibility that one of the methods used for egocentric tracks construction might have a significant negative impact, causing the combination of all three methods to perform worse than the combination of just two methods.\n\n2. In Table 5, the spatial direction and distance estimation performance of Audio is worse than SD. However, Table 6 shows that Audio outperforms SD in terms of sound localization accuracy and egocentric direction and distance estimation. Are these two results in conflict? Could the authors provide an analysis of this discrepancy?\n\n3. Is there a computational overhead comparison between SAVVY and other methods? The pipeline combines SLAM, AV-LLMs, image segmentation, and audio processing. Latency/throughput metrics are absent, making real-time deployment feasibility unclear.\n\nLIMITATIONS:\n1. The question types in SAVVY-Bench are relatively monotonous, with only 4 types of questions included.\n\n2. The computational overhead of SAVVY could be significant because of the combination of so many foundation models.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduce the the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio: SAVVY-Bench, and and a fully training-free two-stage pipeline that leverages off-the-shelf components \n*(pretrained audio-visual LLMs for timestamped snapshot extraction, segmentation and monocular depth networks for object localization, SRP-PHAT/CDR for audio direction-of-arrival and distance estimation, SLAM-based camera poses for global mapping, DBSCAN clustering for static object centroids, and Kalman filtering for dynamic trajectory smoothing)*\nto accurately answer “which way?” and “how far?” questions in moving egocentric video with spatial audio, all without any additional model fine-tuning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-organized and easy to follow.\n- Benchmark (3D spatial QA with multi-channel audio) is novelty.\n- Training-free design is good. Leverages existing models and signal-processing methods without additional fine-tuning.\n- Evaluations and ablation studies are comprehensive, which can well support the paper's main point.\n\nWeaknesses:\n- Limited scope of spatial reasoning: SAVVY-Bench only tests “which way?” and “how far?” Q&A. It doesn’t cover richer 3D reasoning tasks such as mapping unknown space, occlusion handling, trajectory prediction, semantic layout, or path planning, which limits its applicability as a general spatial reasoning benchmark.\n- Pipeline Complexity: Even though it's training-free, it seems to stitch many heavy components: LLM, CLIPSeg/SAM2, depth, SLAM, SRP-PHAT, DBSCAN, Kalman. Which lead to hard to deploy and maintain.\n- The paper omits inference time or resource usage, lacking assessment of real-time feasibility. \n- Data is limited to indoor Aria headsets in controlled environments; outdoor generalization is unclear.\n\nQUESTIONS:\n- Can the author report end-to-end latency (per QA) and peak memory/GPU utilization on representative hardware?\n- Has the author tested performance on non-Aria or outdoor recordings? It would be great if the author could report the metrics.\n- Can the author quantify the accuracy of your zero-shot LLM-based event timestamps? I feel like this misalignment here will affect downstream spatial accuracy.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a new dataset, named as SAVVY-Bench, for visual-audio based 3d spatial reasoning in dynamic audio-visual environments. The problem introduced by the benchmark is interesting and important for community. Besides, the authors also proposed a novel pipeline SAVVY, including egocentric spatial tracks estimation stage and dynamic global map construction for visual-audio based 3d spatial reasoning with AV-LLM. The results on SAVVY-Bench demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThis paper addresses a very interesting and important problem: how to perform 3D reasoning in dynamic environments with multimodal inputs, specifically audio and text. The authors provide a thorough analysis of this problem and propose a benchmark to facilitate further study. In addition, they introduce a localization method that leverages large models. The final results demonstrate the effectiveness of the proposed approach.\n\nWeaknesses:\n1）This paper mainly focuses on the audio modality. However, in real-world 3D reasoning tasks, visual signals play a crucial role. The paper lacks sufficient comparison between audio and visual signals—for example, the advantages of using audio signals versus purely visual signals, and whether combining audio and visual signals provides complementary benefits. It is also unclear whether a purely video-based approach would already be sufficient to address the challenges raised by the benchmark. More analysis is needed in this regard.\n2）The reasoning tasks discussed in this paper are primarily oriented toward navigation and spatial localization. It would be interesting to explore whether audio signals can support higher-level semantic reasoning tasks, such as object grounding via sound, scene-level question answering (e.g., “How many people are there?”), or attribute reasoning (e.g., distinguishing between male and female voices).\n3）The generalizability of the proposed method needs further evaluation. Since the method is training-free, it is important to investigate whether it can be applied to outdoor environments or driving scenarios.\n\nQUESTIONS:\n1）The main question I considered is the synergy between visual and audio signals. Many of the tasks presented in the paper can be solved using visual signals alone, so it is important to clarify what complementary information the audio signals provide comparing with the method using videos to build the scene and preform 3d reasoning, and how much additional value they contribute to the overall task.\n2）In this benchmark, there are many 3D scene reasoning tasks. In this context, would it be possible to incorporate step-by-step reasoning similar to Chain-of-Thought approaches? Moreover, could multi-round reasoning be applied in this process? Multi-round reasoning is particularly important for embodied tasks.\n\nLIMITATIONS:\nThe main limitation of this paper is that it only compares reasoning performance based on large language models. To some extent, this is not entirely fair. The mapping and scene construction methods proposed in the paper could potentially be completed using only video input. Moreover, the relevant information extracted from these processes could also be incorporated into large model-based reasoning. A more detailed and comprehensive comparison is needed to better evaluate the effectiveness of the proposed approach.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-05"}