SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing
Authors: Mingfei Chen
Zijun Cui
Xiulong Liu
Jinlin Xiang
Caleb Zheng
Jingyuan Li
Eli Shlizerman
Venue: NeurIPS 2025 oral ()
Keywords: Audio-Visual
3D Spatial Reasoning
Multi-modal LLMs
Audio-Visual QA
Primary area: applications
TL;DR: A novel 3D audio-visual QA benchmark and training-free spatial reasoning pipeline for Audio-Visual LLMs

Abstract: 3D spatial reasoning in dynamic, audio-visual environments is a cornerstone of human cognition yet remains largely unexplored by existing Audio-Visual Large Language Models (AV-LLMs) and benchmarks, which predominantly focus on static or 2D scenes. We introduce SAVVY-Bench, the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio. SAVVY-Bench is comprised of thousands of carefully curated question–answer pairs probing both directional and distance relationships involving static and moving objects, and requires fine-grained temporal grounding, consistent 3D localization, and multi-modal annotation. To tackle this challenge, we propose SAVVY, a novel training-free reasoning pipeline that consists of two stages: (i) Egocentric Spatial Tracks Estimation, which leverages AV-LLMs as well as other audio-visual methods to track the trajectories of key objects related to the query using both visual and spatial audio cues, and (ii) Dynamic Global Map Construction, which aggregates multi-modal queried object trajectories and converts them into a unified global dynamic map. Using the constructed map, a final QA answer is obtained through a coordinate transformation that aligns the global map with the queried viewpoint. Empirical evaluation demonstrates that SAVVY substantially enhances performance of state-of-the-art AV-LLMs, setting a new standard and stage for approaching dynamic 3D spatial reasoning in AV-LLMs.

SAVVY: Spatial Awareness via Audio Visual LLMs through Seeing and Hearing
Mingfei Chen
Zijun Cui
Xiulong Liu *
Jinlin Xiang
Yang Zheng
Jingyuan Li
Eli Shlizerman #15
Abstract
3D spatial reasoning in dynamic, audio-visual environments is cornerstone of human cognition yet remains largely unexplored by existing Audio-Visse Large Language Models (AV-LLMs) and benchmarks, which predominantly focus static or 2D scenes. We introduce SAVVY- the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio. SAVVY-Bench is comprised of thousands of carefully curated question-answer pairs probing both directional and distance relationships involving static and moving objects, and requires fine- teniggrined temporal grounding, consistent 3D localization, and multi- modal annotation. To tackle this challenge, we propose SAVVY, a novel training- free reasoning pipeline that consists of two stages: (i) Egocentric Spatial Tracks Estimation, which leverages AV-LLMs well other audio-visual methods track the trajectories of key objects related to the query using both visual and spatial audio cues, and (ii) Dynamic Global Map Construction, which aggregates multi- modal queried object trajectories and converts them into unified global dynamic map. Using the constructed map. final QA answer is obtained through a coordinate transformation that aligns the global map with the queried viewpoint. Empirical evaluation demonstrates that SAVVY substantially enhances performance of state- of-the-art AV-LLMs, setting a new standard and stage for approaching dynamic 3D spatial reasoning in AV-LLMs.
-ip "This (from) Phoenix local artist."
Egocentric QA
Allocentric QA
Imagine you
Imagine
camera wearer, when the speech topic 'identifying the artist ghaenix loral" comes up, you facing, where the other person: front-left, front-right, back-left, back-right?
robot standing by the cabinet facing the TV, when the speech topic: 'icentiiknng the ahoenix loca!" comes up, relative where you facing, where is the spea ker: left, right, back?
Speech Topic
A: backeright.
A: right.
Figure 1: 3D spatial reasoning in dynamic audio-vishe environments. The task requires fine-grained 3D question answering across egocentric and allocentric frames dynamic scenes.
"These authors contributed equally.
Deparrment of Ellcctical & Computer Engineering, University of Washington, Seattle, USA. Deparrmeent of Applied Mathematics, University of Washington, Scattle, USA *Corresonding author: shlizee@uw.edht
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
3D spatial reasoning in dynamic scenes is a core aspect of human intelligence, allowing us to navigate and understand changing environments. Imagine watching an egocentric video where a person wearing a head-mounted camera guides someone through a multi-room apartment. During such navigation, a question could arise as shown in "Allocentric QA" denoted in Figure T To answer such question, a human would engage in several mental processes: (i) identify the moments when the referenced speech event occurs and locate the relevant objects ("China Cabinet", "TV", "Speaker") in space; (ii) convert egocentric observations into an allocentric map anchored at the China Cabinet and oriented towards the TV; and (iii) mentally compute the Speaker's s position within this allocentric map. While humans perform these steps naturally, they are cognitively demanding, especially under dynamic, shifting viewpoints, as demonstrated in early human cognitive studies [.]. This raises a key question: can existing foundation models such as Multi-Modal LLMs (MLLMs) reason about dynamic 3D scenes with spatial intelligence?
Despite growing interest in grounding foundation models in 3D environments, most existing works remain limited static scenes. Indeed, previous spatial reasoning benchmarks 2 targeted static visual enviromments with no moving objects. However, real-world scenarios are usually dynamic and involve diverse moving objects and sounds. Existing foundation models that support spatial reasoning in 3D such as 4 2 5 assume a static world, and thus cannot generalize to dynamic scenarios. Moreover, they rely exclusively on visual input, neglecting the critical role of spatial audio in capturing semantics and spatial cues beyond the visual field. These limitations highlight the need for benchmarks and models capable of dynamic 3D spatial reasoning across both audio and visual modalities. We refer to such models Audio- LLMs (AV-LLMs), which are MLLMs that jointly reason over audio and visual inputs.
To fill these gaps, we introduce BAVVY-Bench. first-of- its kind benchmark designed for 3D spatial reasoning in dynamic scenes for AV-LLMs. A key feature of SAVVY-Bench is its coverage of both egocentric and allocentric question types: some questions require reasoning from the camera wearer's s viewpoint (egocentric), while others rely on fixed external references (allocentric), as depicted in Figure D SAVVY-Bench comprises thousands of QA pairs that probe spatial relationships involving both static and dynamic objects, focusing on distance and directional aspects. In terms of modalities, SAVVY-Bench targets audio-visual question answering with strong emphasis on moving objects. To support fine-grained spatial reasoning, we incorporate multi-channel audio that captures directional information beyond what is visible in the video.
Beyond benchmark construction, enabling effective spatial reasoning in 3D dynamic scenes remains a challenge. We impose that an effective AV-LLM for reasoning for such environments must (i) achieve robust temporal grounding to locate keyframes and detect relevant objects, (ii) develop spatial perception in both visual and auditory (spatial audio) to track locations of objects from egocentric views, and (iii) transform egocentric observations into consistent global coordinate frame to accurately reason about spatial relationships. Existing lidoo-langaage models unable to fully incorporate spatial reasoning and
perspective transformation even in static visual scenes (3), while dynamic 3D environments add further complexity, which require tracking the state of moving objects. Moreover, existing AV-LLMs typically rely on monaural audio input. This reliance limits access to spatial audio cues and restricts the model's ability to support human-like spatial understanding.
To support these capabilities, we introduce SAVVY, training-free pipeline that augments AV-LLMs with structured spatial reasoning, integrating spatial audio cues and egocentric-to-glon operates in two stages: (i) Extracting sparse "snapshot" descriptions of key events and objects via an AV-LLM, and constructing egocentric tracks by estimating object direction and distance relative to the camera from video and spatial audio; these tracks align auditory and visual signals at key timestamps relevant to the query. (ii) Aggregating these tracks into a dynamic global map for accurate reasoning over both egocentric and allocentric queries. We perform experiments with the proposed pipeline on SAVVY-Bench. Extensive experiments demonstrate that SAVVY performs best in comparison to existing state-of-the-ar AV-LLMs.
mapping.
To summarize our contributions: (i) We introduce SAVVY-Bench, the first spatial reasoning bench- mark for dynamic 3D scenes, with an integration of both (spatial) audio and visual modality. (ii) We propose a training-free pipeline that augments existing AV-LLMS with strong spatial reasoning capabilities. (iii) Experiments on SAVVY-Bench show that significantly outperforms existing AV-

---

Dynamic Scene
Cross-Room Spatial QA
Dataset
Modality
Allocentric
Direction
Distance
EgoSchema 6
x
K
X
K
X
OpenEQA
X
x
x
MUSIC-AVQA
A+V
X
X
v
X
VSI-Bench
X
Ego4D-AVD 9
A+V
V
X
X
V
X
SAVVY-Bench (Ours)
A+V
Table 1: Comparison of SAVVY-Bench with other Visual and Audio-Visual Benchmarks. SAVVY-Bench focuses on spatial relations (distance and direction) among objects to evaluate 3D spatial reasoning in large and dynamic audio-visual scenes.
LLMs dynamic spatial QA task, with significant improvement of +7.1% % overall QA accuracy against even the best performing AV-LLMs (Gemini-2.5 Pro).
Related Works
2.1 Multi-modal Large Language Models for Spatial Reasoning
Recent advances in Multi-modal Large Language Models (MLLMs) have extended language models to process visual [TO] II 12
T3
14
16 and audio [17 T8 19 20] modalities, giving rise to 26
27 28). However, most MLLMs and AV-LLMs remain limited in spatial reasoning capabilities. While some models incorporate basic 2D localization 22 30 31, spatial reasoning remains largely unaddressed due to reliance 2D training data and the lack of large-scale 3D annotations. Recent efforts incorporate 3D information via point clouds 2 32], or spatial scene representations such graphs
Audio-Visual LLMs (AV-LLMs)
21
22
23 24
25
4 33 34], voxel grids B35 5 37), maps 38 3 and neural fields [39 40). However, these models are limited to static environments
and do not support dynamic scenes.
Moreover, visual cues alone are insufficient for spatial reasoning. dynamic scenes, where objects leave the visual field, spatial audio provides critical cues for localization which is missing in existing AV-LLMS [41] since these downmix multi-channel audio to mono and discard spatial information. Ex- tracting spatial cues from audio remains challenging due to the complexity of real-woor world soundscapes and the lack of high-quality, localized annotations. While learning-
spatial audio localization exist, they trained on synthetic data with specific receiver configurations, and do not generalize to real-world environments that are typically noisy and reverberant.
methods 42
43 44
2.2 Benchmarks for Multi-Modal Understanding and Reasoning
Existing benchmarks for evaluating MLLMs primarily focus on semantic understanding from 6]. For image inputs, benchmarks such as assess reasoning across diverse domains but do
images 45
46
47
or video inputs [48]
49
50
51
MMBench 47 MMMU [46), and MM- Vet 445
not address temporal aspects. Video-based benchmarks like MVBench [4), EgoTaaskA [51].
focus on event- based or temporal concept understanding in either exocentric or egocentric views, while they do not address 3D spatial relation- ships in dynamic scenes. When the modality extends to both visual and audio, benchmarks such as MUSIC-AVQA 8 (55, Ego4D AV Diarization
EgoSchema (6] and additional works 52
53
54
[ and other [56] 57 58 22 address sounding events as well as spatial relationships between sounding objects in 2D image plane, without addressing 3D relations. Benchmarks such as ScanQA [59] and OpenEQA 7 introduce spatial reasoning in 3D environments. These focus solely on static layouts and coarse spatial relations. The closest benchmark to SAVVY-Bench is the VSI-Bench (3). which leverages 3D information for fine-grained visual spatial reasoning, but is constrained to static scenes only. In contrast, SAVVY-Bench the first benchmark for oudio-visual spatial reasoning in dynamic scenes. Table illustrates detailed comparison with related benchmarks.
SAVVY-Bench
3.1 Overview
SAVVY-Bench the first benchmark for evaluation of 3D spatial reasoning of AV-LLMS in dy- namic, multi-room scenes. It builds manually selected subset of the Aria Everyday Activities

---

Egacentriic Direction
Drnzention
25.9%
Epocentric Distance 11.6%
275
15.0%
Allocentric Direction
20.0%
Allocentric Distance 18.4%
187
80%
3mam 3m bm
3.0%
QA Task Distribution (b) Angle Distribution
(c) Distance Distribution (d) Video Duration Distribution
Figure 2: Benchmark Statistics. (a) Task distribution by type. (b) Angle distribution of egocentric
queries over 360". (c) Distribution of egocentric query distances. (d) Video duration distribution.
Dataset(AEA), Meta Reality Labs-R [60], which includes 600 sound events across 58 daily-life scenarios, Each scenario provides the synchronized visual input and the spatial audio captured by 7-microphone array Aria glasses. SAVVY- poses queries spatial relations among moving and static entities in 3D space.
Task Taxonomy. SAVVY-Bent Y-Bechh defines 4 spatial-relationa QA tasks across two reference frames: egocentric (camera-centered) and allocentric (object-centered) (examples shown in Figure [ Each question is anchored to sound event and requires reasoning regarding the relative direction and the absolute distance between a sounding object and a reference point. In egocentric tasks, the reference is the camera wearer; In allocentric tasks, it is hypothetical robot positioned net to single static object and faces another object. Directional reasoning is posed as a multiple- choice question, offering 3 options (left, right, back) for simpler layouts and options (front-left. front-right, back-left, sack-right for more complex ones. Distance reasoning requires to provide a numeric estimate the distance in meters.
Statistics. Figure 2(a) shows the distribution of QA tasks: Egocentric Direction (30.4%), Egocentric Distance (1 (11.6%). Allocentric Distance (18.4%), and Allocentric Direction (39.6%). Relative direc- including the challenging cases that involve
tion questions cover the full 360 azimuth (Figure rear angles (90-270")in Egocentric QA, where the target sounding object is not within the camera view. Distance values range from <0.5 to 9 meters (Figure c))) Video durations span from 30 to 300s (Figure 2 d)).
3.2 Benchmark Construction
We develop systematic data pipeline to generate high-quality question-answe pairs for SAVVY- Bench. The pipeline includes four stages: Data Preprocessing. Annotation, QA Synthesis, and Quality Review. In Data Preprocessing, fisheye videos from the AEA dataset undistorted to a rectilinear format for compatibility with AV-LLM inputs. Multiview videos are temporally aligned into unified timeline, and audio is extracted into seven-channel wav file. In Annotation, we utilize proprietary AV-LLMS [41] to extract word-level transcriptions, speech topics, and sound events. Object locations are detected in 3D using EFM3D 61] and manually refined
point-cloud
interface. Human trajectories are extracted from aligned camera data, recovering both the location and the orientation for all speakers. All annotations are manually calibrated to align spatial and event data. In QA Synthesis, structured QA pairs are generated using templates applied to the annotated metadata. The Quality Review stage involves human verification to ensure that each QA pair is clear, grounded, and unambiguous. Further details are provided in Supplementary Materials.
4 SAVVY
4.1
Formulation and Overview
Given video with No spatial audio channels and natural language question Q, the goal is to predict the relative direction or absolute distance of dynamic targel object (i.e., a sounding object) during an audio event. Each question is framed from either an egocentrie (camera-centered) or allocentric (object-centered) perspective (Section B.1) To bridge multimodal input and spatial reasoning,

---

Audio
"I all1e
direction, and Reference Object (Static)
Question
Facing Object (Static)
Egocentric
Terget Object (Dynamic)
Temporal
Grounding
gas stovetop with
0:05
0:08
Video
SRP-PHAT
Location Clusteiig
Location
multippe burners
Timestamp: 0.07
Direction 75
Dittnce r: 2.0m
built-in refrigerator
qurry
d
light-colored panels
Directional
Kito
Track Track
A woman with long dark text query
Coherent-to- Ratio Distance d, - vK + CDR
X Xii
Audio Track
hali, wearling yellow vest
Dynamic Track Interpolate & smooth
Bin
Dert
(a) Snapshot Descriptor
(b) Text guided Snapshot Sagmontation
(c) Spatial Audia Cues
Stage 1: Egocentric Spatial Track Construction
Stage 2: Global Map Construction
Figure 3: SAVVY consists of two stages: Given a query and video with spatial audio, stage extracts Egocentric Spatial Tracks with (a) "Snapshot" Descriptors via AV-LLMs, (b) Text-Guided Snapshot Segmentation, and (c) Spatial Audio Cues. Stage 2 constructs a dynamic Global Map by converting egocentric tracks to global coordinates, clustering static objects, and smoothing dynamic trajectories.
introduce SAVVY, a training-free free plugin pipeline that augments AV-LLMs by extracting structured spatial information from visual, audio, and language inputs in two stages (Figure 3):
Stage 1: Egocentric Spatial Track Construction. We estimate a per-frame egocentric trajectory for each object referenced by Q, using cues from vision, language, and spatial audio. Each trajectory is defined as {tt, p.. where is the timestamp, 900 right), and is the distance in meters from the camera location.
[-180,, 180%] is the azimuth (0 front, -90 left,
Stage 2: Dynamic Global Map Construction. Egocentric tracks are projected onto 2D zy-plane using the SLAM-derived 62 camera trajectory L(t) E R p(t) = L(t) +
r=cos(0)
resn(())
The target forms global trajectory (Psound (t) E ToJ. while the reference and facing objects are treated as static, with global positions Prot and Piace computed by averaging their tracks. These define the dynamic global map: MM (Psouna(t) t E Ty U (Pret, Place To answer Q, SAVVY uses My to compute the direction and the distance of the target relative to the camera (egocentric) or in an object-centered frame.
4.2 Stage 1: Egocentric Spatial Tracks Estimation
We estimate egocentric spatial tracks with three components (Figure a), (b) and (c)):
Snapshot Descriptor. Given and video, we prompt AV-LLM once to generate structured snapshot description. The model first determines the relevant time span T, (temporal grounding) corresponding to the query-referenced event, as well as whether the question is framed egocentrically or allocentrically. It then identifies up to three object roles: the targe/ object (sound source), the reference object (anchor for allocentric frame), and the facing object (defines orientation). Out of the three roles, egocentric queries require the target object only, while allocentric queries require all three to define a third-party coordinate frame. Each object is represented by a descriptive textual phrase and egocentric trajectory, given as sequence of (t, 0, r) distance (Figure 3 a)).
Text-Guided Snapshot Segmentation. Snapshot descriptors provide sparse spatial cues and often omit intermediate frames, resulting in incomplete trajectories - particularly for dynamic objects or static objects that are visible briefly. To address this, we use visual foundation models to recover missing egocentric trajectory segments (Figure (()).. We uniformly sample N frames from the video and use the textual descriptions generated by the Snapshot Descriptor module queries for
5

---

text-guided segmentation. For each sampled frame, we segment the target, reference, or facing object using foundation models such as CLIPSeg [63] and SAM2 [64] following prior work (2). From each object mask, we compute the centroid relative to the image center to estimate the azimuth angle 0 with respect to the camera orientation. In parallel, we apply a monocular metric depth estimator [6.5] to predict the object distance r. This yields an egocentric trajectory of up to N points per object.
Spatial Audio Cues. Spatial audio provides spatial cues that complement visual input for robust tracking. We estimate both the direction and the distance of sound sources using multi-channel audio recorded by wearable microphone arrays. The method supports training- free, geometry aaare tracking in complex acoustic environments. Specifically, to estimate direction-of-arrive. (DoA), we adopt the ERP- algorithm [66]. Let M microphones at positions Pm (#m; Vm 2m ) record audio sampled at f. Hz, we compute time-domain tross-correlation Rmn for each microphone pair (m,,)) For each candidate azimuth á with unit direction vector un = [cos Φ), sin Φ, 0) the inter-
channel delay Tma (b) (Pm- P=) (()) is quantized into integer lag Lmn (x)
round (T))(()){))) (s)
c
DoA is estimated by maximizing the steered response power: e arg max P(o), where P(ò)
ΣM=1
Rym [d)].
To estimate distance, we adopt the coherent-to-diffure ratio (CDR) approach 67
We compute CDR at each time frame and use distance estimates from the visual-guide modules (Snapshot Descriptor and text-guided snapshot segmentation) to exploit the acoustic property that D; CDR, remains approximately constant in given environment. We estimate this constant K by computing Dj- CDR, per frame I, applying DBSCAN (88) to filter outliers, and minimizing the squared error
K over remaining frames: de V CDR,'
where K = arg min E. (D? CDR, - k)
To
reduce interference from the camera wearer's voice or front-facing background noise, we discard detections within narrow [-50, 59] range around the forward axis. This filtering improves reliability in egocentric direction estimation. Together, these estimates of the direction and the distance yield per-frame egocentric trajectories and serve spatial audio cues.
4.3 Stage 2: Dynamic Global Map Construction
For reasoning of spatial relationships, SAVVY aggregates the three egocentric trajectories from Stage 1 into unified global map. Each per-frame track is transformed global coordinates, resulting in 2D spatial map representation suitable for downstream spatial reasoning.
The track aggregation process is illustrated in Figure Bic). For static objects (e.g.. reference or fucing), globalized positions are clustered using DBSCAN to suppress outliers, and the centroid the dominant cluster is used as the final location. For dynamic target objects, a time-varying trajectory p(l) is constructed by filtering temporally aligned outputs from the three egocentric tracks in Stage and mapping them to global coordinates. A Kalman filter (69 is applied to interpolate and smooth ptt), producing a continuous and robust path.
The final map Mg contains continuous trajectory for the target object and static positions for the reference and facing objects. SAVVY then resolves the target's location based on the predicted query type from the Snapshot Descriptor: either egocentric (relative to the camera) or allocentric. In the allocentric case, the reference-to connceett-faccnn to-facing vector is aligned with the positive y-axis, and the map is rotated accordingly before computing the target's relative position.
Experiments
5.1 Metrics
SAVVY-Bench. SAVVY-Bench includes questions with respect to relative direction and absolute distance for both egocentric and allocentric categories [Section[3.] Direction questions (dir) are multiple choice, and we report accuracy based on exact or fuzzy matching 3 For distance (dist), which ranges from less than 1 m to more than 8 m, we avoid target-scaling 3 Instead, we compute the average relative accuracy across absolute error thresholds from 0.1 m 1.0 (step size 0. m) to allow fair comparison across varying distances.

---

Egocentric
Allocentric
Object Referral
Method
Dir
Dist
Dir
Dist
overall
Allocentric Direction
Egocentric
Chance-Level (Freq)
30.2
32.1
Direction
Human-Level
93.5
71.2
94.0
56.3
78.7
Long VALE 22
41.9
8.3
26.9
18.3
23.8
Jideo-SALMONT 26
35.4
36.9
24.9
14.5
27.9
Ola [24]
43.0
32.5
25.2
26.2
31.7
VideoLLaMMA-77 21
46.4
42.7
24.8
15.2
32.3
Allocentric Distance
Ezocentric
MiniCPM-o 2.6 25
45.8
42.3
25.2
15.9
32.3
Distance
EgoGPT [23
40.2
57.6
25.4
22.8
36.5
Gemini-22 Sommini-2.5-flae 2.5-
74.2
49.7
29.8
29.0
45.7
T-mloU
Gemini-2.5-pro
75.2
59.6
31.7
37.0
50.9
Human Level
SAVV
Gemini-2.5ggro
EgoGPT
VideoLLaMA2-78
MinICPM-o 2.6-78
SAVVY
84.7
62.9
44.0
40.2
58.0
Table 2: Evaluation on SAVVY-Bench. Left: Accuracy on egocentric and allocentric QAs. Right: Radar plot showing QA and SD-Eval accuracy comparison for top-3 open-source AV-LLMs.
Snapshot Descriptor Evaluation. To better understand the capabilities of AV-LLMS on SAVVY- Bench, we evaluate two tasks aligned with the Snapshot Descriptor (Section 4.2), reported under "SD Eval" Table (i) Temporal grounding task measures how accurately a model localizes the queried sound event in time. We use Intersection over Union (IoU) 70 between the predicted and groundtruth time intervals. Performance is reported as Recall@ 1, averaged over IoU thresholds from 0.05 0.5 (step size 0.05), and summarized as mean IoU (t-mloU). (ii) Object referral task tests whether the model correctly deseribes objects given the question and video. Egocentric questions involve only the larger sounding object; allocentric questions require further identification of the reference and the facing objects. We compute accuracy for (referral) via string matching and L.LM-based judging [4] with all required objects expected to match.
Localization Accuracy. We assess localization by comparing predicted and groundtruth positions. We propose a new metric, localization accuracy (loc_ace in Tables 4 and 6 a predicted location correct if the direction angular error __err is below 45 and the distance error r_err is below 1 m.
5.2 Main Results
Benchmark Models. We evaluate AV-LLMS as listed in Table Out of these, are open models designed for joint audio and video understanding. These include models that add an audio branch to a video-language model: VideoI.LaMA2 (21), VALL [22), and EgoGPT (3], with EgoGPT fine-tuned egocentric data. Video-SALMONT [26] adds visual encoder
audio-anguuage
model. Ola (24] and MiniCPM-o-2.6 (55 are trained as omni-modal models. Most models have
around 7B parameters; MiniCPM-o-24. has 8B and SALo-S has 13B. We also evaluate proprietary AV-LLMs: Gemini-2.55 2.5-ppo and Gemini-2.55flaas
All open-source AV-LLMs are evaluated using 32 sampled video frames and mono-channel compressed audio input. We include one chance-level baseline based on Frequency- for multiple-choice direction tasks, and the human-level baseline by aggregating independent responses of 6 annotators. For prompts, inference settings of all AV-LLMs, and human evaluation guidelines, please refer to the supplementary materials.
Human-level Performance. Humans achieve 78.7% accuracy on SAVVY-Bench, outperforming SAVVY (ours), the best method, by 20.7%. Directional tasks yield near near-perfect human performance (93.5-94.0%) reaffirming strong intuitive spatial reasoning, In distance estimation and in egocentric settings humans score 71.2% compared to 62.9% for the best model. For allocentric distance, human accuracy drops to 56.3%, reflecting the added difficulty of measuring distance after coordinate transformation involving various reference / facing objects.
AV-LLMs Results. All AV-LLMs perform better egocentric QA than on allocentric QA. Most models perform or below chance on the allocentric relative direction task. In contrast, performance on the egocentric version is higher; Gemini-2.5 models reach up to 75% accuracy. For absolute distance estimation, proprietary models outperform open-source ones on egocentric tasks. Some open- source models, e.g. Ego-GPT, show accuracy gaps up to 34.8% between egocentric and allocentric
7

---

Localization
DoA
Method
referral
t-mloU
Mic
loc acet 0 0_em] r. r_eml
Mrt
l/bt
LongVALE 22
33.7
0.7
02
19.5
104.8
1.86m
76.2
55.8
Videol.LaMA2-78
21
20.0
3.5
34
18.6
112.1
1.33m
82.3
54.5
MiniCPM-o 2.6 25
23.2
2.3
56
23.6
100.7
2.11m
81.6
55.5
EgoGPT 23
14.9
2.8
Ola 24
21.2
3.0
0234
15.5
116.4
1.54m
78.4
52.6
0256
44.2
39.2²
1.25m
79.8
69.8
Gemini-2.5-flasi
66.2
42.6
3456
44.3
45.6
1.11m
81.8
75.0
Gemini-2.. 2.5-pro
76,2
67.4
Table 4: Sounding object localization and Direc- tion of Arrival (DoA) accuracy left/right (Vrr
Table SD-Eval accuracy on temporal grounding (I-mloU) and object referral.
and front/back (f/b) across different microphones.
Type
loc_acct 0_err] r_err
Track Type
Sound loc_acc
Egocentric dir
Allocentric dir
SD
Audio Seg
dist
dist
Target Sounding Object:
SD
50.0
43.0º
0.84m
55.7
68.3
47.9
42.4
38.9
Seg
72.4
25,6
0,85m
59.0
73.9
48.1
Audio
44.3
45.6"
1.11m
v
72.5
81.2
52.0
34.2
23.1
66.8
74.5
54.6
44.4
41.0
Reference/Facing
Object:
v
73.6
84.2
57.7
34.4
24.0
SD
33.8
58.6
1.10m
76.0
83.2
54.4
44.2
36.9
Seg
38.3
57.7"
1.29m
78.6
84.7
62.9
44.0
40.2
Table 5: Object localization results of various egocentric track types.
Table 6: Egocentric track aggregation ablations on
sounding object localization and SAVVY-Bench QA.
distance tasks. Compared with egocentric questions, allocentric ones require more complex spatial transformation and reasoning of static referencel facing objects that appear briefly in the video.
To better assess AV-LLM capabilities on SAVVY-Bench, we evaluate 2 additional tasks, temporal grounding and object referral, as detailed in SD-Eval (5.1). Table 3 shows most open-source models achieve under 5% temporal mloU, indicating poor event-time alignment. Synchronizing complex events like speech remains challenging for models at 7B parameter scale. In object referral, fewer than 35% of responses are correctly grounded. These limitations may stem from AV-LLM training objectives that prioritize caption-level alignment and visual grounding, rather than learning to synchronize event timelines and spatial object tracks across audio and visual streams, a key requirement for complex spatial reasoning. Gemini-2.5-pre improves referral accuracy by 10.0% and temporal grounding by 24.8% over Gemini-2.5-faas suggesting the benefits of more advanced empporll-patiil reasoning capabilities.
SAVVY. As shown Table 2 adding SAVVY as a plugin to substantially improves relative direction accuracy: +9.5%
additional
training multi-turn AV-LLM for egocentric and +12.3% for allocentric QA. Distance accuracy also improves in both settings. SAVVY integrates Snapshot Descriptor, text-guided snapshot segmentation, spatial audio cues, and explicit spatial transformations to ground reasoning in a global map. These components collectively demonstrate modular path toward enhancing spatial reasoning for AV-LLMs, and motivate future work on training LLMs to internalize such structured spatial reasoning abilities.
5.3
Ablations and Analysis
We analyze how each egocentric track SamposenttSSappshot Descriptor (SD), text-guided snap- shot segmentation (Seg), and spatial audio- tracks (Audio)-contribuuu contributes to spatial reasoning individually and comprchensively.
Egocentric Tracks. Table 5 reports object localization performance for each egocentric track type (Stage 1 before aggregation), using the metrics from Section [31]] Given single image, Seg achieves the highest localization accuracy (loc _acc) and the lowest relative angle error (0 (0_err) for all objects type. For distance estimation, SD yields the lowest distance error (d (d_err), benefiting from temporal context and the advanced reasoning capabilities of AV-LLMs, as illustrated in the allocentric distance example in Figure Audio tracks perform competitively for sounding object localization, comparable SD, using spatial audio cues alone. Table studies the impact of different microphone channel combinations (more details in the supplementary materials) on localization and
8

---

DoA accuracy. Microphone combinations that include both front and rear positions (e.g. 0256, 3456) significantly improve front/back (f/b) DoA accuracy, while all configurations yield strong left/right (Vr) performance due to symmetric mie placement. Our SAVVY uses setup 3456, which achieves 81.8% (1/r), 75.0% (fb) on DoA, and the highest localization accuracy (44.3%).
Track Aggregation for Global Map Construction. Table eableieex examines how different combinations of egocentric track sources (SD, Audio, and Seg) in Stage 2 of global map construction affect spatial QA accuracy and sounding object localization (loc_acc) at the query moment. SD alone yields strong performance on allocentric QA, improving direction accuracy by 10.7% over Gemini-2.5-pre due to more precise static object localization. However, without audio, SD underperforms on egocentric QA due to limited tracking capability for dynamic sounding objects. Audio-only tracking achieves comparable egocentric direction accuracy to Gemini-2.5 both components, improving localization accuracy by +7.8% over Audio-only and boosting allocentric direction and distance QA by +12.7% and +4.0% over Gemini-2.5 achieve the highest standalone localization accuracy (72.5%). It also achieves very high egocentric direction accuracy due to precise localization of sound sources. However, it shows clear weaknesses in allocentric QA compared to SD, likely due to weaker static reference grounding from uniformly sampled frames. In contrast, SD leverages Gemini's 's stronger reasoning ability to more reliably identify static reference / facing objects. Finally, SAVVY integrates all track types, achieving the highest localization accuracy (78.6%) and the best egocentric QA by +3.5% on direction and SD+Audio by +8.3% on distance, its closest competitors in each task. A slight drop in allocentric QA is observed relative to SD+Audio due to noise introduced by Seg's 's static estimates, but overall SAVVY delivers the most balanced and accurate spatial reasoning across tasks.
SD+Audio combines the strengths of
respectively. Seg- Seg-based tracks
outtmmoomeeotttrtoominnnnn Seg
These findings highlight the complementary strengths of each modality: 1) Seg significantly con- tributes to localizing dynamic sounding objects, improving direction grounding, especially for egocentric questions. 2) Audio cues enhance distance estimation largely and also improve both localization and direction QA accuracy. 3) SD performs best on allocentric reasoning by more accurately localizing reference and facing objects to construct meaningful spatial relationships. These results underscore the value of integrated reasoning across modalities for robust spatial understanding.
5.4 Qualitative Analysis
Figure 4 presents two example cases from Gemini-2.5-pro, the strongest AV-LLM model in our experiments: the top shows egocentric direction reasoning, and the bottom shows allocentric distance reasoning. These illustrate the model's 's step-by-step reasoning and how SAVVY addresses its errors.
How do AV-LLMs perform spatial reasoning using video and monoaural audio? AV-LLMs process video with compressed monoaural audio, while SAVVY-Bench tasks require spatial sound localization- task humans perform via binaural hearing. In the egocentric direction task (Figure + top), Gemini-2.5-pe links sound to visible objects-in in this case, identifying "the other person" as the sound source-grounds the event time (0:39-0:41), tracks the sound source and camera wearer's s trajectory, and infers direction. For distance measurement (Figure bottom), the model further relies on visual cues and commonsense priors.
What errors do AV-LLMs make in spatial reasoning? Common errors of AV-LLMs on SAVVY. Bench often root in temporal grounding, object referral, and spatial relations (direction and distance). While Table accuracy on the first two, the examples Figure highlight highhihht spatial relation errors. In the queried event, the sound source ("the other person") disappears from view for tens of seconds. Gemini-2.5 2.5-pro infers its trajectory based only on its last visible location, leading to incorrect sound source location estimation. Because the model underutilizes spatial audio-despite its key role in human egocentric perception--i performs modestly when the object appears briefly but fails when it is absent for longer durations.
How does SAVVY address these errors? In Figure 4 SAVVY uses spatial audio cues to correctly localize the sound source at approximately 130 (back-right). The SD and Seg modules lack egocentric tracks in the visual context, but audio enables correct tracking of the dynamic sound source, yielding accurate back-right direction inference. In the allocentric distance case, SD and Seg help localize the egocentric track of the reference object-a a two-seater coffee with the accurate track of the sounding object, SAVVY produces correct distance estimate. By
well. Combined

---

Egocentric Direction
n' Time Range rtS Referral Description [ Spatial Relation
Q: Imagine you the camera
when the speech topic "confirming they have Croix drinks" comes up, relative to where you are facing, where is the other person: front-left, front-right, back-left, back-right?
1. Identify the relevant timestamp: The speech topic occurs when and the other "Yehh, let's see, we have some Croix for (0:39-0:41). Determine the camera wearer's facing direction: At 0:39-0:41, the camera wearer looking down at the plate food on the kitchen counter. This their primary facing direction. 3. Determine other person's location relative to the camera wearer: The other person is the refrigerator, which is the left the kitchen counter where the camera wearer focused. The refrigerator also further into the kitchen area, meaning it front the camera wearer's general orientation 4. Conclusion: The other person located the front-left the camera wearer.
responds,
AV-LLM: front-left
GT:
back-right
SAVVY: back-right
Speech Topic
SD Description:
Spatial Audio Cues
Co 01:10
SD Description:
00:01
A small, semi-circular
00:39
0=130
A small, semi-circular wooden dining table... Seg Prediction: Distance: 1.00m
wooden dining table...
SRP-PHA:
SD Prediction: Distance: 4.50m
Direction: 130"
Direction: -60
Direction: 8.39
SAVVY Prediction: back-right
Allocentric Distance
[ Time Range r'y Referral Description Tip Spatial Relation
HTM
3.82m
Q: When the speech topic: "confirming they have Croix drinks" mentioned, what the distance between the twa-seater dining table and the speech sound source meters?
The speech topic "confirming they have La Craix drinks' accurs at 00:40-00:41 when Person B wearing a light
AV-LLM:
2.30m
says, "Yeah, see, they the kitchen island near plate of cooked food, and Person A (holding the camera) is standing right located by sliding glass 00:55-00:57. Between 00:54 and 00:56, Person A walks from the island to the table in about 3 steps. 2.28 meters.
some
standing
two-seater dining
clearly
GT:
3.82m
Using average step length Rounding to one decimal place, the distance from the speech sound source to the center of the table is approximately meters.
estimated walking distance
SAVVY:
3.49m
Figure 4: Example reasoning process of AV-LLMs. Top (Egocentric direction); Bottom (Allocen- tric distance). Middle (SAVVY): SAVVY successfully fixes the spatial relation errors.
combining snapshot descriptors, segmentation, spatial audio cues, and explicit coordinate mapping, SAVVY offers a proof concept for potential solutions of improving AV-LLM spatial reasoning.
Conclusions
We introduce SAVVY-Bench and SAVVY, the first benchmark and training-free pipeline for 3D spatial reasoning in dynamic audio-visual environments. SAVVY-Bench poses thousands of questions grounded in egocentric video and multi-channel audio, covering both egocentric and allocentric perspectives. It targets core spatial object referral with focused evaluations on these aspects. Building on this, SAVVY boosts spatial reasoning over standard AV-LLMs by integrating snapshot-base perception, audio-visual tracking, and dynamic global mapping. Together, they offer practical foundation for advancing spatial intelligence in multimodal AI systems.
distance, temporal grounding, and grounded
Looking ahead, SAVVY is designed to be modular and extensible. While current data is based on indoor Aria captures, the spatial audio-video pipeline naturally generalizes to outdoor and driving scenarios. As richer spatial computing devices emerge, SAVVY-Bench can connect high-quality data, models, and applications, helping catalyze robust ecosystem for spatial AI.
7 Acknowledgments
The authors acknowledge the partial support of HDR Institute: Accelerated AI Algorithms for Data- DriveD Discovery (A3D3) National Science Foundation grant PHY-21 17997. The authors also acknowledge the partial support by the Departments of Electrical Computer Engineering and Applied Mathematics. The authors are thankful to the eScience Center at the University of Washington.
10

---

References
[1] Amy L. Shelton and Timothy P. McNamara. Systems of spatial reference human memory. Cognitive Psychology, 43(4))274-312, 2001.
[2] Boyuan Chen, Zhuo Xu, Sean Kirmani, Brain Ichter, Dorsa Sadigh, Leonidas Guibas, and Fei Xia. SpatialvIm: Endowing vision-language models with spatial reasoning capabilities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14455-14465, June 2024.
[3] Jihan Yang, Shusheng Yang, Anjali Gupta, Rilyn Han, Li Fei-Fei, and Saining Xie. Thinking in Space: How Multimodal Large Language Models See, Remember and Recall Spaces. arXiv preprint arXiv:2412.14177 2024.
[4] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li. Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum, Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfusion:
Open-set multimodal 3d mapping. Robotics: Science and Systems (RSS), 2023.
[5] Chenming Zhu, Tai Wang, Wenwei Zhang, Jiangmiao Pang, and Xihui Liu, Llava-3d: A simple yet effective pathway to empowering Imms with 3d-awareness. arXiv preprint arXiv:2409, 18125, 2024.
[6] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances Neural Information Processing Systems, 36:46212-46244, 2023.
[7] Arjun Majumdar, Amurag Ajay, Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Arnaud, et al. Openeqa: Embodied question answering in the era of foundation models. Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages (6488-16499 2024.
[8] Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. Learning to answer questions in dynamic audio- scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108-19118, 2022.
[9] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995-19012, 2022.
[10] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[11] Zhengyuan Yang, Linjie Li, Kevin Lin, Jianfeng Wang, Chung-Ching Lin, Zicheng Liu, and Lijuan Wang. The dawn of Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309.1443 17421, 9(1::1 2023.
[12] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. beetvannex--ntereeaae Tackling multi- aulti-image, video, and 3d in large multimodal models. arXiv preprint arXiv:240.007895, 2024.
[13] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint 230.003055.0655, 2023.
[14] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306. 02858, 2023.
[15] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:23 1,,12022.. 2023.
11

---

[16] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li. Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.03326, 2024.
[17] Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. Listen, think, and understand. arXiv preprint arXiv:2305. 2023.
[18] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unitied large-scale audio-language models. arXiv preprint arXiv:2311.07911 2023.
[19] Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, et al. Kimi-audio technical report. arXiv preprint arXiv:2504.18425, 2025.
[20] Zhifei Xie, Mingbao Lin, Zihang Liu, Pengcheng Wu, Shuicheng Yan, and Chunyan Miao. Audio-reasoner: Improving reasoning capability in large audio language models, 2025.
[21] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial- temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07477. 2024.
[22] Tiantian Geng, Jinrui Zhang, Qingni Wang, Teng Wang, Jinming Duan, and Feng Zheng, Longvale: Vision-audio-lane waa---uddi--l-ggu--eeeeeeee benchmark towards time-aware omni-modal perception of long videos. arXiv preprint arXiv:241 197722, 2024.
[23] Jingkang Yang. Shuai Liu, Hongming Guo, Yuhao Dong Xiamengwei Zhang, Sicheng Zhang. Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang. Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang, Bo Li, and Ziwei Liu. Egolife: Towards egocentric life assistant, 2025.
[24] Zuyan Liu, Yuhao Dong, Jiahui Wang. Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Ola: Pushing the frontiers omni-modal language model with progressive modality alignment. arXiv preprini arXiv:2502. 043288 2025.
[25] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: gpt-4v level milm on your phone. arXiv preprint arXiv:24088 240.000000000000 01000 2024.
[26] Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, Yuxuan Wang, and Chao Zhang. video-SALMONN: Speech-enhanced audio-visus visual large language models. Forty-first International Conference on Machine Learning, 2024.
[27] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[28] Artemis Panagopoulou, Le Xue, Ning Yu, Junnan Li, Dongxu Li, Shafiq Joty, Ran Xu, Silvio Savarese, Caiming Xiong, and Juan Carlos Niebles. X-instructblip: A framework for aligning x-modal instruction- aware representations to lims and emergent cross-modal reasoning, 2023.
[29] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824. 2023.
[30] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv: 2311:2331.00704, 2023..
12

---

[31] Haotian Zhang, Haoxuan You, Philipp Dufter, Bowen Zhang, Chen Chen, Hong- You Chen, Tsu- Jui Fu, William Yang Wang, Shih-Fu Chang, Zhe Gan, et al. Ferret-v2: An improved baseline for referring and grounding with large language models. arXiv preprint arXiv:2404.07999 2024.
[32] Sourav Garg, Krishan Rana, Mehdi Hosseinzadeh, Lachlan Mares, Niko Sünderhauf, Feras Dayoub, and Ian Reid. Robohop: Segment- topological map representation for opon-world visual navigation. In 2024 IEEF International Conference on Robotics and Automation (ICRA), pages 4090-4097 IEEE, 2024.
[33] Qiao Gu, Ali Kuwajerwala, Sacha Morin, Krishna Murthy Jatavallabhula, Bipasha Sen, Aditya Agarwal, Corban Rivera, William Paul, Kirsty Ellis, Rama Chellappa, et al. Conceptgraphs: Open-vocabulary 3d scene graphs for perception and planning. In 2024 IEEE International Conference Robotics and Automation (ICRA), pages 5021 5021-5028. IEEE, 2024.
[34] Yuncong Yang, Han Yang, Jiachen Zhou, Peihao Chen, Hongxin Zhang, Yilun Du, and Chuang Gan. 3d-mem: 3d scene memory for embodied exploration and reasoning, 2024.
[35] Jiajun Deng, Tianyu He, Li Jiang, Tianyu Wang, Feras Dayoub, and Ian Reid. 3d-llava: Towards generalist 3d imms with omni superpoint transformer. arXiv preprint arXiv:2501.01166 2025.
[36] Duo Zheng, Shijia Huang, and Liwei Wang. Video-3d Ilm: Learning position-aware video representation for 3d scene understanding. arXiv preprint arXiv:2412.00493, 2024.
[37] Zihan Wang, Xiangyang Li, Jiahao Yang. Yeqi Liu, and Shuqiang Jiang. Gridmm: Grid memory map for vision-and- language navigation. In Proceedings of the IEEE/CVF International conference on computer vision, pages 15625-15636, 2023.
[38] Chenguang Huang, Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
[39] Hanlin Chen, Fangyin Wei, and Gim Hee Lee. Chatsplat: 3d conversational gaussian splatting. arXiv preprint arXiv:2412.00734, 2024.
[40] Jin-Chuan Shi, Miao Wang. Hao-Bin Duan, and Shao-Hua Guan. Language embedded 3d gaus- sians for vpca-locabulary scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5333-53433 2024.
[41] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.088 200.000300 2024.
[42] Sharath Adavanne, Archontis Politis, Joonas Nikunen, and Tuomas Virtanen. Sound event localization and detection of overlapping sources using convolutional recurrent neural networks. IEEE Journal of Selected Topics in Signal Processing. 13(1):34 48, 2018
[43] David Diaz- Guerra, Antonio Miguel, and Jose R Beltran. Robust sound source tracking using srp-phat and 3d convolutional neural networks. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:300-311,
[44] Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath. Bat: Learning to reason about spatial sounds with large language models. arXiv preprint 200.v:2402.0.55) 2024.
[45] Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.028 02490, 2023.
[46] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
13

---

[47] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, al. Mmbench: your multi-modal model an all-arount player? In European conference computer vision, pages 216-233. Springer, 2024.
[48] Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models. arXiv preprint X 1. 16103, 2023.
[49] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195 22206, 2024.
[50] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal iims in video analysis. arXiv preprint arXiv:2405.21075, 2024.
[51] Baoxiong Jia, Ting Lei, Song- Song-Chun Zhu, and Siyuan Huang. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 35:3343-3360, 2022.
[52] Shicheng Li, Lei Li, Yi Liu, Shuhuai Ren, Yuanxin Liu, Rundong Gao, Xu Sun, and Lu Hou. Vitatecs: A diagnostic dataset for temporal concept understanding of video-language models. In European Conference on Computer Vision, pages 331-348. Springer, 2024.
[53] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint arXiv:2403.00477. 2024.
[54] Chiara Plizzari, Alessio Tonioni, Yongqin Xian, Achin Kulshrestha, and Federico Tombari. Omnia de egotempo: Benchmarking temporal understanding of multi-modal Ilms in egocentric videos. arXiv preprint arXiv:2503. 2025.
[55] Xiulong Liu, Zhikang Dong, and Peng Zhang. Tackling data bias in music-avqa: Crafting a balanced dataset for unbiased question-answering. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4478-4487, 2024.
[56] Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pages 3480-3491, 2022.
[57] Heeseung Yun. Youngjac Yu, Wonsuk Yang. Kangil Lee, and Gunhee Kim. Pano-avqa: Grounded audio- question answering on 360deg vidcos. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2031-2041, 2021.
[58] Chao Huang, Yapeng Tian, Anurag Kumar, and Chenliang Xu. Egocentric audio-visual object localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11910-229220 2023.
[59] Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 19129-19139, 2022.
[60] Zhaoyang Lv, Nicholas Charron, Pierre Moulon, Alexander Gamino, Cheng Peng, Chris Sweeney, Edward Miller, Huixuan Tang, Jeff Meissner, Jing Dong, et al. Aria everyday activities dataset. arXiv preprint 2422 2024.
[61] Julian Straub, Daniel DeTone, Tianwei Shen, Nan Yang, Chris Sweeney, and Richard Newcombe. Efm3d: A benchmark for measuring progress towards 3d egocentric foundation models. arXiv preprint arXiv:2406.100) 10224, 2024.
14

---

[62] Huajian Huang. Longwei Li, Hui Cheng, and Sai-Kit Yeung, Photo-slam: Real-tine real-time simul- tancous localization and photorealistic mapping for monocular stereo and rgb-d cameras. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21584-21593, 2024.
[63] Timo Lüddecke and Alexander Ecker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7086-7096, 2022.
[64] Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao- Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos. arXiv preprini arXiv:2408.00714. 2024.
[65] Shariq Farooq Bhat, Reiner Birkl. Diana Wofk, Peter Wonka, and Matthias Müller, Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. 12288, 2023.
[66] Joseph Hector DiBiase. A high-accuracy, low-latency technique for talker localization in reverberant environments using microphone arrays. Brown University, 2000.
[67] Andreas Schwarz and Walter Kellermann. Coherent-to-diffus power ratio estimation for dere- verberation. IEEE/ACM Transactions on Audio, Speech, and Language Processing. 23(6):1006- 1018, 2015.
[68] Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. A density- algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD'96, page 226 231. AAAI Press, 1996.
[69] Rudolph Emil Kalman, new approach to linear filtering and prediction problems. Transactions of the ASME-Joumnn of Basic Engineering, 82(Series D):35-45, 1960.
[70] Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voe) challenge. International Journal of Computer Vision, 88:303-338, 06 2010.
[71] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu. Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.1277.. 2024.
[72] Xiulong Liu, Sudipta Paul, Moitreya Chatterjee, and Anoop Cherian. Caven: An embodied conversational agent for efficient audio-visual navigation in noisy environments. Proceedings of the AAAI Conference on Artificial Intelligence, 3865):376533773, Mar. 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The claims precisely reflect the paper's contribution and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We mention the limitations in the supplemental material.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
16

---

Justification:
Our work is application-font focus and experiment-drem driven.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We incorporate sufficient details of our pipeline, baseline methods, evaluation metrics and etc.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [Yes]
Justification: Code and benchmark dataset will be released for upon request from the corresponding author. We also aim release the code and the benchmark dataset as public repository on the Github upon obtaining involved approvals.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nipe) cc/
pubo httlc//guie////eeeeeeeee
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See NeurlPS code and data submission guidelines (https
httpp/scccppbblcc//occc// Sodeeeero/oodebeemrcc ss onPol icy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all implementation details of our proposed pipeline and baseline methods well as evaluation metrics necessary to achieve the results claim in the experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [NA]
Justification: Our method and baseline methods provide stable results across multiple runs with negligible variance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
18

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All experiments are inference-typee experiments without training. For baseline single A100 (40GB) to perform experiments. For Gemini 2.5, we use Google Cloud Platform's 's API to perform experiments. We include these information in the supplementary materials.
AV-LLMs ( 7B scale),
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp//wwearrrr....ccooolileeea
Answer: [Yes]
Justification:
We have reviewed and carefully followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss broader impacts in our supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
19

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (c.g., pretrained language models, image generators, scraped datasets)?
Answer: [Yes]
Justification: We discuss the need for safeguards regarding the negative impact associated with AV-LLMs and propose precautions and guidelines to prevent misuse. These details are included as part of the Broader Impact section in the supplementary materials.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We cited all open-source works that we use including the data, models and codes.
Justification:
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
20

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppapersw paperswithcood com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We clearly described the details of statistics and distributions, how we curate the dataset in details. We aim to release the dataset upon obtaining involved approvals.
benchmark and dataset in the paper, including
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: We included detailed description of human study instructions, and and the setup of the of the study in the supplementary materials.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [No]
Justification:
There are no risks associated with our human evaluation experiment.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification:
The core contribution this paper is around AV-LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: http://ppp..ccooooooeenneeeee//s 2025/ LLM) for what should or should not described.
22

---

A Summary of Supplementary Materials
In this supplementary materials, provide:
1. A video demonstrating the case examples detailed in Figure of the main paper is available at ourr webpage here For the best viewing experience, we recommend watching the video with headphone device that supports spatial audio playback. See Section B for details.
2. Details of benchmark construction pipeline, including data processing, annotations, QA synthesis and quality review, see Section
3. Evaluation details of SAVVY-Bench, including open-source AV-LLMS, proprietary AV- LLMs and human evaluations, see Section
4. Details of input data the pipeline, including video input settings, multi-channel audio settings (microphone configurations), as well as camera trajectory, see Section E
5. Additional implementation details of all stages in SAVVY, see Section
6. Efficiency analysis of SAVVY, see Section G
7. Additional ablation studies of SAVVY-Bench on input modalities, see Blind Testing in Section
8. Limitations of SAVVY, see Section !
9. Broader impacts of the work with safeguards, Section
10. Additional qualitative results which showcase the reasoning process SAVVY as well as the error types analysis, Section K
B Video Examples
The demo videos contain case examples-on egocentric direction task and one allocentric distance lask-captured in single video clip featuring two people conversing in an indoor setting. We recommend watching the video with headphones device that supports spatial audio playback.
These examples correspond to the qualitative results presented in the main paper. In both cases, the queried event is: confirming they have La Croix drinks, corresponding to the spoken sentence, "Yeah, let's see ... grab some La Croix for us," from guest (a male wearing blue shirt) speaking the camera wearer.
Egocentric Direction Example. The question asks for the relative direction of the other person, with options: front-left, front-right, buck-left, or back-right. In this clip, the other person is not visible at any timestamp during the event, as he is located in the back-right quadrant relative to the camera wearer. While the direction must be inferred from spatial audio cues, a human viewer can clearly perceive the sound coming from the back-right when watching the video with spatial audio. SAVVY correctly predicts this as back-right, whereas Bemini-2..5ppm incorrectly classifies it as front-left.
Allocentric Distance Example. This question asks for the distance between the two-seater dining table and the speech sound source (the male guest in the blue shirt). The table is clearly visible in several frames throughout the video. SAVVY localizes both the table and the sound source using combination of egocentric tracks via Snapshot Descriptor, text-guided snapshot segmentation and spatial audio cues. SAVVY estimates the distance as 3.49 meters, which is close to the ground truth of 3.82 meters. In contrast, Gemini-2.5- predicts significantly incorrect distance of 2.30 meters.
These examples illustrate SAVVY's robustness in both directional and quantitative spatial reasoning. especially in challenging, partially observed scenarios.
C Benchmark Construction
We implement four-stage pipeline construct SAVVY-Bench. The stages are Data Preprocessing, Annotation, QA Synthesis, and Quality Review. Fach stage combines automated tools with human checks to ensure that every Question-Answer (QA) pair is precise.
23

---

Metadata
Human Annotations
Troes
poaausees [6 80,,,, 5,009, 2.809).
tragecter"'s {
1-3,1582, 3. 3.7955, 79555/014446 2244
Trei, Lef "rees"
"eenttence:: alt coun.",
orssestties asstormeet (8.725, 19,35, 9.485).
"red_astrrsaae"" 3.500.
ongilar aetoccyy's [-0.662, 6.040, -0.076),
groddyy [-0,9, -0.0, -8.13]
treci_ sto.ssaaraaaaa. 4.000
tnecL_nndaaas" 4.499.
feect www.ooorrrrrrrss" N67
Fisheye
Undistorted
SEEP
Object Locations
Audio Transcription
Data Preprocessing
Annotation and GT Generation
QA Synthesis from Templates
Inagine you robot standing by the (reference object) and freference facing object), {sound event} comes up, relative to facing,
this good
where you where is the speaker?
Question:
Imagine you are the camera wearen, when the (sound event) cones up, relative to where are facing, other person?
Q: Imagine where is the speaker? A: Front-left
prev
next
Submi t
QA Synthesis
Quality Review
SAVVY-Bench
Figure 5: Human-in-the-Loogy Dataset Curation and Benchmark Construction Workflow for SAVVY- Bench.
C.1 Data Preprocessing
We preprocess the video data from the Aria Everyday Activities (AEA) Dataset (60) and integrate raw annotations-such as word-level transcriptions, camera-wearer trajectories, and other sensor signal records-inte unified metadata schema, illustrated in Figure
For video preprocessing, the original fisheye recordings are undistorted into rectilinear frames to ensure compatibility with AV-LLMs. In scenarios with two wearer-mounted camera streams, the videos are temporally aligned to form unified timeline. This alignment supports consistent segmentation of speech into sentences and facilitates accurate speech topic extraction.
C.2 Annotation and Ground Truth Generation
Our annotation focuses primarily on objects and events.
Static Object Annotation. Static objects are automatically detected using EFM3D 61 based on predefined list object categories (e.g., couch, fireplace). We use Vision-LLM 41 to generate informative description phrase for each detected object. Annotators then inspect the 3D coordinates and descriptions in a point-cloud viewer, correcting any errors in location, category, or description as needed.
24

---

Sounding Event Annotation. For each sound event, we annotate the event description transcrip- tion, its start and end times, and the identity and 3D location of the sound source- if the source is tied to physical object (e.g., running water with faucet, a thud with door). Human annotators adjust the event time span and label the source object and its position accordingly. Specifically for speech events, we first cluster raw word-level transcripts into complete sentences. Annotators then label speech events on sealence-by-sentene basis. A prompted, rule-based agent [41] converts these validated sentences into concise speech topics that describe individual conversational moments. The prompt design used for this process is shown in Figure 6
Prompt: Word-Level Transcriptions Speech Topic
[Task]
You are an agent to annotate conversation data:
[Rule]
1. Create concise speech topics for each sentence that summarize what was said.
2. Use verb+ing format for all speech topics (e.g., "Hello, how it going. " "initiating conversal iion).
3. Ensure each speech topic is unique, using differentiating language for similar sentences.
4. Make topics concrete and specific enough that someone could identify the original sentence when hearing it.
5. Only reference what can be heard in audio (avoid visual elements like "pointing" ng").
6. Avoid abstract descriptions (e.g.. use "eating di directly from bowl"
"announcing eat ing method")
7. Maintain the entire original CSV structure with all timestamps and durations.
[Output]
1. Add "speech_topic column right after the "sentence" column in the CSV.
2. Output Format: rec id, sentence, speech__oppiee recl_startTime, .tc.
Figure 6: Prompt used to generate speech topics from word-level transcripts.
Sound Event Annotation System and UI. To streamline the annotation process and reduce errors, we developed a desktop annotation tool using PyQt5. This system integrates video playback, speech and non-speech event labeling, and timestamp editing in single interface (see Figure 7 It supports dual- camera views with synchronized playback and saves annotations locally. The tool is self-contained, works offline, and requires no server backend.
Human Annotation Guideline for Sound Events. Annotators follow five key principles:
1) Accuracy: For speech events, correct the original word-level transcription to ensure that every spoken word and audible event is captured exactly heard. Remove filler words and non-informative tokens, retaining only meaningful content.
2) Completeness: Label the full audible span of each event, setting start and end times as close as possible to the actual boundaries to avoid clipping or omission.
3) Symchronization: For speech events involving two participants, maintain the temporal alignment between the recordings from both devices throughout the annotation process.
4) Label Uniformity: For non-speech sound events, ensure that each description is unique, unambigu- ous, and consistent across the entire video.
5) Language and Mechanics: Use standard spelling, punctuation, and capitalization. Maintain consistent formatting across all annotations.
C.3
QA Synthesis
We use template scripts to generate QA pairs for SAVVY-Bench. These scripts integrate the unified metadata (described Section C.T) with the new annotations and ground truth data (from Section C.2 using well-defined question schemas, resulting in unambiguous and structured QA pairs.
25

---

Sprech
Transorist Filec
iet aeren, serert.eee..cr
net perpptt,tneee
sed sersit, aeroil,aeaton
sod wew.eetttta..
sic aasssott..am
sed sctora, cesicer astor1 ae22
nnd se2.ttt ow
sel reluremnnueeee
Jerost,, scl peroit, jeaticer
kcc setoz2 aead
vensit wealow ket ceneit seat eo
Damant Timx:
21232
Play Rec
Current Time:
22:150
Play Rec
Senteroae
3 Arggt Coning on
Fii: lock wwc_ssrrpsssseee..ooo cav
Arrgtt pat some procetes
21
mey Thorss
autna to these obwe an
yetting these
Sentence
Years just gaing out these down in hitchen hern. Set co for you.
10 Lur years par some gai sarna waaa cel
going
These cartes.
Camer:
.0.11
Sixe
Airgtt
recorting re do...
Times (Foitaabe))
Recarding Times
Youls rihak ubout
night sanve Voa,
Start
21260 0 Bndt
26 660
Duratin: 4.620
Start
21,500
End:
25 370
Curntie:: 4.400
Save Changes
Auto savt: Activ
Devte Sentence
Figure 7: Interface for sound event annotation. The tool displays dual-camera videos with synchronized playback and saves annotations locally.
CAVVY-Bench includes six templates covering four task types: egocentric direction, egocentric distance, allocentric direction, and allocentric distance. For both egocentric and allocentric direction tasks, we design two levels of difficulty: a simple template with three options (left, right, back) and hard template with four options (front-left, front-right, back-left, back-right).
We provide the complete set of templates for all six QA types, each specified for both speech and non-speech sound events follows:
Egocentric Direction Simple
1. Imagine you are the camera wearer, when the (non-speech. sound event) sound comes up, relative where you are facing, where the sound source: left, right, or back? If the object is generally to your left and facing it requires turning less than 120 degrees left, choose 'left'. the object generally to your right and facing requires turning less than 120 degrees right, choose 'right". If the object is generally behind you and facing it requires turning 120 degrees or more, choose 'back'.
2. Imagine you are the camera wearer, when the speech topic ( (speech topic) comes up, relative to where you are facing, where the other person left, right, or back? If the object generally your left and facing it requires turning less than 120 degrees left, choose 'left', If the object is generally to your right and facing it requires turning less than 120 degrees right, choose 'right.. If the object is generally behind you and facing it requires turning 120 degrees more, choose 'back'.
26

---

Egocentric Direction Hard
Imagine you are the camera wearer, when the (non-speed speech sound event } sound comes up, relative where you facing, where is the sound source: front-left, front-right, back-left, or right? The directions refer to the quadrants of a Cartesian plane (if you are standing at the origin and facing along the positive y-axis). Consider the center point location of the object as the its location.
2. Imagine you the camera wearer, when the speech topic { (speech topic) comes up, relative to where you are facing, where is the other person: front-left, front-right, back-left, or back-right? The directions refer to the quadrants of Cartesian plane (if you are standing the origin and facing along the positive y-axis). Consider the center point location of the object as the its location.
Egocentric Distance
1. Imagine you are the camera wearer, when the (non spen-spechh sound event sound comes up, relative to where you are standing, what the distance between you and the sound source in meters? Consider the center point location of the object as the its location. Calculate the Euclidean distance between the two points in the horizontal plane. Answer in numeric format.
2. Imagine you are the camera wearer, when the speech topic: (speech topic) comes up, relative to where you are standing, what is the distance between you and the other person in meters? Consider the center point location of the object as the its location. Calculate the Euclidean distance between the two points in the horizontal plane. Answer numeric format.
Allocentric Direction Simple
1. Imagine you are a robot standing by the { (reference object? } white recessed fireplace and facing (facing object), ), when the (non-speeds sound event sound comes up, relative to where you are facing, where is the sounding object: left, right, or back? If the object is generally 'left'. If the object is generally to your right and facing it requires turning less than 120 degrees right, choose 'right'. If the object is generally behind you and facing requires turning 120 degrees or more, choose 'back".
your left and facing it requires turning less than 120 degrees left, choose
Imagine you are robot standing by the (reference object) and facing the (facing ob jecte] when the speech topic: (speech topic) comes up, relative to where you are facing. where the speaker: left, right, or back? If the object is generally to your left and facing it requires turning less than 120 degrees left, choose 'left'. If the object is generally to your right and facing it requires turning less than 120 degrees right, choose 'right'. If the object is generally behind you and facing it requires turning 120 degrees or more, choose "back".
Allocentric Direction Hard
1. Imagine you are robot standing by the (reference object) and facing the ( (facing ng ob jeject], when the (non coon-speect sound event) sound comes up, relative where you are facing, where is the sounding object: front-left, front-right, back-left, or back-right? The directions refer the quadrants of Cartesian plane (if you are standing the origin and facing along the positive y-axis). Consider the center point location of the object as the its location.
2. Imagine you robot standing by the (reference object) and facing the { (faci ng ob object), when the speech topic: (speech topic) comes up, relative to where you are facing, where the speaker: front-left, front-right, back-left, or back-right? The directions refer to the quadrants of Cartesian plane (if you are standing at the origin and facing along the positive y-axis). Consider the center point location of the object the its location.
27

---

Video GA Viewer
Videcs:
Lorkert Lec"U
bet serp22
Lec
ledt Nec
bet Lacree sec
lick sereer secluuc
kedlae (r8_3992..v
bet serptz 19s3
Search
Filte ousstion....
Cumant Time
10010
Questiens:
When the saeech maik
spooch hore
Scripc,
sputal tomporai
When speech teaic
Q4 Type:
MC4
Modility:
aadio vbual
Leve::
hard
when the speech booic
Queet on
When the spooch lopic
When the conech tople. "ddntiyyng lettner quess" was mentioned, red, i''m standing the round eitic whth a and facing the ezainiess steel double wall owen, speech sound source other tha me to my fromt left, front- right back-eett bace-right directions riter qundhants
Your Anowwon
Options:
Answer Fornas
fiset- lift
rearanical:: 2:,
Flont right
mulitale cholos A pr prrow: 2.00
back-lent
Figure 8: Review interface for QA pair quality review. The tool displays each video clip alongside its associated question and predicted answer, allowing reviewers to efficiently assess correctness, clarity, and formatting, and make a decision on whether the QA pair good QA that should be retained.
Allocentric Distance
When the (non-speed)) sound event] sound is happening, what is the distance between the ( (re ference object) and the sounding object in meters? Consider the center point location of the object as the its location. Calculate the Euclidean distance between the two points in the horizontal plane. Answer in numeric format.
When the speech topic: (speech topic} is mentioned, what is the distance between (re Ireference object) and the speech sound source in meters? Consider the center point location of the object as the its location. Calculate the Euclidean distance between the two points in the horizontal plane. Answer in numeric format.
C.4 Quality Review
We combine automated QA generation with manual review to ensure both scalability and quality. This hybrid pipeline enables efficient creation of large-scale QA pairs while preserving high annotation accuracy. The resulting dataset offers reliable benchmark for evaluating 3D spatial reasoning in AV-LLMs. this section, we detail the human quality review process that supports this workflow.
Review Interface. To secure the final data quality, we construct a review system with PyQr 5. The system presents each video clip together with its question and answer and offers a simple interface for reviewers to validate revise the pair efficiently, illustrated in Figure
28

---

Review Guideline. Reviewers follow five principles:
Correctness: The stored answer must be fully supported by what visible and audible the clip.
2) Clarity: The question text must be clear and free of ambiguity.
3) Relevance: A question must refer only to content that is explicitly present in the clip or its metadata. should not rely on commonsense inference or assumptions beyond what is observable.
4) Consistency: Answers must respect the predefined format, units, and option labels.
5) Traceabiliry: Each reviewed QA pair is labeled as accepted rejected based whether it qualifies as "good" question. All edits are logged to support future auditing and reproducibility.
D SAVVY-Bench Evaluation Details
D.1 Open-Source AV-LLMs
All experiments are run in inference mode without model training. For open-souree AV-LLMs at around 7B scale, VRAM A100 GPU. Evaluation follows the LMMs--Eal module [71]. We use greedy decoding with temperature O, and both top-p and top-k uniformly across the entire video duration. For audio, we average multiple channels to produce compressed monaural input, with a sampling rate of 16kHz.
single A 100 GPU (40GB). For 13B scale AV-LLM,
single 80GB
1. Following [71), sample 32 video frames
The input for the models is formatted as [Video Frames), [Audio Content] and [Prompt]
Prompt details:
Relative Direction Questions simple
[Question]
Options: A: left B: right C: back.
Answer in single letter or numeric format.
Relative Direction Questions
hard
[Question]
Options: A: front-left B: front-right C: back-left D: back-right. Answer in single letter or numeric format.
Relative Distance Questions
[Question]
Answer in single letter or numeric format.
D.2 Proprietary Models
For Gemini-2.5-fte and Gemini-2.5-pro, Google Cloud Platform's API. We upload and feed the full video with audio to the model, following API guidelines.
Prompt details:
29

---

Prompt:
Proprictary Models on SAVVY-Bend
Given the Video: [Video Frames].
Question: [Question].
Options: [Options]
[Prompt]
Answer the question.
[Format Instructions]
1. Your output must be a single, valid JSON object conforming to the schema defined below.
2. Do NOT output any thinking steps or reasoning steps.
[JSON Schema]
"prediction": "Your final answer (A, B, or A, B, C, or numeric value). you can't decide, please output JSON with the "prediction" key's s value being null."
D.3 Human Evaluation Guidelines
Question Level: Simple
Question Level: Hard
Forward vector
Forward vector
Front-left
Front-right
Left
Right
Back-left
Back-right
Back
Figure 9: Direction quadrant guide for human evaluation. Egocentric directions are relative to the camera wearer's facing direction, while allocentric directions use fixed world frame.
Evaluation Setup. We recruited six independent evaluators to participate in the human evaluation. The question set shuffled and evenly divided among the evaluators. Each evaluator was allowed to pause, replay, or scrub through the video clip as many times as needed before submitting their answer. For direction-based tasks, evaluators followed the quadrant chart shown in Figure 9 For distance-based tasks, the correct response corresponds to the Euclidean distance between the two referenced points projected onto the horizontal plane.
Evaluation Rules. Evaluators followed four key rules:
1. Perspective:
Identify whether the question requires egocentric allocentric reasoning, and apply the appropriate frame of reference.
2. Exactness:
Select the most accurate answer supported by visual and audio evidence, avoiding reliance commonsense inference.
3. Consistency: Use the labels and answer formats provided (c.g., A. B, C or numerical values in the specified format).
4. Independence:
use any external tools such as object trackers or scene maps; rely solely on the provided video clip.
30

---

Ethical Statement. Participation was voluntary, involved no known physical or psychological risks, and did not collect any personal data beyond the evaluators' responses.
E
Input Data Details
E.1
Visual Input Settings
Original fisheye vidcos from the Aria-Everyday Activities (AEA) dataset [60] were undistorted to a standard rectilinear format for compatibility with common AV-LLM inputs. We also manually aligned the two cumera-weards videos for each conversation, creating unified timeline to facilitate consistent speech sentence segmentation and speech topic generation. For all open-source AV-LLMs, we evaluated using 32 sampled video frames via uniform sampling.
E.2 Microphone Configuration
We detail the microphone geometric configuration used the AEA dataset. The data is collected using Meta's 's Aria Glasses, which are equipped with a 7-channel 48kHz microphone array distributed around the frame. Specifically, five microphones are positioned along the front frame, and two mounted near the rear temple arms. This configuration enables rich spatial audio capture from both forward- and backward-face farinng-fingggg directions.
The specific microphone locations (in meters, relative to the center the glasses) are as follows:
Mic gight-front-bottnn corner (0.05, -0.04, 0.00)
Mic centered at the bridge of the nose ( (-0.005, 0.00, 0.00)
Mic 2: left-front-hotton corner (-0.05, - -0.04, 0.00)
Mic 3: far-left-up along the front frame (-0.07, 0.00, 0.00)
Mic 4: far-right-up along the front frame (0.07, 0.00, 0.00)
Mic 5: rear left leg (-0.07, 0.00, -0.10)
Mic 6: rear right leg (0.07, 0.00, -0.00
A visualization of this microphone configuration is available on the Project Aria Hardware Specifica- tions GitHub page.
E.3 Camera Trajectory
SAVVY uses 6DoF camera trajectories at 1kHz. These trajectories approximate the continuous motion of the egocentric observer and are computed using the foundational visual-inertial odometry (VIO) and simultaneous localization and mapping (SLAM) systems onboard the Project Aria device. In our work, we use the calibrated closed-loop trajectories, represented by 3D position and orientation in quaternion form.
F SAVVY Details
F11 Snapshot Descriptor
As described the main paper, the Snapshot Descriptor aims (I) identify the start and end times of the event; (2) determine whether the question requires an egocentric or allocentric view; (3) identify the target sounding object, reference object, and facing object, along with their text descriptions; and (4) track the egocentric direction and distance of each object at key frames.
To distinguish between views:
Egocentric view refers the camera wearer's perspective. In this case, the reference object is the camera, and no facing object needed. Since the camera trajectory known, only the target sounding object needs be identified and tracked.
31

---

Prompt: Open-Source AV-LLMS
[Task]
Analyze the given video based the question: "question". The total video length is duration seconds. Identify the Sounding Object (source of sound). Identify the start_time and end time of the event mentioned in the question. Determine the mode:
If I'm the camera wearer's view (egocentric), set mode egocentric. If I'm in different perspective rather than the camera's view (allocentric), set mode to a loceentrcc.
[Output]
Return single JSON object with the following structure:
"start_time" //start time of the event asked in the question "end_time": //end time "mode": spocetziccalllocntrii. "sounding joanddinn_oojcct::
"description": "A detailed description of the sounding object source of scund). Include physical characteristics like type, color, material, and approximate size/shape.", "is_static"; true/false // True if the object generally nen- typically moves location
moving, false
h "atand baind_bby_ee by_obecct";
"object oojject_name": "Name", //set to camera if requires allocentric is false "description": "Description"
"faciingdireccton" " "object_name": "Name", "description": "Description"
Figure 10: Prompt for Open-Source AV-LLMS on SAVVY-Bench.
Allocentric view requires perspective other than the camera's. new coordinate frame is built using the reference object (as the origin) and the facing object (defining the positive y-axis from the reference). In this case, all three objects must be identified and accurately tracked.
Open-source AV-LLMs, typically at the 7B or 13B scale, often struggle to track all objects through prompt guidance. Therefore, request these AV-LLMs perform only the first three objectives: identifying the event time span, determining the view mode, and generating accurate object descrip- tions in correct object categories (target sounding reference / facing object). For all models, greedy decoding with temperature set to 0, and both top-p and top-k set to 1.
Detailed prompts used for both open-source AV-LLMs and proprietary models to generate Snapshot Descriptor are provided in Figure 10 and respectively.
F.2 Text-Guided Snapshot Segmentation
We uniformly sample 128 frames from each video. For each object, we use its descriptive phrase, extracted from the Snapshot Descriptor, as input to ClipSeg [63] to generate segmentation mask. Within the segmented region, we sample 10 keypoints and compute the average ClipSeg confidence. A detection is considered valid if the average score exceeds threshold: 0.5 for dynamic sounding objects and 0.6 for reference and facing objects. We then use the selected keypoints and object descriptions to prompt the SAM model [64], obtaining refined segmentation masks.
32

---

Prompt: Proprictary Models [Task]
Analyze the video uploaded_ obj based on the question: question.
Identify the Sounding Object, the Reference Object, and the Facing Object (stand by the Refer- ence Object and face the Facing Object). Identify the start time and end time of the event mentioned in the question.
Determine the mode:
If I the camera's 's view (egocentric), mode to egocentric.
If I am in different perspective rather than the camera's view (allocentric), set mode to
allo ioloocettz ric.
Perform audio-visual tracking for these objects throughout the entire duration of the video.
[Tracking Data]
For each object, provide its estimated position time.
Record positions at key moments across the full video timeline when the object is clearly
visible in the frame.
Estimate distance meters from the camera to the object center.
Estimate direction in degrees (-90 left to 90 right, 0 forward) from the camera.
[Output]
Your complete and sole output must be single JSON object with the following structure:
"event": "Brief description of the event from the question",
"start_time" "minutos:seconds"" sand_time": "minutes:
"mode": procentric/alloeeett ooouedinggooject"
"description": "A detailed description of the sounding object. Include physical characteristics like type, color, material, and approxinate size/shape.",
iis_statcc; crue/false, // Set to true if the object is generally non-moving (like furniture, walls) and false if it typically Location (like person, animal, vehicle).
"key_france frames": I //tentire videow key visible frames "minutes:socondee :"distance;; "mcters", "diroction"; "degrees "}
treference_object // Stand by Reference Object camera "object_hame"; "Name"
"description; "Description",
"eem_frmmes: //ventire videos key visible frames "minutea:seconds ["distance": "maters", "di rectiiiin"": "degrees
"facing "faeing_object"" Becinngobjjee { // Facing the facing_object, empty for camera cbbject_namer "Name",
"description": "Description",
"key franes": I ventire video* key visible frames "minutes:seconds"" :"distance"; "meters", "direction"; "degrees
Figure 11: Prompt for Proprietary AV-LLMS (Gemini 2.5 models) on SAVVY-Bench.
33

---

To evaluate the robustness of SAVVY with the text-guided segmentation module (Seg). we conduct ablation studies on the ClipSeg confidence threshold (Seg thr) and the number sampled frames (N N_frame). We report sounding object localization accuracy (loc_ace scc_acc) and QA accuracy on both egocentric and allocentrie tasks from SAVVY-Bench. See the Experiments section of the main paper for detailed metric definitions.
For Seg thr, with all valid detections required to have at least one keypoint above 0.5. Results in Table 7 show stable performance across thresholds 0.3 0.7, with less than 3% variation. Lowering the threshold increases object recall, which improves sounding object localization accuracy (loc_ she__cc), as SAVVY's egotrack-basee outlier filtering and aggregation can effectively leverage the additional recalled samples. For QA tasks, 0.5 threshold yields the highest overall accuracy, while 0.3 improves distance-related QA but reduces directional accuracy.
values 0.3, 0.5, 0.7, and 0.9, using the average ClipSeg score across keypoints,
For N_frame, we evaluate 8, 16, 32, 64, and 128 frames (Table 8). Higher sampling rates lead to more valid detections from Seg, boosting sounding object loc_acc acc by 6.6% from 8 to 128 frames and improving egocentric QA accuracy. However, for allocentric QA, segmentation on static objects may introduce noise. As result, lower frame counts like 32 even can perform comparably frames. These findings suggest a hybrid strategy: use Seg for sounding objects and rely more on other egotrack types such as the Snapshot Descriptor for static objects.
128
Seg thr
Sound Loc loc_ace
Egocentric QA distance
Allocentric QA
direction
direction
distance
0.3
79.2
83.8
64.1
43,9
41.0
0.5
78.6
84.7
62.9
44.0
40.2
0.7
77.1
81.4
61.2
43.4
39.9
0.9
69.8
77.3
59.2
43.5
40.9
Table Ablation results on the average snapshot segmentation confidence threshold (Seg thr). We report sounding object localization accuracy (loc_acc) and accuracy on egocentric and allocentric QA tasks. Lower thresholds generally yield higher sounding object recall, improving localization and distance-related QA accuracy with SAVVY, while moderate thresholds provide balanced performance.
Seg N frame
Sound Loc loc_ace _c__ccc
Egocentric QA distance
Allocentric QA
direction
direction
distance
128
78.6
84.7
62.9
44.0
40.2
64
76.7
82.7
61.6
43.0
40.2
32
74.8
81.9
61.1
43.7
41.4
16
73.8
81.9
59.8
43.2
40.5
8
72.0
80.1
59.4
44.7
39.9
Table Ablation results on the number of sampled frames (N
(N_ frame) used text-guided snapshot segmentation. Increasing the number of frames improves sounding object localization and egocentric QA accuracy. However, allocentric QA performance is less sensitive and can degrade at high frame counts due noise in static object segmentation.
F.3 Spatial Audio Cues
We process spatial audio signals at 0.25s per segment, with sampling rate of 48 kHz. For each segment, we estimate the direction of arrival (DoA) by evaluating candidate angles over the full azimuthal range from -800 to 180", sampled 1 resolution. For each candidate angle, apply the Generalized Cross-Correlation with Phase Transform (GCC-PHAT) method on each microphone pair to compute ofiee-diffeeeneeo--eeee afrrivaa (TDOA) estimates. The angle à that maximizes the summed GCC-PHAT responses across all pairs is selected as the most likely direction of the source.
To assess the spatial diffuseness of the sound field for the sound source distance estimation, we compute the Coherent-to- Diffuss Ratio (CDR) from the multi-channel microphone signals. The input
34

---

to this process includes the raw microphone waveforms, the sampling frequency fs. microphone positions, and the estimated TDOAs for each pair. The analysis is constrained to the 500-20001 Hz frequency band for speech-related audio cues.
We estimate the power spectral densities (PSDs) and cross-spectral densities (CSDs) using Welch's s method, with a segment length of 1536 samples (around 32ms) and 50% overlap. We clip negative values to zero and compute the mean CDR over the selected frequency band. The final CDR is averaged across all microphone pairs and serves as a global indicator of the ratio between coherent tdirect-path) and diffuse (reverberant) components in the scene.
F.4 Egocentric Track Aggregation
In the second stage of SAVVY, we aggregate three egocentric object tracks- -pokduce by the Snapshot Descriptor, text-guided snapshot segmentation, and spatial cues-inte a unified global map. Each per-frame trajectory is transformed into global coordinates, forming global spatial map for downstream reasoning. The target object forms a time-varying global trajectory {Psouna | T while reference and facing objects are treated as static, with global positions Pref and Prace computed by averaging their per-frame locations. These together define the dynamic global map:
Mg = (t) |t E Tg) (Peet: Piace )
We describe the aggregation strategies for static and dynamic objects below.
Static objects. Since the Snapshot Descriptor (SD) are better at localizing static objects (reference/- facing) after track aggregation based our ablation results (see main paper ablations), we prioritize the SD track. If the SD captures the object, we apply DBSCAN clustering (maximum distance of 1 m) the SD track to determine stable location. If the SD fails to detect the object, we fall back to the text-guided segmentation-bass track (Seg), and apply DBSCAN with the same clustering threshold.
Dynamic sounding object. The Seg method is more accurate for tracking sounding objects (see main paper ablations), so we prioritize its trajectory when aggregating dynamic sound source tracks. We log Seg-tracked positions at each timestamp. For timestamps not covered by Seg, we query the SD track and filter outliers based on spatial consistency with the existing Seg trajectory. The resulting track is then extended by spatially fitting a smooth trajectory and removing outliers through the Seg-tracked points.
We then incorporate spatial audio cues to refine this trajectory. Specifically, we define a frustum- search region for audio tracks around the target direction and distance, spanning a distance range of +1 meter and an angular span of 45 degrees. We sample candidate points at the centers of 10 angular bins and 5 distance bins within this region. If the audio indicates that the object is located behind the camera (i.e., absolute angle 0 >> 90%), or provides positional information for timestamps not covered by Seg or SD, we refine the track by comparing with audio-based predictions. Inconsistent points are filtered based on spatial agreement with nearby extended accordingly to produce the final track.
estimates, and the trajectory is
The aggregation process can be summarized as Algorithm I
Discussion: What roles does the global mapping play in SAVVY?
Camera trajectory serves as the bridge between Stage 1 egocentric tracks and the Stage 2 dynamic global map. can be obtained using real-time SLAM technologies (2 60 with devices such as AR glasses or robotic sensors. Given camera pose (location and orientation), egocentric direction 0 and distance 7 can be transformed into global 3D coordinates. This transformation allows tracks from multiple modalitiss-Snapshoo Descriptor (SD). text-guided snapshot segmentation (Seg), and spatial audio cues (Audio)-to be aligned in a shared 3D coordinate system (global mapping). Different modalities may capture object trajectories at different timestamps; by mapping them to global frame, these partial observations can complement each other. Through outlier filtering and temporal smoothing, obtain reliable tracks for dynamic objects and stable positions for static ones.
Table compares performance with and without global mapping in terms of sounding object localiza- tion accuracy (loc_acc) and egocentric QA accuracy (direction and distance) on SAVV Y-Bnhh.. In the w/o Global Mapping setting, we directly take egocentric tracks from SD, Seg, and Audio based on the Snapshot Descriptor's grounded time span, then vote on direction and take the median angle and
35

---

Algorithm Track Aggregation Algorithm for Global Map Construction
1: Input: S, D (dense segmentation, SD, audio tracks); o (object type); L(t) (camera trajectory); Ty (query time range)
2: Define: MapToGl cb L()) L(t) +
(ree((0) r-ain(0) where (t, 0, r)
3: Initialize map M2 4-2
4: if o is static then
5: for each < D,S do
6: p(t) (apToGloball(, break end for
7:
8:
9:
P - centroid of clustered p(t)
10:
MM M, (p}
11:
else
Initialize trajectory p()) < Ø
13:
for each T, do for each in {S, A. if 1 E T do
14:
15:
Filter outhiers near p(t') p(t) Map end for end for
17:
18:
19: Interpolate and smooth p(t) over T
20: My M, (p(())
21:
end if
22:
return M,
distance at the queried time. Global mapping improves single-modality performance, especially for dense tracks like Seg and Audio, which see localization accuracy (loc_ace) gains of about 10%. SD, being sparse, is less sensitive global mapping and may perform better without it. For combined modalities, global mapping not only supports self-correction within each modality but also enables cross-modality completion, yielding even greater improvements-upp to 11.5% on egocentric distance accuracy and loc acc. Full SAVVY with all three tracks shows the strongest gains: +1 +11.9% in loc acc, + 14.3% in egocentric distance accuracy, and +4.1 4.11% % in direction estimation.
Track Type
w/ Global Mapping (SAVVY) direction
w/o Global Mapping direction
SD
Audio
Seg
loc_acc
distance
loc ace
distance
55.7
68.3
47.9
56.3
71.1
52.6
59.0
73.9
48.1
49.7
75.6
40.1
V
72.5
81.2
52.0
62.3
75.8
43.7
66.8
74.5
54.6
55.3
73.0
43.3
78.6
84.7
62.9
66.7
80.6
48.6
Table Ablation study on the impact of global mapping. We evaluate combinations of egocentric track modalities-Snapshoo Descriptor (SD), Spatial Audio (Audio), and Segmentation (Seg) and without global coordinate transformation. Metrics include sounding object localization accuracy (loc_acc) and egocentric QA accuracy on SAVVY- Bench (direction and distance), Global mapping consistently enhances performance, particularly when aggregating dense tracks (Seg and Audio) and integrating multiple modalities.
G Efficiency Analysis
We report average latency and peak GPU memory for each stage over 200 test samples on a single
NVIDIA A100. 1) SLAM: real-time on Aria glasses. 2) SD: one AV-LLM forward pass per QA
(latency comparable to standard AV-LLM inference). 3) Audio & Global Map: fewer than 0.1 per sample on CPU. 4) Seg: 0.52 s/frame 512x384 per object; peak GPU memory 9.4 GB. Depth estimation costs x 0.44 s/frame (up to 6 GB). While Seg is the primary bottleneck of the efficiency, Table X shows that using 32 frames per video for Seg maintains strong accuracy. Besides, removing Seg [Table main paper) substantially reduces runtime while still outperforming LLM-
36

---

only baselines, particularly on allocentric questions. Overall, modular design of SAVVY enables uccuracy-efficiene trade-offs meet real-time deployment constraints.
H
Blind Testing
We conduct blind testing to evaluate the contribution of the visual modality in audio-visual spatial reasoning on SAVVY-Bench, using AV-LLM baseline models. Specifically, we compare performance between settings: Audio Only (removing visual frames, using only audio and the text query as input) and Audio + Visual (using both modalities). We evaluate on egocentric QA tasks to assess how models infer the direction and distance of sound sources relative the camera.
We test the top five open-source 7B models and the strongest proprietary model, Gemini-2.5-pp As shown in Table I0 Gemini demonstrates strong grounding capabilities (67.4% t-mloU, reported in the main paper), and its performance shows clear dependence on visual input. Under Audio Only, Gemini's direction accuracy drops sharply by 32.4%, while distance accuracy decreases by only 2.8%. This aligns with observations from our reasoning process visualizations: Gemini relies heavily on visual input for spatial direction reasoning. whereas distance estimation is less affected- due to the role of commonsense priors from audio and language.
Other AV-LLMs exhibit similar trends: direction accuracy degrades more under Audio Only, while distance accuracy remains relatively stable or even improves for some AV-LLMS such as MiniCPM-o. However, the performance gap is smaller than with Gemini, likely because these models fail to reliably ground events in time achieving less than 5% As result, even with visual input, their spatial reasoning remains limited.
of the input modality.
Audio Only Distance
Audio + Visual
Method
Direction
Direction
Distance
Videol.LaMA2-78 (1
39.1
40.7
46.4
42.7
MiniCPM-o 2.6 (25
41.9
50.7
45.8
42.3
EgoGPT 23
39.3
37.0
40.2
57.6
Gemini-2.5-pro
42.8
56.8
75.2
59.6
Table 10: Blind testing SAVVY-Bench: comparison between Audio Only and Audio + Visual input settings. Reported metrics are egocentric QA accuracy for direction and absolute distance. Gemini-2.5-pro shows the largest gap, indicating strong reliance visual input for accurate direction estimation.
I Limitations
One limitation of SAVVY is that it currently relies strong foundational AV-LLM AV-LLM--peecfficll Gemini-and inherits its capabilities temporal grounding and object referral. The pipeline may underperform if the base model lacks these abilities in the initial stage. Additionally, the spatial audio tracking module uses rule-based signal processing: while effective for direction estimation, distance estimation remains challenging, particularly given the wide variance of near- and far-field cases in the current dataset. Future work could improve audio-visur track aggregation by enhancing this module through large-scale training realistic spatial audio data.
J Broader Impacts
This work contributes to the development of AV-LLMs capable of fine- ain-ggrined spatial reasoning in dynamic 3D environments. By introducing benchmark and training -free pipeline that enables structured spatial understanding across audio and visual modalities, our work opens new avenues for intelligent multi-modal systems domains such as assistive roboties, AR/VR, human-computer interaction, and audio-visual navigation [72). These capabilities have the potential to significantly enhance accessibility tools (e.g., guiding visually impaired users through complex spaces), improve AR/VR user experiences, and support more context-aware AI agents in embodied environments.
37

---

However, alongside these benefits, the increasing power of AV-LLMs introduces potential risks. Models capable of interpreting spatial relationships from audio-visual input could be misused in surveillance applications, unauthorized tracking, or context inference without user consent. Moreover, as our method builds on these foundation models, inherits their limitations and biases, which can propagate through the pipeline and affect real-world deployments. There is also the risk that such models may make confident but incorrect spatial inferences safety-critical settings. To mitigate these concerns, recommend that future systems incorporating AV-LLMs for spatial reasoning include safeguards such as: (1) explicit transparency about model uncertainty and failure modes; (2) data collection and evaluation guidelines that prioritize privacy and ethical use of human-centert audio-visual data; and (3) usage restrictions for sensitive applications, especially those involving biometric data or real-time environmental monitoring. Furthermore, research into interpretability and robustness of spatial reasoning components will be critical for safe deployment.
K Additional Qualitative results: Reasoning Error Analysis
In this section, we show additional reasoning examples of Gemini-2.5-pa and conclude four major types of errors in the visualization:
1) Referral Error: This error occurs when the model fails to correctly identify, locate, or interpret the properties of specific objects, persons, or abstract reference points mentioned the question. is particularly common when the referenced object descriptions are complex, rely on relative positioning (e.g.. "the armchair further from the wall painting"), or refer to abstract sound events (e.g., "a thud sound") that are not tied clearly visible object and must be inferred from broader video context. The model may select an incorrect referent or misinterpret its attributes, leading to flawed premise for subsequent spatial reasoning. An example is shown in Figure [2 where the model incorrectly identifies the queried armchair (the facing object) as the one at the arched opening.
This error occurs when the model fails to accurately identify the correct time span of the queried sound event in the question. As a result, the model analyzes the spatial context at incorrect point in time, leading to flawed reasoning about object locations or spatial relationships. Figure 13 shows an example where the model confuses the speech event "suggesting trying the coffee" with another semantically similar topic, "complimenting the coffee taste," leading to an error in egocentric direction prediction.
2) Temporal Localization Error:
3) Spatial Relationship Error: This error occurs when the model misinterprets or misapplies funda- mental spatial relationships (e-g., left/right, front/back, in front of/behind, next to, between) between correctly identified entities, even within correct frame of reference. In Figure 14, the model suc- cessfully identifies the correct event time span, detects all relevant objects as well as their locations. However, it fails to interpret the relative direction correctly, placing the object on the right side of the robot's view instead of the left, resulting in an incorrect prediction of "front-right" rather than the correct "front-left."
4) Spatial Measurement Error: This error arises in tasks that require quantitative responses- such as estimating distances making precise angular judgments (e.g.. in Snapshot Descriptor-based tasks). Even when the model correctly identifies the relevant objects and understands their qualitative spatial relationships, it may still make significant errors in geometric reasoning (e.g., applying Pythagorean theorem incorrectly, flawed calculation logic), scale estimation, or numerical calculations. Figure [55 presents example where the model correctly identifies both the sound source and the queried reference object, and even retrieves a relevant navigation path between them. However, fails to calculate the distance accurately. This case also reveals typical reasoning pattern in AV-LLMs for distance estimation: the model anchors the sound source and reference object to static landmarks in the scene, recalls the relevant navigation routes observed in the video, and then estimates the distance according to the routes.
38

---

Sound Event
Allocentric Direction
Q:
Imagine robot standing by the and facing the light-gray armchair further from the wall painting, when the creak sound comes up, relative to where you are facing, where sounding object: front-left, front-right, back-left, or back-right?
The robot stands by the TV (right of the fireplace) facing the light-gray gray armchair at the arched opening, the furthest from painting, 00:11-00:12, creak occurs the woman walks the front door toward the coffee table. 3. She is near the coffee table in front of the sofa, left of the fireplace.
SAV-LLM: ffottleff
GT:
front-right
origin facing into room (+y), y.negative x), so the sound is the front:left quadrant.
left (positive
[C Referral
D Temporal Localization
m.p Spatial Relationship
Figure 12: Example of Referral Error. The model incorrectly selects the armchair (facing object) at the arched opening, failing to identify the correct object description queried in the question. This misinterpretation further causes error for the spatial relationship.
Speech Topic
Egocentric Direction
Imagine you the wearer, relative to where you are facing, where is the other person: front-left, front-right, back-left, or back- right?
speech topic "suagesting trying the coffes" up,
A: The speech topic "suggesting trying the coffee" comments on This happens around 01:33 01:37 when the other person says "Tastes pretty good."! 2. At this paint in the video, the doors leading to the pool. The other person sitting at small table to the left of the camera wearer".
when the other
tastes the coffee and
LLV-LMM: front-left
facing towards the kitchen island and the sliding glass
GT:
back-right
( Temporal Localization
Referral
Inp Spatial Relationship
Figure 13: Example of Temporal Localization Error. The model incorrectly grounds the speech event "suggesting trying the coffee," confusing it with similar topic. This mismatch causes incorrect egocentric direction prediction.
39

---

Speech Tonic
Allocentric Direction
Imagine you are a robot standing by the stainless steel double wall oven and facing the countertop drip coffee maker, when the speech topic: "commenting
comes up, relative where you are facing, where is the speaker: front-left, front-right, back-left, or back-right?
enlovable
1. Timestamp of Speech: The speech topic occurs when the male speaker says, Thht was good game' approximately 00:00-00:01.- 2. Spea is sitting the dining table. Kitchen Leyout Analysis from Video: The double wall is located on the far wall of the kitchen, to the right of the stove/boftee maker area one fecing that counter the dining room. The countertop drip coffec maker on main kitchen counter, the the stove. 4. Determining Robot's Facing Direction: Let the robot
Timestamp: ... male speaker
the origin (0,0). The direction it is facing (towards the coffee maker) is the positive Y-8xis ('front'.. 5. Relating Speaker's Location to Robot's Orientation: "aut' from kitchen counter, the dining counter the dining table area is the robot's side. The 'righ' side of the robot is the positive X- axis. The dining table ls generelly in the space front of the kitchen counter. Therefore, speaker at the dining table in quadrant that the robot's right and in forward field view.
AV-LLM:
front-right
male speaker is
dining table. dining table located the robot the and facing left along the
GT:
front-left
i Temporal Localization
Referral
0 Spatial Relationship
The model correctly identifies the queried time span, relevant objects, and their locations, but fails to determine the correct relative direction of the sounding object with respect the robot's view. It incorrectly predicts the object being the right side instead of the left, leading to a misclassification of "front-right" instead of the correct "front-left."
Figure 14: Example of Spatial Relationship Error.
Sound Event
wimmonmnnn AEATEII www.eae......................... www. CEITIII
Allocentric Distance
When the water dripoing mirror and the sounding object in meters? Consider the center point location of the object as the its location,
happening, the distance the white dresser
Identify objects: format.
A:
bedroom start of the video (0:00). The second object the ssunding object', which is the Keurig coffee machine making coffee. The sound of the Keurig (water dripping/brewing// is audible starting around 01:56, and the machine person (camera's point of view) with a woman who walks from the bedroom to the kitchen. We need to estimate the displscement vector from the dresser to the Keurig. Segment [S1]: Effective movement the bedroom.
object
dresser mirror,
3.99m
the kitchen counter.
the and
lengths:
the dresser's location door, Estimated length L1 meters. Let's assume this is in the East (E) direction. Segment (S2): Movement through the empty room after exiting the bedroom. She walks straight. Estimated length right
crosses bedroom
the
AV-LLM:
6.80m
GT:
3.99m
corridor. Estimated length L3 1.5 meters. Direction: South (S). Segment [54): Movement through the living room, continuing straight from the corridar. Estimated length
meters. Direction: (E). Segment [53]:
South)
meters. Direction: South (S). kitchen/dining
Segment (55):
right again (from South,
West)
Keurig. Estimated length LS meters. Direction: West (W). Calculate Net Displacement: Net East-West West displacement: Initial 6m East (L1+12) 2m West (L5) meters East. Net North-South displacement: South Calculate Euclidean Distance: dresser considered the origin The Keurig's final pasition relative to the dresser is [4m East, 5.5m South). Distance sqrt[[Net East West displacement)22 (Net North-South displacement)22) Distance sqrt(4*2 5.5*2) Distance sqrt(16 30.25) Distance sqrt(46.25), The distance 6.80
to Referral
ró Temporal Localization D, Spatial Measurement
Figure 15:
Example of Spatial Measurement Error The model correctly identifies the sound source and reference objects, but fails to compute the distance accurately along with the navigation route from the reference object to the sound source.
40


Reviews:
[{"review_id": "vprkyeYWBM", "reviewer": "Reviewer_jLXv", "summary": "This paper proposed a new dataset, named as SAVVY-Bench, for visual-audio based 3d spatial reasoning in dynamic audio-visual environments. The problem introduced by the benchmark is interesting and important for community. Besides, the authors also proposed a novel pipeline SAVVY, including egocentric spatial tracks estimation stage and dynamic global map construction for visual-audio based 3d spatial reasoning with AV-LLM. The results on SAVVY-Bench demonstrate the effectiveness of the proposed method.", "questions": "1）The main question I considered is the synergy between visual and audio signals. Many of the tasks presented in the paper can be solved using visual signals alone, so it is important to clarify what complementary information the audio signals provide comparing with the method using videos to build the scene and preform 3d reasoning, and how much additional value they contribute to the overall task.\n2）In this benchmark, there are many 3D scene reasoning tasks. In this context, would it be possible to incorporate step-by-step reasoning similar to Chain-of-Thought approaches? Moreover, could multi-round reasoning be applied in this process? Multi-round reasoning is particularly important for embodied tasks.", "limitations": "The main limitation of this paper is that it only compares reasoning performance based on large language models. To some extent, this is not entirely fair. The mapping and scene construction methods proposed in the paper could potentially be completed using only video input. Moreover, the relevant information extracted from these processes could also be incorporated into large model-based reasoning. A more detailed and comprehensive comparison is needed to better evaluate the effectiveness of the proposed approach.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\nThis paper addresses a very interesting and important problem: how to perform 3D reasoning in dynamic environments with multimodal inputs, specifically audio and text. The authors provide a thorough analysis of this problem and propose a benchmark to facilitate further study. In addition, they introduce a localization method that leverages large models. The final results demonstrate the effectiveness of the proposed approach.\n\nWeaknesses:\n1）This paper mainly focuses on the audio modality. However, in real-world 3D reasoning tasks, visual signals play a crucial role. The paper lacks sufficient comparison between audio and visual signals—for example, the advantages of using audio signals versus purely visual signals, and whether combining audio and visual signals provides complementary benefits. It is also unclear whether a purely video-based approach would already be sufficient to address the challenges raised by the benchmark. More analysis is needed in this regard.\n2）The reasoning tasks discussed in this paper are primarily oriented toward navigation and spatial localization. It would be interesting to explore whether audio signals can support higher-level semantic reasoning tasks, such as object grounding via sound, scene-level question answering (e.g., “How many people are there?”), or attribute reasoning (e.g., distinguishing between male and female voices).\n3）The generalizability of the proposed method needs further evaluation. Since the method is training-free, it is important to investigate whether it can be applied to outdoor environments or driving scenarios.", "was_revised": true, "final_justification": "The authors have addressed most of my concerns. As the authors proposed in rebuttal, I think the benchmark and method proposed in this work are good starting points for dynamic audio-visual environment understanding and reasoning, and will bring some inspiration to the community. Therefore, I raised my rating to Accept.", "rebuttal": "We thank the reviewer for providing detailed and helpful feedback, and we are glad that the reviewer thinks that our work is interesting, provides thorough analysis, and the proposed method is effective. We address the questions and concerns below.\n\n**1. Synergy between Visual and Audio Signals.**\n\n+ **Limitations of Pure Visual Input:** While scene construction and 3D reasoning can, in principle, be approached using video inputs only, relying solely on visual signals typically introduces limitations—especially in the context of audio-centric questions and events. **Firstly, questions from SAVVY-Bench focus on audio events** such as speech, making them difficult to answer using visual input alone. **Secondly, visual cues are often unreliable in out-of-view or occluded scenarios.** When the sound source disappears from view for extended periods, the model inaccurately estimates its trajectory based on its last visible location. \n\n+ **Blind Testing (Pure Audio Input):** As shown in Supplementary Table 4, the blind testing that we conducted reveals that adding visual input to pure audio input can offer limited benefit for 7B open-source models, with increased benefit for models such as Gemini 2.5 Pro, which possess stronger visual understanding and temporal reasoning capabilities. For example, in Section 5.4, we show that our testing reveals that Gemini-2.5-Pro significantly relies on visual input to localize sound sources—linking audio to visible objects, grounding event time, and inferring direction.\n\n+ **Complementary information from combining visual and audio:** **Spatial audio naturally captures off-screen and out-of-view information, providing continuity where visual input cannot provide it. SAVVY-Bench highlights such challenging though typical cases and SAVVY demonstrates that integrating spatial audio cues with vision enables robust reasoning in these situations.**\nWhile human visual attention and egocentric cameras (e.g., Aria’s 110° HFOV) are restricted to a limited frontal field of view, the acoustic environment inherently provides 360-degree coverage. Auditory cues—such as appliance alerts, speech from adjacent rooms, or footsteps from behind—convey critical contextual information beyond the visible scene, highlighting the role of spatial audio in achieving a more complete perception of real-world environments.\nThis is key for developing safer and more reliable spatial intelligence in multi-modal LLMs. Our results underscore the complementary strengths of both modalities and point to promising directions for future research in multi-modal spatial reasoning.\n\n**2. Limited Reasoning Tasks.**\n\nThe evaluated tasks aim to represent realistic interplay between audio-visual-language understanding. While more abstract reasoning (e.g., voice gender classification or crowd counting) is valuable, **our focus is on fundamental skills—such as direction, distance, temporal grounding, and grounded object referral—that are critical to real-world applications and pave the way for broader reasoning capabilities.** During our work on this paper we found that even such fundamental tasks, without further complexities, require novel approaches in multi-modal learning and preferred to focus on these as a first benchmark (SAVVY-Bench) and approach (SAVVY).\n\n\nNoteworthy, we do introduce some amount of complexity into the benchmark. **Beyond direction and localization QAs, our benchmark also introduces Snapshot Descriptor Evaluation (SD-Eval) in Table 3 (Section 4.2)**, covering: **(i) Temporal Grounding** – measurement of how accurately a model would localize queried sound events in time. **(ii) Object Referral** – tests whether the model correctly describes objects given the question and the video. We also propose a new localization accuracy metric comparing predicted and ground truth positions to test SAVVY and its variants.\n\nIndeed, most 7B open-source models achieve <5% temporal mIoU, under 35% referral accuracy, and perform only slightly above chance on direction and distance QA—highlighting that current AV-LLMs still struggle with temporal alignment, semantic grounding, and basic audio spatial reasoning. These limitations make it difficult to meaningfully evaluate more complex tasks beyond what SAVVY-Bench currently covers.\n\n**3. Generalizability to Diverse Scenarios.**\n\nWhile our current dataset focuses on high-quality indoor scenarios, the SAVVY model and input/output formats are not inherently limited to indoor environments. When evaluating LLMs, we use stereo audio and rectified camera views to promote broader applicability.\n\nWe acknowledge that full generalization to outdoor or driving environments is challenging, primarily due to degraded audio quality in such settings (e.g., wind, road noise). Our current focus is on high-quality, controlled indoor scenarios as a foundational step for studying spatial audio-visual reasoning. This allowed us to establish a reliable benchmark and isolate core model capabilities. That said, we see this as the starting point of an iterative process.\n\n\n**4. Step-by-Step Reasoning.**\n\nWe thank the reviewer for this proposal and fully agree—supporting step-by-step and multi-round reasoning is one of the key motivations behind the SAVVY pipeline and SAVVY-Bench. While current models have plenty of room for improvement with grounding and spatial alignment, **we see a great potential in future work that distills SAVVY-style reasoning chains into LLMs to enhance their spatial and temporal understanding with audio and visual modality**. Our benchmark provides a structured foundation to study such reasoning processes, and we hope it encourages future research in this direction—especially toward more embodied and interactive tasks.\n\nSpecifically, our pipeline naturally decomposes spatial reasoning into interpretable steps:\n+ Temporal grounding (when did the event occur?)\n+ Object referral (what objects are involved?)\n+ Spatial localization (where are they in 3D space?)\n+ Trajectory tracking (how do they move over time?)\n+ Spatial relationship reasoning (what are the relative distance and direction?)\n\nEach intermediate output in SAVVY represents a reasoning step that current end-to-end models struggle to produce explicitly. These outputs could be used to create training data for teaching LLMs to perform structured spatial reasoning."}, {"review_id": "sWTxUjDMEY", "reviewer": "Reviewer_cjrp", "summary": "This paper introduce the the first benchmark for 3D spatial reasoning in dynamic scenes with synchronized spatial audio: SAVVY-Bench, and and a fully training-free two-stage pipeline that leverages off-the-shelf components \n*(pretrained audio-visual LLMs for timestamped snapshot extraction, segmentation and monocular depth networks for object localization, SRP-PHAT/CDR for audio direction-of-arrival and distance estimation, SLAM-based camera poses for global mapping, DBSCAN clustering for static object centroids, and Kalman filtering for dynamic trajectory smoothing)*\nto accurately answer “which way?” and “how far?” questions in moving egocentric video with spatial audio, all without any additional model fine-tuning.", "questions": "- Can the author report end-to-end latency (per QA) and peak memory/GPU utilization on representative hardware?\n- Has the author tested performance on non-Aria or outdoor recordings? It would be great if the author could report the metrics.\n- Can the author quantify the accuracy of your zero-shot LLM-based event timestamps? I feel like this misalignment here will affect downstream spatial accuracy.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper is well-organized and easy to follow.\n- Benchmark (3D spatial QA with multi-channel audio) is novelty.\n- Training-free design is good. Leverages existing models and signal-processing methods without additional fine-tuning.\n- Evaluations and ablation studies are comprehensive, which can well support the paper's main point.\n\nWeaknesses:\n- Limited scope of spatial reasoning: SAVVY-Bench only tests “which way?” and “how far?” Q&A. It doesn’t cover richer 3D reasoning tasks such as mapping unknown space, occlusion handling, trajectory prediction, semantic layout, or path planning, which limits its applicability as a general spatial reasoning benchmark.\n- Pipeline Complexity: Even though it's training-free, it seems to stitch many heavy components: LLM, CLIPSeg/SAM2, depth, SLAM, SRP-PHAT, DBSCAN, Kalman. Which lead to hard to deploy and maintain.\n- The paper omits inference time or resource usage, lacking assessment of real-time feasibility. \n- Data is limited to indoor Aria headsets in controlled environments; outdoor generalization is unclear.", "was_revised": true, "final_justification": "My final recommendation is to Accept.\n\nMy initial concerns about the pipeline's efficiency and the benchmark's limited scope were resolved by the authors' rebuttal.\n\n- Key Issues Resolved: The authors provided the latency and memory metrics, confirming the method's practical feasibility.\n- Remaining Limitations: The use of indoor-only data is an acknowledged limitation but is acceptable for a paper establishing a novel benchmark.", "rebuttal": "We thank the reviewer for the helpful feedback and fruitful comments, and appreciate that the reviewer finds our paper novel in the aspect of benchmark, and notes favorably the design of the proposed training-free method. We address the questions and concerns below.\n\n**1. Limited Reasoning Scope.**\n\nWe view SAVVY as a **modular, extensible foundation for future 3D spatial reasoning benchmarks**—including occlusion-aware reasoning, semantic layout, and trajectory forecasting. The evaluated tasks aim to represent realistic interplay between audio-visual-language understanding. While more abstract reasoning is valuable, our focus is on fundamental skills—such as direction, distance, temporal grounding and grounded object referral—that are critical to real-world applications and pave the way for broader reasoning capabilities. During our work on this paper, we find that even such fundamental tasks, without further complexities, require novel approaches in multi-modal learning. We preferred to focus on these fundamental tasks, proposing the first benchmark (SAVVY-Bench) and corresponding approach (SAVVY).\n\nNoteworthy, we do introduce a sufficient amount of complexity into the benchmark. **Beyond direction and localization QAs, our benchmark also introduces Snapshot Descriptor Evaluation (SD-Eval) in Table 3 (Section 4.2)**, covering: **(i) Temporal Grounding** – measurement of how accurately a model would localize queried sound events in time. **(ii) Object Referral** – tests whether the model correctly describes objects given the question and the video. We also propose a new localization accuracy metric comparing predicted and ground truth positions to test SAVVY and its variants.\n\nIndeed, most 7B open-source models achieve <5% temporal mIoU, under 35% referral accuracy, and perform only slightly above chance on direction and distance QA—highlighting that current open-source AV-LLMs still struggle with temporal alignment, semantic grounding, and basic audio spatial reasoning. These limitations make it difficult for these models to be evaluated meaningfully on more complex tasks beyond what SAVVY-Bench currently covers.\n\n\n**2. SAVVY Complexity and Runtime Efficiency.**\n\nWe acknowledge that SAVVY is a multi-stage pipeline. The multiple-stage design is necessary due to the complexity of multi-modality in the task setup. However, an important advantage of the pipeline is that it is designed to be training-free and modular, which allows components to be adapted or replaced as needed. Each stage serves a targeted role in spatial reasoning (e.g., segmentation for dynamic object tracking, audio cues for out-of-view events). Furthermore, such a structured pipeline also lays the groundwork for future research on reasoning knowledge distillation as well as chain-of-thought in AV-LLMs.\n\nIn regards to runtime efficiency, we include detailed latency and peak memory usage metrics below (will also be incorporated in revised version of the manuscript).\nThe time and memory costs (where applicable) for each step of the SAVVY pipeline are:\n+ SLAM: real-time slam on device (aria glasses).\n+ Stage 1 Step 1 (Snapshot Descriptor): Runs the AV-LLM once per QA to determine temporal spans and object roles—this + matches standard AV-LLM inference latency on our benchmark (Table 2 and 3).\n+ Stage1 Step 2 (Text-Guided Snapshot Segmentation with Depth Estimation): Segmentation (CLIPSeg/SAM2) is the most time-consuming (~0.52s for one frame at 512×384 per object), but we show in Supp. Table 2 that using just 32 frames uniformly sampled in one video still yields strong performance.\nDepth estimation takes ~0.44s per frame and ~6GB GPU memory; segmentation up to ~9.4GB.\nThe speed and memory were tested on one NVIDIA A100 GPU, average on 200 testing samples. \n+ Stage1 Step 3 (Spatial Audio Cues): Lightweight—0.06s per second of audio, CPU-only.\n+ Stage 2 (Global Map Construction): Fast and efficient—0.08s per video, CPU-only.\n\nImportantly, removing the segmentation step (Table 6) significantly improves efficiency, while still outperforming LLM-only baselines, especially on the allocentric question category. This shows SAVVY can flexibly trade off between accuracy and runtime.\n\n**3. Limited to Indoor Aria Datasets.**\n\nWe acknowledge that our current dataset is based on indoor scenes captured with Meta Aria glasses. This choice allows for high-quality, spatially synchronized multi-modal data—key for establishing a strong benchmark consisting of fundamental tasks. It is noteworthy to mention that SAVVY’s design is not limited to indoor environments. The pipeline operates on spatial audio and video with 6-DoF camera poses, and can generalize to outdoor or driving scenarios when higher-noise, less-structured data becomes available. We view this as an important next step in extending SAVVY-Bench toward more diverse, real-world settings.\n\n**4. Quantify the Accuracy of Zero-Shot LLM-based Event Timestamps.**\n\n**We evaluated temporal event grounding accuracy in Table 3; detailed settings are provided in lines 226–229 of the paper.** This task measures how accurately a model localizes the queried sound event in time, using IoU between predicted and ground-truth intervals. We report Recall@1, averaged over IoU thresholds from 0.05 to 0.5, summarized as mean temporal IoU (t-mIoU). \nAs the reviewer points out, temporal alignment is important. Results show that most open-source 7B models achieve under 5% t-mIoU, indicating poor event-time alignment. Synchronizing complex events like speech remains a major challenge at this scale. To further investigate, we conduct experiments using the ground-truth Target Clip (i.e., the relevant event segment only), as shown in Supplementary Table 5. Results confirm that temporally aligned input consistently improves direction accuracy across all models compared to full-video input."}, {"review_id": "e5fWSUOViq", "reviewer": "Reviewer_tKUJ", "summary": "The authors design a benchmark, SAVVY-Bench, to evaluate the spatial reasoning capabilities of audio-visual large language models (AV-LLMs). This is the first work to incorporate spatial audio as a condition in spatial perception tasks for LLMs. The benchmark evaluates existing AV-LLMs from both egocentric and allocentric perspectives, across both direction and distance dimensions. The authors also propose SAVVY, a training-free multimodal spatial reasoning system built on top of a state-of-the-art AV-LLM. SAVVY integrates three distinct spatial direction and distance estimation methods: Snapshot Descriptor, Text-Guided Snapshot Segmentation, and Spatial Audio Cues. The outputs from these three methods are denoised (outlier removal) and aggregated to construct egocentric spatial tracks, which are then used to build a Dynamic Global Map.\n\nIn experiments, the authors demonstrate that SAVVY outperforms all existing state-of-the-art AV-LLMs on the SAVVY-Bench benchmark. However, all AV-LLM systems, including SAVVY, still lag behind human performance in spatial perception tasks, indicating that there remains significant room for improvement in audio-visual spatial reasoning for LLMs. Ablation studies further validate that the three proposed methods for egocentric spatial track construction each contribute uniquely to different aspects of spatial perception. Moreover, the experiments show that combining all three methods yields the best overall performance in audio-visual spatial reasoning.", "questions": "1. In Table 6, the spatial reasoning capabilities of the combinations SD+Seg and Audio+Seg are not compared. It would be helpful if the authors could include results for these two cases. There is a possibility that one of the methods used for egocentric tracks construction might have a significant negative impact, causing the combination of all three methods to perform worse than the combination of just two methods.\n\n2. In Table 5, the spatial direction and distance estimation performance of Audio is worse than SD. However, Table 6 shows that Audio outperforms SD in terms of sound localization accuracy and egocentric direction and distance estimation. Are these two results in conflict? Could the authors provide an analysis of this discrepancy?\n\n3. Is there a computational overhead comparison between SAVVY and other methods? The pipeline combines SLAM, AV-LLMs, image segmentation, and audio processing. Latency/throughput metrics are absent, making real-time deployment feasibility unclear.", "limitations": "1. The question types in SAVVY-Bench are relatively monotonous, with only 4 types of questions included.\n\n2. The computational overhead of SAVVY could be significant because of the combination of so many foundation models.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "a) Strengths\n1. The authors are the first to introduce spatial audio into spatial reasoning for LLMs. From the perspective of multimodal spatial perception, integrating both audio and visual spatial cues aligns more closely with the way humans perceive space in real life. This task is both meaningful and novel.\n2. The SAVVY-Bench constructed by the authors has undergone meticulous data preprocessing, annotation, QA synthesis, and quality review processes. The methodology used to build the benchmark is rigorous.\n3. The proposed SAVVY system estimates spatial direction and distance from the vision perspective using SD and Seg methods, and from the audio perspective using existing spatial audio localization and distance estimation models. The system effectively removes outliers and integrates information from both vision and audio, making excellent use of both modalities and achieving performance that surpasses the state-of-the-art AV-LLMs.\n4. In terms of both writing and experimentation, the authors provide comprehensive quantitative and qualitative analysis.\n\nb) Weaknesses\n1. The question types in SAVVY-Bench are relatively monotonous, limited to only egocentric/allocentric and direction/distance, resulting in the evaluation of a rather narrow spatial reasoning task.\n2. The system is described as a training-free approach, which is a strength. However, this may come with computational trade-offs. It would be useful to discuss the computational cost of the SAVVY system in comparison to other state-of-the-art models, especially since it integrates multiple methods (e.g., Snapshot Descriptors, Segmentation, and Audio Cues). This could be particularly important for practical deployment in real-time systems.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for detailed and supportive feedback, and we are glad to see that the reviewer acknowledges the novelty of our work, the quality of the proposed benchmark, the effectiveness of SAVVY, and solid experiments along with detailed analysis. We address the questions and concerns below.\n\n**1. Limited Question Types.**\n\nThe evaluated tasks aim to represent realistic interplay between audio-visual-language understanding. While more abstract reasoning (e.g., voice gender classification or crowd counting) is valuable, **our focus is on fundamental skills—such as direction, distance, temporal grounding and grounded object referral—that are critical to real-world applications and pave the way for broader reasoning capabilities**. During our work on this paper, we find that even such fundamental tasks, without further complexities, require novel approaches in multi-modal learning. We preferred to focus on these fundamental skills, proposing the first benchmark (SAVVY-Bench) as well as the approach (SAVVY).\n\nNoteworthy, we do introduce some amount of complexity into the benchmark. **Beyond direction and localization QAs, our benchmark also introduces Snapshot Descriptor Evaluation (SD-Eval) in Table 3 (Section 4.2)**, covering: **(i) Temporal Grounding** – measurement of how accurately a model would localize queried sound events in time. **(ii) Object Referral** – tests whether the model correctly describes objects given the question and the video. We also propose a new localization accuracy metric comparing predicted and ground truth positions to test SAVVY and its variants.\n\nIndeed, most 7B open-source models achieve <5% temporal mIoU, under 35% referral accuracy, and perform only slightly above chance on direction and distance QA—highlighting that current open-source AV-LLMs still struggle with temporal alignment, semantic grounding, and basic audio spatial reasoning. These limitations make it difficult for these models to be evaluated meaningfully on more complex tasks beyond what SAVVY-Bench currently covers.\n\n**2. SAVVY Complexity and Runtime Efficiency.**\n\nWe acknowledge SAVVY’s multi-stage pipeline, but it is training-free and modular by design—allowing components to be adapted or replaced as needed. Each stage serves a targeted role in spatial reasoning (e.g., segmentation for dynamic object tracking, audio cues for out-of-view events) and also lays the groundwork for future research on reasoning knowledge distillation and chain-of-thought in AV-LLMs.\n\nIn regards to runtime efficiency, we include detailed latency and peak memory usage metrics below (will also be incorporated in revised version of the manuscript): \n\nThe time and memory costs (where applicable) for each step of the SAVVY pipeline are:\n+ SLAM: real-time slam on device (aria glasses).\n+ Stage 1 Step 1 (Snapshot Descriptor): Runs the AV-LLM once per QA to determine temporal spans and object roles—this matches standard AV-LLM inference latency on our benchmark (Table 2 and 3).\n+ Stage1 Step 2 (Text-Guided Snapshot Segmentation with Depth Estimation):\nSegmentation (CLIPSeg/SAM2) is the most time-consuming (~0.52s for one frame at 512×384 per object), but we show in Supp. Table 2 that using just 32 frames uniformly sampled in one video still yields strong performance.\nDepth estimation takes ~0.44s per frame and ~6GB GPU memory; segmentation up to ~9.4GB.\nSpeed and memory were tested on one NVIDIA A100 GPU, average on 200 testing samples. \n+ Stage1 Step 3 (Spatial Audio Cues): Lightweight—0.06s per second of audio, CPU-only.\n+ Stage 2 (Global Map Construction): Fast and efficient—0.08s per video, CPU-only.\n\nNotably, the most time-consuming stage is segmentation, while other steps—such as audio processing and global mapping—are lightweight and run efficiently on CPU. Without segmentation, the total latency is comparable to a single AV-LLM inference (Table 6), and the pipeline remains feasible for near real-time use in controlled settings. \n\nWe view SAVVY as a modular and extensible framework rather than a fixed system—paving the way for real-time, efficient adaptations in future work.\n\n**3. On Missing SD+Seg and Audio+Seg Results in Table 6.**\n\nThank you for the suggestion to add results of SD+Seg and Audio+Seg —we have added the missing combinations in Table 6:\n\n\n| Track Type | loc_acc | Ego Dir | Ego Dist | Alloc Dir | Alloc Dist |\n|------------|---------------|---------|----------|-----------|------------|\n|SD+Seg|76.0|83.2|54.4|44.2|36.9|\n|Audio+Seg|73.6|84.2|57.7|34.4|24.0|\n\n\nTogether with other results in Table 6, these findings highlight the complementary strengths of each modality:\n+ **Text-guided segmentation significantly contributes to localizing dynamic sounding objects**, improving direction grounding, especially for egocentric questions.\n+ **Spatial audio cues enhance distance estimation largely and also improve both localization and direction QA accuracy**.\n+ **Snapshot Descriptor (SD) performs best on allocentric reasoning by more accurately localizing reference and facing objects** to construct meaningful spatial relationships.\n+ Importantly, **combining all three modalities does not introduce negative interference.**\nThese results underscore the value of integrated reasoning across modalities for robust spatial understanding.\n\n\n**4. On the Discrepancy Between Table 5 and Table 6 (Audio vs SD).**\n\nWe are glad to explain the interpretation of Table 5 and Table 6. **The results in Table 5 and Table 6 are not contradictory—they reflect evaluations under different settings**:\n+ Table 5 evaluates direction and distance estimation accuracy for egocentric tracks **without global trajectory mapping**.\n\n+ Table 6 shows results **after global mapping and trajectory integration**, where audio cues that are denser and more continuous can be accumulated and refined over time, leading to stronger performance than SD after integration.\n\nBoth SD and Audio localization accuracy improve with global mapping and trajectory integration. **This highlights the value of global mapping, which aligns egocentric tracks into a shared 3D coordinate frame, which enables the fusion of observations into reliable dynamic object tracks and stable spatial references** (see detailed discussion in lines 277–297 of the supplementary PDF)."}, {"review_id": "SVN4pYDSRX", "reviewer": "Reviewer_vh2k", "summary": "This paper introduced a benchmark called SAVVY-bench that evaluate the dynamic audio-visual reasoning capabilities of models. They also introduced a training-free pipeline that augments existing AV-LLMs to achieve state-of-the-art performance on SAVVY-bench.", "questions": "1. what is the bottleneck for models on this benchmark at this point?", "limitations": "no", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. audio-visual spatial reasoning through time has been a missing piece in LLM evaluation, and this paper does exactly that. The task construction is based on real world dataset rather than simulated environment, which make it more reliable. \n2. The construction of the data is rigorously documented in main paper and supp. It's an impressive amount of work.\n3. The proposed pipeline improve baseline Gemini 2.5 pro significantly without any training\n\nWeakness:\n1. Not really a weakness, but the approach SAVVY basically shows that compared to using LLM E2E, we get better results if we applied traditional methods in segmentation, mapping and localization to extract information (and using LLM also just as a feature extractor). It's hard to pin point the novelty in the approach, as it's a combination of existing methods.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for detailed and helpful feedback and we appreciate that the reviewer acknowledges the uniqueness of our proposed task, and the real-world aspect of the constructed benchmark, and the effectiveness of the proposed reasoning pipeline. We address the questions and concerns below.\n\n**1. Novelty Clarifications.**\n\nWe claim our novelty in the following two aspects on benchmark and reasoning framework:\n+ **First-of-its-kind Benchmark:** SAVVY introduces a novel benchmark, pipeline, and reasoning structure grounded in multi-modal alignment in the 3D space. SAVVY-Bench is the first benchmark specifically designed for 3D spatial reasoning with synchronized spatial audio. Unlike existing AV benchmarks that focus on 2D video understanding or simple audio-visual correspondence, we introduce temporally-grounded, spatially-aware questions that require true 3D understanding. \n\n+ **Novel Reasoning Framework**: While SAVVY combines existing tools, its core contribution lies in how these components are orchestrated to enhance spatial reasoning in AV-LLMs. \n\nSAVVY motivates and formalizes the integration of structured egocentric tracks (via the Snapshot Descriptor), spatial audio cues, and visual cues  into a shared global 3D map. This design allows us to study each modality’s contribution for the audio-visual spatial reasoning tasks.\n\n**Importantly, SAVVY not only improves performance over E2E LLM baselines, but also opens up new directions: building multi-step reasoning chains, distilling knowledge back into AV-LLMs, and eventually enabling end-to-end systems that reason more effectively, without relying on heavy modular components. Thus, the novelty lies in framing the right questions, designing a principled pipeline, and enabling new research paths in audio-visual spatial reasoning.**\n\n**2. Bottlenecks for Models on SAVVY-Bench.**\n\nWe identify the following bottlenecks in current AV-LLMs on our benchmark:\n+ **Limited Spatial Audio Understanding:** All models take mono audio as input and overlook the spatial audio cues for spatial audio reasoning tasks. \n+ **Over-Reliance on Visual Cues:** Models often depend heavily on visual input, which becomes unreliable in out-of-view or occluded scenarios. As illustrated in Figure 4 (top) of Section 4.2, when a sound source disappears from view, models extrapolate from its last seen location, leading to incorrect spatial reasoning.\n\n**For open-source 7B models specifically, Table 3 shows two major bottlenecks:**\n+ **Temporal Grounding:** Models achieve under 5% t-mIoU, reflecting poor event-time alignment. Synchronizing complex audio events like speech remains a challenge at this scale.\n\n+ **Object Referral Accuracy:** For spatial reasoning, models must identify and describe relevant objects to establish spatial frames. However, they reach only ~33% accuracy, revealing limitations in visual-language grounding under spatial context.\n\nLooking ahead, we envision next-generation wearable devices with enhanced audio-visual capabilities that will make LLM-based spatial reasoning ubiquitous in daily life. By establishing SAVVY-Bench now, we’re laying the groundwork for this future – creating a bridge between high-quality multi-modal data collection and practical AI applications. Our goal is to catalyze a complete ecosystem where advances in data, models, and applications reinforce each other, ultimately transforming how humans interact with and understand their spatial environment."}]

Author rebuttal:

Meta-review: (a) The authors' claims include -- spatialization task via localization cues as a candidate for inducing reasoning in LLMs, using perception from localization as a way to training AV models, connecting training free optimization in LLMs to spatial audio tasks (first of their kind), novel AV model that can handle dynamics in scene via prompting. All these are well validated claims and the paper's novelty and evaluations justify the claims. 

(b) There are a number of important things the work is highlighting 
- spatialization is implicit in LLMs if we can probe/create a prompt that is 'detailed' enough abut context 
- introducing a new benchmark for 3d spatial reasoning in dynamics -- this is a very hard problem, relatively less is known on it 
- training-free methods and DSP/perception driven tuning helps scale and adopt the work into other areas in future 
- generally rigorous evaluations 

(c) Given the new-ness of this work / area, some of these are not really weakness, rather potential directions that the field/others can take in future : extension to more complicated tasks beyond simple navigation and localization tasks (e.g., what happens next, can you chart a path for me to take etc.); generalization of the proposal model for large-scale training of AV-LLMs i.e., connecting the dots to training driven reasoning models from this training-free setup; directly inducing physics information into the optimization constraints for DPO (again moving into training-dependent setups) 

(d) The authors tackle a difficult reasoning problem (spatialization in context of LLMs), bridge human perception style thinking into machine perception (navigation driven tasks and localization cues into prompting), build a training free method (scales well, modular) and rigorously evaluate the proposal; and the paper is written well. This work will influence future works on perception driven reasoning in language models. It ticks all the bars for an oral. 

(e) All reviewers agree on the strengths and impact of the work. Most of the discussion was about clarification of evaluations, modeling choice, simplicity of task definition, generalization to other tasks etc. These were addressed.
Decision: Accept (oral)