Title: Hear you are: Teaching LLMs Spatial Reasoning with Vision and Spatial Sound
Venue: NeurIPS 2025
Accepted: no
Keywords: Audio-visual Spatial Reasoning

Spatial audio

Multi-modal LLMs
Abstract: Many audio-visual learning methods have focused on aligning audio and visual information, either through semantic or temporal correspondence. However, most of these works have utilized monaural audio, which does not contain information about the spatial location of the sound source. In contrast, humans and other animals utilize binaural hearing to perceive this spatial information. Combining spatial sound and visual perception enables powerful high-level reasoning: for example, a person looking for their phone may hear the ringing sound coming from a backpack sitting on a table, and quickly infer that the missing phone is inside the backpack. In this paper, we investigate the problem of Audio-Visual Spatial Reasoning. We design a spatial audio-visual question answering dataset to cover scenarios where semantic correspondence between audio and visual signals is absent but spatial alignment exists, as well as cases with multiple audio-visual semantic correspondences that require spatial reasoning to disambiguate. We propose a model that learns spatial comprehension across the audio and vision modalities by connecting them with a large language model and experimentally demonstrate that spatial sound perception is an essential part of our task.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Audio-Visual Spatial Reasoning, a task requiring models to infer spatial relationships between binaural audio and panoramic images. To support it, the authors build Hear You Are QA, a large-scale simulated dataset of 1 M question–answer pairs covering spatial correspondence, relative location, semantic alignment, and co-occurrence scenarios. They propose Hear You Are LLM, which encodes binaural audio and 360° vision via pretrained encoders, projects them into a shared token space, and feeds them (with the question) into a frozen LLM fine-tuned end-to-end. Experiments show that only the full model (vision + binaural + text) consistently solves tasks requiring spatial cues, outperforming monaural or vision-only baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Constructs a diverse QA dataset using SoundSpaces 2.0, VGGSound clips, and automatically generated 3D objects, enabling controlled evaluations across nine question types.\n2. Compares against strong baselines: two audio-visual localization methods and a monaural-audio VideoLLaMA2 adaptation.\n3. Ablation studies systematically vary modalities (R+B+Q, R+M+Q, B+Q, etc.) and visible vs. invisible sources.\n\nWeaknesses:\n1. All experiments use simulated scenes. It remains unclear how models transfer to real binaural recordings with natural reverberation and visual complexity. I think the paper would benefit from recording a small real-world test set (e.g., 360° video + binaural audio in indoor rooms) and reporting zero-shot performance or lightweight fine-tuning results to demonstrate generalizability.\n2. Real scenes often have simultaneous sounds, but each scene of this dataset contains a single sound source, which simplifies the task. Please consider including experiments with two concurrent sound sources (e.g., speech + music) to test the model’s capacity for multi-source spatial reasoning or clarify why single-source suffices for the proposed task.\n3. Although the paper mentions inserting distractor objects to mask seams, it does not quantify whether visual \"stitching\" artifacts could leak location. Could the authors analyze or ablate reliance on low-level cues (e.g., by blurring object boundaries) to ensure the model truly uses spatial audio rather than visual shortcuts?\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a dataset of spatial audio-visual question answering, which targets to evaluate the model capability of reasoning about sound origins (particularly the location) given the accompanying visual perception. The authors also propose a promising model to address this task, which generally outperforms the baselines. Additional ablation studies reveal the nature of each type of question.\n\nSTRENGTHS AND WEAKNESSES:\nIn what follows, I will use S, W, C, and Q to denote Strengths, Weaknesses, Comments, and Questions, respectively, for ease of reference in the discussion.\n\nS1. The task is highly interesting and important in embodied AI.\n\nS2. The presented model is fundamentally simple, and the authors have included a comprehensive analysis through ablation studies for all covered categories.\n\nW1. I am concerned about the QA setup. While it provides a natural language interface to understand the model performance, the current QA setup has too many confounding factors---for example, a poor performance could be because of (1) the model's inability to understand the question, (2) the model's inability to understand the visual input, (3) the model's inability to understand the audio input, (4) the model's inability to reason about the relationship between the audio and visual inputs, or (5) the model's inability to translate the reasoning into a natural language answer, to name a few inexhaustive potential reasons. I don't see a reason why not release the standard annotations and make the QA interface a higher-level interface on top of the standard annotations, which would make the evaluation more interpretable and easier to understand, as well as enabling more systematic analyses of the model performance.\n\nI understand that the authors may be affected much by the \"standard\" approaches of the current community in building benchmarks, among which many simply release a bunch of QA pairs for anything, but I would strongly encourage the authors to consider more principled approaches when conducting scientific research.\n\nThere could also have been much more interesting ablation studies and analyses with more structured annotations, for example, how does slightly moving an object (represented by different input scenarios) affect the model reasoning? Is there an internal neuron that accounts for a specific type of task?\n\nWhile I'm not suggesting the authors conduct all the suggested experiments above in one paper, I feel the impact of the current data format will be highly limited, and I suggest the authors consider more controlled environments that enable systematic analyses of the model performance. A good example for what to consider and what to release in a dataset, though a bit out of domain, is https://openreview.net/forum?id=84pDoCD4lH\n\nW2. There is an important missing reference: when it comes to sound localization, this paper https://openaccess.thecvf.com/content_ECCV_2018/papers/Relja_Arandjelovic_Objects_that_Sound_ECCV_2018_paper.pdf comes to my mind but is unfortunately not cited in this paper. I would suggest the authors do a more comprehensive literature review.\n\nW3. While I understand this is unfortunately not a standard process in AI/ML, the paper could benefit a lot from reporting the standard errors in the figures and tables---for example, it wouldn't be hard to add a standard error to the results in Tables 2 and 3, and this would add another theoretical layer of confidence to the findings.\n\nW4. There are a few technical designs that I would suggest the authors consider revising:\n\n1. The visual encoder is SigLIP, a CLIP-style image and text matching model, which may not sufficiently elicit the visual features that are particularly important for this task. I would suggest the authors consider using Dino as an additional source of visual features.\n\nC1. This doesn't affect my judgment on the technical contribution of the paper, but I strongly object to having a comic teaser in a scientific paper. It would be good to replace Figure 1 with a more scientifically rigorous figure that better illustrates the point, and it would be a good usage of the limited spaces of the main content. Such content should better go to social media or other channels for dissemination, not in the main paper nor an academic poster presentation.\n\nC2. I suggest some more direct illustrations through figures, instead of a huge table for the ablation study in Table 3. When using tables, it's also good to align the decimal points for better readability---the current presentation is quite hard to digest.\n\nQUESTIONS:\nQ1. I didn't see a discussion that the visual and audio perception may have position mismatch because of the physical volume of the sensors, e.g., in many robots, the audio sensor is usually below the visual one. Have you taken this into consideration in design?\n\nIf possible, it would also be good to include the ground-truth positions of these sensors in the dataset.\n\nQ2. Did you also finetune the baseline models in Table 2?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new task audio-visual spatial reasoning, which requires the language model to perform both spatial localization and semantic matching for audio-visual scene. To address this, the authors introduce a new large-scale simulated dataset, consisting of 1 million qa pairs for understanding complex spatial relationships between sound and visual context in 3D environments. The authors also propose a multi-modal llm to integrate spatial audio and visual encoders with a language backbone to perform these tasks. Empirical evaluations demonstrate that the approach outperforms strong audio-visual baselines, supporting the core claims that incorporating binaural audio enables richer spatial reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The proposed “Hear You Are QA” dataset covers a wide range of spatial reasoning problems, including alignment/disambiguation tasks that prior datasets and benchmarks did not address. The question types provided in tab1 give a clear taxonomy, motivating why audio-visual spatial reasoning matters.\n\n2. The authors also propose a model, adopting modality-specific encoders and projectors to transform vision and audio data into representations to be fed to the large language model. The model is capable of doing spatial audio-visual understanding, which is new.\nSince there isn’t spatial audio-visual llm baselines to be compared with, the authors made great efforts to adapt existing non-llm spatial audio-visual methods and videollama2 for comparison.\n\n\nWeaknesses: \n\n1. The model relies on a Q-Former-style projector to compress vision and audio features before passing them to the LLM. This type of compression often loses information, which is why many recent works prefer the simpler LLaVA-style MLP projector. Why did the authors keep the Q-Former method rather than switching to a lightweight MLP? A clearer justification is needed. \n\n2. The whole dataset is rendered with SoundSpaces. Both the images, using meshes, and the binaural audio, using a fixed HRTF, are synthetic. How will the system transfer to real-world RGB frames and real binaural recordings? The paper didn’t show real-world test results, so the generalization ability is still unknown.\n\n3. The model predicts very precise coordinates such as (−30, −12), 2.549m from binaural audio only. In practice, such accuracy usually depends on the carefully designed microphone array and actual scene geometry visible in the image. Are these outputs absolute or relative to the rendered environment? I would like the authors to clarify how this would work on real data.\n\n4. There are multimodal LLMs already able to reason about spatial audio, e.g., Gemini 2.5-Flash/Pro. Qwen-Omni is also known to handle audio-visual inputs and should outperform VideoLLaMA-2. The authors should run or at least discuss these stronger baselines.\n\nQUESTIONS:\nWhen preparing the rebuttal, I would suggest that the authors focus on the main weaknesses as outlined above. To summarize:\n\n1. Real-World Gap: can the authors comment more concretely on how they foresee bridging the simulation-to-real gap?\n2. More comparison methods: including gemini and qwen-omni, at least on part of the dataset.\n3. Clarify how the model produces such fine spatial coordinates from binaural audio alone.\n\nLIMITATIONS:\nYes, the authors adequately addressed the limitations and potential negative impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors address the problem of imparting spatial comprehension of relative locations in space of sound sources using vision and binaural audio. The authors propose a new entirely synthetic dataset constructed using SoundScape 2.0, for generating audio and Matterport3D + Stable diffusion for creating the visual scene with the corresponding visual objects representing source of sound or distractors. They use this dataset to train their model - \"Hear you are\" LLM which essentially comprises of 1. Modality specific encoders:  visual encoder to tokenize the panoramic view of the visual scene and a binaural audio encoder to encode the spatial audio and a text encoder. 2. Modality specific Q-Formers 3. An LLM with LoRA adapters. As this is a novel task, there are no existing benchmarks and baseline models. So the authors resort to creating some interesting baselines: 2 models based on audio-Visual sound source localization and another baseline with the exact configuration as the proposed approach but with mono audio (lacking spatial information of sound sources). Since the dataset is synthesized it allows for crafting Questions and their respective Answer. Authors construct 9 categories of questions to span questions probing the model about relative distances, semantic categorization of sound sources, disambiguating sounds from plausible sources in the visual scene, and localizing source when it is visible and not visible in the visual scene.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. The paper is very well written, easy to understand and widely accessible. \n2. The proposed task and dataset are relevant and the evaluations are very interesting. Clearly shows the benefit of video and binaural audio. This is a good initial work in this area. \n\nWeakness:\n1. Entire dataset is synthetic and the real challenge is in making this work in the real world. It would be good to supplement a large syntetic data with some real data to assess feasibility of generalization. However this in itself would be a pretty significant task to design real world data collection and having the synthetic environments mimic that.\n\nQUESTIONS:\nSee the Strengths and Weakness section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}