STAR-Bench: Probing Deep Spatio-Temporal Reasoning as Audio 4D Intelligence

## Abstract

Abstract Despite rapid progress in Multi-modal Large Language Models and Large Audio-Language Models, existing audio benchmarks largely test semantics that can be recovered from text captions, masking deficits in fine-grained perceptual reasoning.
We formalize audio 4D intelligence that is defined as reasoning over sound dynamics in time and 3D space, and introduce STAR-Bench to measure it.
STAR-Bench combines a Foundational Acoustic Perception setting (six attributes under absolute and relative regimes) with a Holistic Spatio-Temporal Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories.
Our data curation pipeline uses two methods to ensure high-quality samples.
For foundational tasks, we use procedurally synthesized and physics-simulated audio.
For holistic data, we follow a four-stage process that includes human annotation and final selection based on human performance.
Unlike prior benchmarks where caption-only answering reduces accuracy slightly, STAR-Bench induces far larger drops (-31.5% temporal, -35.2% spatial), evidencing its focus on linguistically hard-to-describe cues.
Evaluating 19 models reveals substantial gaps compared with humans and a capability hierarchy: closed-source models are bottlenecked by fine-grained perception, while open-source models lag across perception, knowledge, and reasoning.
Our STAR-Bench provides critical insights and a clear path forward for developing future models with a more robust understanding of the physical world.

## 1 Introduction

As a fundamental modality of human perception, audio serves a pivotal role in communication, aesthetic appreciation, and situational awareness, complementing the limitations of visual perception.
With the rise of Multimodal Large Language Models (MLLMs) (Comanici et al., 2025; Achiam et al., 2023) and especially Large Audio-Language Models (LALMs) (Chu et al., 2024; Goel et al., 2025), these models have shown impressive capabilities in understanding audio, representing a crucial step toward diverse applications such as embodied intelligence (Paul et al., 2022).

To drive progress, a series of audio benchmarks has been introduced (Yang et al., 2024; Sakshi et al., 2025), covering traditional tasks like Automatic Speech Recognition (ASR) and sound event classification.
While some recent efforts are beginning to emphasize reasoning abilities (Ma et al., 2025; Kumar et al., 2025), we observe that existing benchmarks predominantly focus on coarse-grained semantic content, which is audio information that can be distilled into textual descriptions with minimal loss.
As shown in the left part of [Fig. 1](https://arxiv.org/html/2510.24693v1#S1.F1), we first use Gemini 2.5 Pro (Comanici et al., 2025) to generate detailed audio captions for samples in recent representative audio benchmarks MMAU (test-mini) (Sakshi et al., 2025) and MMAR (Ma et al., 2025).
We then prompt the model to answer questions based only on these audio captions, and its performance drops by only 5.9% and 9.0%, respectively, compared to when it processes the raw audio.
This result suggests that existing benchmarks primarily evaluate audio information that is easily representable by text.
However, human auditory intelligence is not limited to this coarse-grained understanding.
For example, humans can intuitively judge the water level in a container from the dynamic changes in the pouring sound, even without being able to precisely articulate the underlying acoustic features.
Similarly, we can infer the trajectory and distance of a vehicle approaching from behind to ensure our safety.
These abilities are rooted in deep reasoning of audio cues that are difficult to represent linguistically.

To capture this human-like audio competence, we propose a new paradigm, called audio 4D intelligence.
This is defined as the ability to perform deep reasoning over the dynamics of sound sources in time (1D) and three-dimensional space (3D), grounded in an understanding of the physical world.
Mastering 4D audio intelligence is crucial for various applications.
In embodied AI and robotics, for instance, agents must integrate fine-grained auditory cues to interact naturally with their surroundings, such as using sound to infer the trajectory of an object or to monitor the subtle operations of a machine.
To systematically evaluate this paradigm and bridge the gap between current audio benchmarks and real-world auditory intelligence, we introduce the Spatio-Temporal Audio Reasoning (STAR-Bench) benchmark.

STAR-Bench is designed through a hierarchical task structure with two levels.
At the Foundational Acoustic Perception level, we conduct a fine-grained, quantitative evaluation of six core audio attributes (pitch, loudness, duration, azimuth, elevation, distance) across both absolute perception ranges and relative discrimination sensitivity.
We also introduce a Holistic Spatio-Temporal Reasoning level that evaluates an audio model’s ability to infer both event order and 3D scene structure.
Temporal reasoning is tested via segment reordering that spans continuous processes and discrete event scripts, while spatial reasoning covers static localization, multi-source relations, and dynamic trajectory tracking.
As shown in the right part of [Fig. 1](https://arxiv.org/html/2510.24693v1#S1.F1), every question in our holistic tasks is designed to probe a synthesis of three core pillars, such as multi-step reasoning.
A failure in any one of these pillars will lead to an incorrect response.
Our data curation pipeline couples procedurally synthesized, fully parameterized audio for foundational perception with large-scale real-world corpora for holistic reasoning.
For the latter, we use a four-stage process including human annotation and final selection by human performance to ensure the high quality of our benchmark samples.

Figure: Figure 1: (Left): A comparison between humans and the Gemini 2.5 Pro with and without audio captions on various audio benchmarks. Our STAR-Bench evaluates linguistically hard-to-describe audio cues. See [Sec. B.1](https://arxiv.org/html/2510.24693v1#A2.SS1) for audio caption details. (Right): The three core abilities required to solve tasks in the STAR-Bench benchmark.
Refer to caption: x1.png

Our comprehensive evaluation of 19 models (16 open-source and 3 closed-source) reveals a clear capability hierarchy between the two groups.
Leading closed-source models like Gemini 2.5 Pro excel in knowledge and reasoning, shifting their primary bottleneck to the more difficult challenge of fine-grained perception. In contrast, open-source models exhibit fundamental weaknesses across all three core capabilities. Through our detailed error analysis and ablation studies, we highlight several key insights for the future development of open-source audio models:
1) Enhancing dense audio captioning. Open-source models struggle to produce dense, fine-grained captions, which limits their perceptual sensitivity and ability to extract embedded knowledge. Bridging this gap is a crucial first step.
2) Improving multi-audio reasoning. Open-source models lag significantly in comparing, integrating, and grounding information across multiple audio clips.
3) Moving beyond channel-averaged audio preprocessing. The common practice of averaging multi-channel audio into a mono signal is a major bottleneck for spatial reasoning. Developing architectures that natively process multi-channel cues is essential for unlocking genuine spatial awareness.

Our contributions are summarized as: (1) We formalize audio 4D intelligence, and empirically show that prior benchmarks largely probe text-representable semantics, motivating a shift toward fine-grained, non-linguistic auditory cues.
(2) We introduce the STAR-Bench with foundational acoustic perception and holistic spatio-temporal reasoning tasks, together with a rigorous curation pipeline with expert validation.
(3) We provide a comprehensive evaluation of 19 LALMs/OLMs. Our analyses and standardized protocols establish strong baselines and testbeds for future research.

## 2 Related Work

The recent progress of Large Audio-Language Models (LALMs)(Kong et al., 2024; Chu et al., 2024; Wu et al., 2025; Xiaomi, 2025) and Omni-Language Models (OLMs)(Xu et al., 2025; Yao et al., 2024; AI et al., 2025) has significantly advanced audio understanding. At the same time, it has spurred the development of numerous benchmarks to comprehensively evaluate their capabilities. Earlier benchmarks(Wang et al., 2024; Yang et al., 2024) mainly focused on semantic-level understanding tasks (transcription, captioning, and simple question answering), and recent benchmarks(Sakshi et al., 2025; Ma et al., 2025; Kumar et al., 2025) have begun to investigate logical audio reasoning tasks. However, existing benchmarks do not address 4D audio intelligence or deep spatio-temporal reasoning across multiple audio inputs, and instead remain limited to single-clip understanding and reasoning. To fill these gaps, we propose a benchmark designed for multi-audio and deep spatio-temporal reasoning, enabling more comprehensive evaluation of audio 4D intelligence. See Table [1](https://arxiv.org/html/2510.24693v1#S2.T1) for a comparison with existing benchmarks, and [Appendix A](https://arxiv.org/html/2510.24693v1#A1) for further related works.

**Table 1: A comparative overview of our benchmark against other representative audio benchmarks. (✓: Fully supported, ◐: Partially supported, ✗: Not supported)**
| Benchmark | Temporal<br>Deep Reasoning | Spatial<br>Deep Reasoning | Quantitative<br>Attribute<br>Evaluation | Robust<br>Evaluation | Multi-<br>Audio | Fully<br>Human-<br>Annotated | Fully<br>Expert<br>Verified |
| --- | --- | --- | --- | --- | --- | --- | --- |
| AIR-Bench [45] | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ | ✗ |
| MMAU [30] | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
| Dynamic-SUPERB Phase-2 [16] | ✗ | ✗ | ✗ | ✗ | ◐ | ◐ | ✗ |
| MMAR [27] | ✗ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ |
| MMAU-Pro [20] | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ |
| STAR-Bench (ours) | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

## 3 STAR-Bench

Understanding dynamic sound sources in both time (1D) and three-dimensional space (3D) is a crucial skill for MLLMs to comprehend the physical world.
To address this need, our benchmark, STAR-Bench, is designed to comprehensively evaluate this 4D intelligence in the audio domain.
As illustrated in [Fig. 2](https://arxiv.org/html/2510.24693v1#S3.F2), our evaluation has two complementary sub-tasks: (1) Foundational Acoustic Perception ([Sec. 3.1](https://arxiv.org/html/2510.24693v1#S3.SS1)), which uses procedurally synthesized audio to quantitatively profile a model’s basic perceptual abilities under controlled conditions, and (2) Holistic Spatio-Temporal Reasoning ([Sec. 3.2](https://arxiv.org/html/2510.24693v1#S3.SS2)), which uses real-world audio to evaluate more complex reasoning in dynamic and authentic scenarios.
We also elaborate our data curation pipeline in the [Sec. 3.3](https://arxiv.org/html/2510.24693v1#S3.SS3).

Figure: Figure 2: Data examples from STAR-Bench: (1) the foundational perception task (upper) and (2) the holistic spatio-temporal reasoning task, which includes both temporal reasoning (bottom left) and spatial reasoning (bottom right). Zoom in for the best view.
Refer to caption: x2.png

### 3.1 Foundational Acoustic Perception

The Foundational Acoustic Perception task is motivated by the need for a robust, quantitative evaluation of the core perceptual abilities that underpin 4D audio intelligence.
A model’s capacity for complex reasoning about dynamic audio scenes in the physical world is directly dependent on its ability to accurately perceive fundamental acoustic properties.
Our foundational acoustic perception task systematically probes a model’s understanding of three critical auditory attributes: Loudness, Pitch, Duration, and the three spatial dimensions: Azimuth, Elevation, and Distance.
Just as a solid understanding of grammar is required for writing a complex narrative, a model must be able to accurately perceive these core attributes before it can reason about the dynamic, spatial relationships of sound sources in the physical world.
Without a firm grasp of these foundational elements, a model cannot accurately interpret complex, real-world acoustic scenes, which require understanding how sounds change over time and move through space.

We employ a targeted synthesis strategy to generate precise evaluation samples in a controlled environment for the foundational perception task.
For non-spatial attributes (Loudness, Pitch, Duration), we synthesize pure sine waves by directly specifying their parameters.
For spatial attributes (Azimuth, Elevation, Distance), we use the Pyroomacoustics (Scheibler et al., 2018) physics-based simulation engine to render acoustic scenes.
The targeted synthesis strategy allows us to investigate a model’s audio perceptual abilities under the following two sub-tasks:

1) Absolute Perception Range, which defines the sensory limits of MLLMs for acoustic attributes.
For pitch and loudness, we adapt the design of human audiometry tests to create an “audiogram” for the MLLMs. Specifically, we synthesize sine waves with frequencies ranging from $125$ Hz to $8000$ Hz and loudness levels from $-10$ to $110$ dB HL and require the model to identify if a clear beep is in the first or second part of an audio clip, or if it’s not there at all.
For spatial attributes, we design interval localization tasks that require the model to identify a sound’s azimuth within one of four 90° quadrants (from 0° to 360°), its elevation relative to ear-level (above, at, or below, from -90° to 90°), and its distance category (near, medium, or far, within a 0 - 10m range). [Tab. 3](https://arxiv.org/html/2510.24693v1#A2.T3) presents detailed examples of these absolute perception range tasks. Through these precise tasks, we establish the absolute limits of what the model can hear, which is crucial for developing AI systems that can safely and effectively interact with the physical world.

2) Relative Discrimination Sensitivity, which investigates how well a model can detect small changes in acoustic attributes.
The ability to detect small changes allows a model to make nuanced judgments, like determining if a sound is getting louder or a pitch is rising.
Analogous to measuring the human Just Noticeable Difference (JND), the relative discrimination task presents the model with an audio clip containing two sounds and requires it to compare them based on a specific attribute.
We meticulously designed four to six distinct difficulty levels for each of the six attributes, as detailed in [Tab. 3](https://arxiv.org/html/2510.24693v1#A2.T3).
Level 1 serves as a control group to test for random guessing, presenting identical sounds ($\Delta$=0) for non-spatial attributes and a sub-threshold difference for spatial ones. Subsequent levels then introduce progressively larger differences, ranging from subtle variations perceptible to humans to more significant, real-world changes.
By analyzing the model’s performance across these different levels of stimulus differences, we can quantitatively assess its discrimination sensitivity for each attribute.

### 3.2 Holistic Spatio-Temporal Reasoning

Building on the model’s fundamental audio perceptual abilities ([Sec. 3.1](https://arxiv.org/html/2510.24693v1#S3.SS1)), we further introduce holistic temporal reasoning ([Sec. 3.2.1](https://arxiv.org/html/2510.24693v1#S3.SS2.SSS1)) and spatial reasoning ([Sec. 3.2.2](https://arxiv.org/html/2510.24693v1#S3.SS2.SSS2)), which are designed to systematically evaluate a model’s reasoning ability that is required for audio 4D intelligence.

#### 3.2.1 Temporal Reasoning Tasks

The core of temporal reasoning lies in understanding the intrinsic logic of event sequences, encompassing physical causality, functional procedures, or social conventions.
To evaluate this capability, we design a novel Audio Segment Reordering setting.
Specifically, we curate a collection of audio events characterized by strong sequential uniqueness, semantic clarity, and logical universality.
Each event is segmented into three clips, which are then shuffled as inputs to the model.
The models are required to restore the original temporal sequence based solely on the audio content.
Our temporal reasoning tasks are organized into two meta-categories (continuous processes, discrete event sequences) and five subcategories based on their core logical principles.

The continuous processes assess a model’s ability to track the subtle, continuous evolution of acoustic features within a single, uninterrupted acoustic event.
The object spatial motion subcategory reconstructs the spatio-temporal trajectory of moving sources (e.g., passing cars, airplanes) by interpreting key acoustic cues, such as the Doppler effect (frequency shifts indicating relative velocity) and the inverse-square law (loudness changes indicating distance).
Besides, the in-situ state evolution subcategory assesses a model’s ability to track the intrinsic evolution of a stationary object’s state, a process governed by predictable trend patterns.
These trend patterns arise from various underlying principles, including: Fluid & Pneumatic Dynamics, where the sound is governed by principles of turbulence, resonance, and pressure changes (e.g., a toilet flushing, water being poured); Thermodynamic Processes, involving irreversible state changes driven by heat (e.g., water boiling, food frying); Energy Decay, a process governed by resonant decay and frictional damping after a single excitation (e.g., a bell’s chime, an explosion’s echo); and complex Biological Rhythms that reflect an evolving physiological or emotional state.

The discrete event sequences category requires the model to understand the logical and temporal relationships between multiple, distinct acoustic events, which are governed by function, convention, or causality.
The tool & appliance operation sub-category follows the standardized operating procedure for tools and appliances (e.g., a microwave, a power drill), where the sequence is correct when it follows the tool’s designed function.
The daily scene scripts sub-category applies commonsense and contextual script knowledge to follow the conventional sequence of actions in a daily activity (e.g., brushing teeth, drinking water).
The event-triggered consequences sub-category applies causal reasoning to infer that a trigger event (e.g., a firework explosion) will be followed by an automatic and irreversible outcome, whether physical (glass shattering) or social (a crowd cheering).

#### 3.2.2 Spatial Reasoning Tasks

Humans effortlessly perceive complex 3D auditory scenes (e.g., hearing a voice from behind, following an approaching car, or locating multiple speakers). Such an ability is fundamental for egocentric interaction and embodied AI systems, for instance, robots that navigate and interact with their surroundings. However, existing benchmarks focus primarily on the localization of static sound sources, whereas real-world scenarios demand reasoning that integrates both spatial and temporal cues. To address this gap, we organize the spatial reasoning task into three subcategories.

The single-source static localization evaluates the model’s ability to identify the direction of a target sound source among multiple static sources (e.g., judging whether a sound comes from the left or right). It assesses the basic spatial perception capability of the model and provides the foundation for more advanced reasoning.
The multi-source spatial relation requires the model to determine the relative spatial relationships among multiple simultaneous sound sources (e.g., comparing the placement of two speakers to decide which one is further to the right). Beyond localizing each source individually, the model must infer their spatial placement and choose the appropriate relational description from multiple candidates.
The dynamic trajectory tracking introduces moving sound sources, which require the model to go beyond basic spatial perception to dynamically model spatio-temporal relations for reasoning about complex movement trajectories (e.g., tracking a passing car moving from left to right). This task extends spatial reasoning into the temporal domain and is more faithful to the complexity of real-world acoustic scenarios.

Figure: Figure 3: Audio preprocessing in existing models results in the loss of dual-channel information.
Refer to caption: x3.png

However, evaluating existing LALMs on multi-channel spatial tasks is challenging. The common practice of these models is to average multi-channel audio into a mono signal, resulting in the loss of substantial spatial information.
We conduct a simple experiment as shown in [Fig. 3](https://arxiv.org/html/2510.24693v1#S3.F3).
We construct 20 pseudo-stereo signals by assigning the original audio to the left channel and its additive inverse to the right.
While human listeners could easily perform sound event classification on these signals, the models consistently failed due to signal cancellation during the mono conversion.
The result confirms their lack of explicit support for genuine stereo audio processing.
To provide a comprehensive assessment, we design two complementary strategies: native input, where the model directly processes stereo audio, follow their default processing pipeline to probe its intrinsic ability to exploit spatial cues; and the channel-wise input, where each channel is presented separately with explicit textual instructions, as shown in the bottom right of [Fig. 2](https://arxiv.org/html/2510.24693v1#S3.F2), and allow the model to approximate human-like use of interaural cues.

Figure: (a) Data distribution across the foundation perception, temporal reasoning, and spatial reasoning three tasks.
Refer to caption: x4.png

### 3.3 Data Curation Pipeline

Our data curation pipeline integrates procedural synthesis with real-world data collection to ensure both comprehensive coverage and ecological validity.
[Fig. 4](https://arxiv.org/html/2510.24693v1#S3.F4) shows the distribution and statistics of our STAR-Bench.
All audio for the foundational perception task is synthesized using precise parameterization or the Pyroomacoustics (Scheibler et al., 2018) physics-based simulator, providing complete control over acoustic parameters.
Domain experts rigorously validate the task difficulty levels, which are then calibrated through human testing.
For the holistic spatio-temporal reasoning task, the curation process comprises four key stages (see [Fig. 5](https://arxiv.org/html/2510.24693v1#S3.F5)):

1) Taxonomy Construction and Data Sourcing: We build a hierarchical task taxonomy through a collaborative process involving domain experts and the Gemini 2.5 Pro (Comanici et al., 2025).
This framework guides the sourcing of candidate data from large-scale, real-world audio libraries: Clotho (Drossos et al., 2019) and FSD50K (Fonseca et al., 2022) for temporal reasoning, and STARSS23 (Shimada et al., 2023), along with audio sourced from the internet for spatial reasoning.

Figure: Figure 5: The four-stage data annotation pipeline for constructing our STAR-Bench.
Refer to caption: x5.png

2) AI-Assisted Automated Filtering: This process employs an efficient three-stage funnel. First, we discard unsuitable samples based on basic properties like duration and energy. Next, an LLM (e.g., DeepSeek-V3 (Liu et al., 2024a)) performs an initial screening based on textual metadata, providing justifications for its decisions. Finally, a powerful multimodal model (e.g., Gemini 2.5 Pro (Comanici et al., 2025)) analyzes the audio, metadata, and the LLM’s outputs.
The final step yields a judgment, a quality score, and a preliminary classification, further filtering irrelevant samples.
The detailed prompts used to query the LLMs are provided in [Sec. B.3](https://arxiv.org/html/2510.24693v1#A2.SS3).

3) Human Annotation and Quality Control: We recruit and train 10 undergraduate annotators to label the data using a professional platform.
During this process, AI-generated information is provided as an auxiliary reference. To ensure high-quality labels, we implement a stringent two-round review process: the first round involves inter-annotator cross-validation until a consensus is reached, while the second consists of random spot-checks by three domain experts.

4) Final Validation via Human Performance Evaluation: To ensure all items in the benchmark are fair, unambiguous, and solvable by humans, we implement a final validation stage. In this phase, domain experts act as examinees and solve our tasks. Only items that are independently and correctly solved by at least two-thirds of the experts are retained. Our rigorous protocol ensures that all problems in our benchmark are well-posed and reliably solvable by human experts.

## 4 Evaluation

Benchmarking Models.
Our evaluation covers 19 models (16 open-source and 3 closed-source models). The open-source models span three categories: (1) Large Audio Language Models designed for universal audio-text understanding, including SALMONN (Tang et al., 2024), Qwen2-Audio Instruct (Chu et al., 2024), Audio Flamingo 3 (Goel et al., 2025) with its ‘think’ variant, DeSTA2.5-Audio (Lu et al., 2025), Kimi-Audio (KimiTeam et al., 2025), Step-Audio-2-mini (Wu et al., 2025), MidashengLM (Dinkel et al., 2025), and Xiaomi-MiMo-Audio (Xiaomi, 2025) with its ‘think’ variant; (2) a specialized model for spatial audio, BAT (Zheng et al., 2024); and (3) Omni Language Models with fully multimodal support, including Qwen-2.5-Omni (Xu et al., 2025), Phi4-MM (Abouelenin et al., 2025), Gemma-3n-E4B-it (Team et al., 2025), and Ming-Lite-Omni-1.5 (AI et al., 2025).
We also include three leading closed-source models: Gemini 2.5 Pro (Comanici et al., 2025) (updated June 2025), Gemini 2.5 Flash (updated June 2025), and GPT-4o-audio-preview (Achiam et al., 2023) (version 2025-06-03).

Robust Evaluation.
All questions in STAR-Bench are presented as multiple-choice questions and evaluated using classification accuracy, with correctness determined via string matching of option labels or their full text. To ensure robustness, we evaluate each question multiple times under minor prompt perturbations, a strategy detailed in [Appendix C](https://arxiv.org/html/2510.24693v1#A3). This approach yields two key metrics: Average Accuracy (AA), the mean accuracy across all runs, and All-Correct Rate (ACR), the proportion of questions answered correctly in every run, which serves as a stronger indicator of model reliability. Due to space limitations, we primarily report AA in the main text, while complete experimental results are available in [Appendix D](https://arxiv.org/html/2510.24693v1#A4).

**Table 2: Evaluation results of various models on STAR-Bench. The best performance is highlighted in bold, and the second-best ones are underlined. MA (Macro Accuracy) denotes the unweighted mean of class-wise accuracies, while OA (Overall Accuracy) denotes the proportion of correctly answered instances. All reported values are AA (Average Accuracy across multiple runs) only; for ACR (All-Correct Rate), see [Appendix D](https://arxiv.org/html/2510.24693v1#A4).**
| Models | Size | Foundational Perception | Temporal Reasoning | Spatial Reasoning | MA (%) |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Range | Sensitivity | MA | Continuous | Discrete | OA | Localization | Relation | Trajectory | OA |  |  |  |
| Random Guess | - | 23.75 | 26.38 | 25.33 | 14.29 | 14.29 | 14.29 | 33.33 | 33.33 | 33.33 | 33.33 | 24.32 |
| Human | - | 79.42 | 74.55 | 75.60 | 90.12 | 85.51 | 88.00 | 70.00 | 80.00 | 77.00 | 73.72 | 79.11 |
| SALMONN [33] | 13B | 27.32 | 25.48 | 26.22 | 14.88 | 13.30 | 14.15 | 26.15 | 28.61 | 39.94 | 29.62 | 23.33 |
| Audio Flamingo 3 [14] | 8.4B | 31.79 | 35.72 | 34.15 | 9.23 | 8.01 | 8.67 | 37.22 | 38.35 | 44.03 | 38.91 | 27.24 |
| Audio Flamingo 3 think [14] | 8.4B | 25.54 | 34.08 | 30.66 | 13.22 | 14.02 | 13.59 | 35.45 | 37.46 | 38.05 | 36.45 | 26.90 |
| Qwen2-Audio-Instruct [7] | 8.4B | 29.88 | 26.47 | 27.84 | 13.29 | 12.10 | 12.74 | 21.32 | 24.78 | 15.09 | 20.78 | 20.45 |
| DeSTA2.5-Audio [26] | 8.8B | 29.87 | 19.79 | 23.82 | 16.53 | 17.39 | 16.93 | 23.67 | 34.81 | 37.74 | 29.15 | 23.30 |
| BAT [51] | 7B | 22.81 | 6.25 | 12.87 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 0.00 | 4.29 |
| Phi4-MM [1] | 5.5B | 19.14 | 29.85 | 25.56 | 16.74 | 16.99 | 16.85 | 33.10 | 27.14 | 34.28 | 32.01 | 24.81 |
| Kimi-Audio [18] | 7B | 23.29 | 27.50 | 25.82 | 19.97 | 16.83 | 18.52 | 27.56 | 38.94 | 44.03 | 33.60 | 25.98 |
| MiDashengLM [10] | 7B | 36.94 | 30.78 | 33.24 | 15.43 | 17.31 | 16.30 | 43.11 | 45.43 | 46.23 | 44.29 | 31.28 |
| Step-Audio-2-mini [39] | 7B | 29.65 | 27.14 | 28.14 | 15.36 | 15.87 | 15.59 | 33.33 | 31.27 | 37.74 | 33.80 | 25.84 |
| Gemma-3n-E4B-it [34] | 7.5B | 18.55 | 25.02 | 22.43 | 16.87 | 16.27 | 16.59 | 23.32 | 41.89 | 33.96 | 29.75 | 22.92 |
| Ming-Lite-Omni-1.5 [3] | 18.9B | 26.76 | 26.76 | 26.76 | 17.08 | 15.54 | 16.37 | 20.14 | 35.10 | 38.36 | 27.35 | 23.49 |
| Qwen-2.5-Omni [43] | 7B | 28.76 | 32.32 | 30.90 | 16.32 | 17.71 | 16.96 | 39.46 | 41.30 | 27.04 | 37.25 | 28.37 |
| Xiaomi-MiMo-Audio [40] | 7B | 34.95 | 31.59 | 32.93 | 18.18 | 19.15 | 18.63 | 36.16 | 41.30 | 45.28 | 39.24 | 30.27 |
| Xiaomi-MiMo-Audio-think [40] | 7B | 29.90 | 24.93 | 26.92 | 16.80 | 19.39 | 18.00 | 34.28 | 44.54 | 36.79 | 37.12 | 27.35 |
| MiniCPM-O-v2.6 [48] | 8B | 31.02 | 31.87 | 31.53 | 15.36 | 17.39 | 16.30 | 29.92 | 43.36 | 38.36 | 34.73 | 27.52 |
| GPT-4o Audio [2] | - | 27.58 | 34.55 | 31.76 | 15.91 | 23.56 | 19.44 | 41.81 | 43.97 | 39.94 | 41.70 | 30.97 |
| Gemini 2.5 Flash [8] | - | 33.46 | 43.88 | 39.72 | 27.55 | 34.38 | 30.70 | 24.62 | 43.07 | 22.64 | 28.35 | 32.92 |
| Gemini 2.5 Pro [8] | - | 39.90 | 51.13 | 46.64 | 54.88 | 62.74 | 58.52 | 40.87 | 48.97 | 45.28 | 43.62 | 49.59 |

### 4.1 Main Result Analysis

We present a comprehensive evaluation on STAR-Bench, as shown in [Tab. 2](https://arxiv.org/html/2510.24693v1#S4.T2). Due to the space limit, detailed results on each task are provided in [Appendix D](https://arxiv.org/html/2510.24693v1#A4). Our key findings are as follows:

STAR-Bench is Challenging
STAR-Bench presents a considerable challenge for existing models. Human evaluators achieve high accuracy across all task categories (e.g., 75.6% on perception, 88.0% on temporal, and 73.7% on spatial tasks), whereas all tested models fall well below this baseline. Most open-source models perform close to random guessing, and even the best closed-source model, Gemini 2.5 Pro, reaches only 49.59% average accuracy.
In addition, model predictions on STAR-Bench exhibit low reliability, as evidenced by the pronounced gap between their Average Accuracy (AA) and All-Correct-Rate (ACR) scores. A detailed discussion of this issue is provided in [Sec. E.1](https://arxiv.org/html/2510.24693v1#A5.SS1).
Although the underlying audio data for the temporal tasks (e.g., FSD50K, Clotho) is commonly used for model pre-training, our novel task formulation of temporal reasoning deliberately departs from conventional audio QA formats. This design allows for a more thorough evaluation of the integrated capabilities of current models.

A Clear Performance Gap between Closed-Source and Open-Source Models
On the foundational perception and temporal tasks, Gemini 2.5 Pro establishes a commanding lead among all models. On spatial tasks, however, nearly all models, both closed- and open-source, perform poorly. As indicated by the prior experiment ([Fig. 3](https://arxiv.org/html/2510.24693v1#S3.F3)), this is likely because most models (except BAT) discard multi-channel information during preprocessing, thereby losing key acoustic cues needed for spatial reasoning.
Among closed-source models, Gemini 2.5 Pro surpasses Gemini 2.5 Flash, suggesting that stronger reasoning capabilities deliver substantial gains. In contrast, open-source models show the opposite pattern: the “think” modes of Audio Flamingo 3 and Xiaomi-MiMo-Audio perform worse than their no-thinking counterparts, implying that without sufficiently solid perceptual and knowledge foundations, reasoning can be ineffective or even detrimental.

Figure: Figure 6: Error distribution across temporal and spatial Tasks.
Refer to caption: x6.png

### 4.2 Discussion: Why Do Existing Models Struggle on STAR-Bench ?

To better understand the underlying causes of the poor performance of existing models, we conduct a detailed error analysis along with a series of ablation studies. Due to space limitation, the ablation study on spatial reasoning is provided in [Sec. E.2](https://arxiv.org/html/2510.24693v1#A5.SS2).

Error Analysis.
We conduct a manual error analysis on 200 failed predictions sampled equally from temporal and spatial tasks of three representative models (Gemini 2.5 Pro, GPT-4o-audio, and Qwen-2.5-Omni).
For temporal tasks, our analysis reveals a clear capability hierarchy across the models. The open-source Qwen-2.5-Omni shows major deficiencies in all three core abilities: its perception is coarse-grained and unable to capture subtle inter-segment distinctions, and a substantial knowledge gap (54%) leads to reasoning that often appears specious due to the absence of physical-world grounding. GPT-4o-audio demonstrates stronger knowledge, but still suffers from perceptual and reasoning limitations, along with low-level issues such as misalignment between reasoning and final answers. In contrast, Gemini 2.5 Pro excels in knowledge and reasoning, shifting its primary bottleneck to the more advanced challenge of fine-grained perception (84%). As shown in [Fig. 7](https://arxiv.org/html/2510.24693v1#S4.F7), Gemini 2.5 Pro is the only model to succeed by providing a remarkably detailed description of acoustic nuances. Our finding suggests that the advanced world knowledge is deeply embedded within detailed audio-text captioning. While open-source models largely remain at a coarse semantic level (e.g., sound event classification), our analysis highlights that enabling them to generate fine-grained acoustic descriptions is critical toward more robust reasoning.
On the other hand, most models demonstrate a lack of native spatial awareness in audio tasks, with weaknesses in perception, knowledge, and reasoning. Additionally, a prevalent type of error involves vision-centric hallucinations (e.g., “…based on the car’s trajectory in the video…”). This may be attributable to the models’ training on visual spatial tasks, leading them to misapply visual reasoning to auditory inputs.

Figure: Figure 7: An error case in temporal reasoning task. More cases are provided in the [Appendix F](https://arxiv.org/html/2510.24693v1#A6).
Refer to caption: x7.png

Figure: Figure 8: The sensitivity analysis in fine-grained perception.
Refer to caption: x8.png

Lack of Human-like Sensitivity in Fine-Grained Perception.
To quantify the gap in perceptual sensitivity, we present model audiograms in [Fig. 9](https://arxiv.org/html/2510.24693v1#S4.F9) (a)(b)(c).
[Fig. 9](https://arxiv.org/html/2510.24693v1#S4.F9) (e)(f)(g) further track the performance of both models and human subjects on the three core acoustic attributes (pitch, loudness, and duration) as task difficulty decreases. The results reveal a stark performance gap between all models and the human baseline, particularly in the perception of fine-grained loudness differences.
A clear trend is visible even for the top-performing Gemini 2.5 Pro: its accuracy, while competent on easier tasks, plummets as perceptual granularity increases. This directly corroborates our error analysis, identifying fine-grained perception as its primary bottleneck. Notably, its performance on duration perception is an exception, showcasing temporal grounding capabilities superior to those of other models by accurately assessing audio segment lengths.

Ablation Study on Temporal Reasoning.
To further pinpoint the specific limitations of temporal reasoning, we augment the baseline audio segment reordering task with two progressively easier settings: (1) + Global Caption, where a single sentence describing the overall scene is provided as a contextual guide; and (2) + Uncut Audio, where the complete, unsegmented audio track is offered as a reference, reducing the task to a straightforward process where the correct order can be determined simply by comparing and grounding each segment within the full audio.
As shown in [Fig. 9](https://arxiv.org/html/2510.24693v1#S4.F9), Gemini 2.5 Pro’s performance scales effectively with task simplification, culminating in a near-perfect 99% accuracy in the + Uncut Audio setting. In contrast, the open-source models show minimal to no improvement across these settings. Their performance remains stagnant even when provided with the complete audio reference, despite the simplified nature of the task. This finding starkly exposes a core weakness in current open-source models: a fundamental inability to effectively compare, ground, and integrate information from multiple audio inputs.

## 5 Conclusion

We introduce STAR-Bench, a comprehensive benchmark for evaluating 4D audio intelligence over time and 3D space.
We use rigorous human annotation, consensus review, and expert validation to ensure the high quality of data samples.
STAR-Bench establishes standardized tasks and protocols for studying 4D audio intelligence, offering actionable diagnostics for model developers.
We expect STAR-Bench to accelerate progress on advanced audio models and training with spatialized corpora, capabilities that are crucial for embodied agents.

## References

- Abouelenin et al. (2025)
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al.
Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.
*arXiv preprint arXiv:2503.01743*, 2025.
- Achiam et al. (2023)
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al.
Gpt-4 technical report.
*arXiv preprint arXiv:2303.08774*, 2023.
- AI et al. (2025)
Inclusion AI, Biao Gong, Cheng Zou, Chuanyang Zheng, Chunluan Zhou, Canxiang Yan, Chunxiang Jin, Chunjie Shen, Dandan Zheng, Fudong Wang, et al.
Ming-omni: A unified multimodal model for perception and generation.
*arXiv preprint arXiv:2506.09344*, 2025.
- Cai et al. (2024)
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, Xiaoyi Dong, Haodong Duan, Qi Fan, Zhaoye Fei, Yang Gao, Jiaye Ge, Chenya Gu, Yuzhe Gu, Tao Gui, Aijia Guo, Qipeng Guo, Conghui He, Yingfan Hu, Ting Huang, Tao Jiang, Penglong Jiao, Zhenjiang Jin, Zhikai Lei, Jiaxing Li, Jingwen Li, Linyang Li, Shuaibin Li, Wei Li, Yining Li, Hongwei Liu, Jiangning Liu, Jiawei Hong, Kaiwen Liu, Kuikun Liu, Xiaoran Liu, Chengqi Lv, Haijun Lv, Kai Lv, Li Ma, Runyuan Ma, Zerun Ma, Wenchang Ning, Linke Ouyang, Jiantao Qiu, Yuan Qu, Fukai Shang, Yunfan Shao, Demin Song, Zifan Song, Zhihao Sui, Peng Sun, Yu Sun, Huanze Tang, Bin Wang, Guoteng Wang, Jiaqi Wang, Jiayu Wang, Rui Wang, Yudong Wang, Ziyi Wang, Xingjian Wei, Qizhen Weng, Fan Wu, Yingtong Xiong, Chao Xu, Ruiliang Xu, Hang Yan, Yirong Yan, Xiaogui Yang, Haochen Ye, Huaiyuan Ying, Jia Yu, Jing Yu, Yuhang Zang, Chuyu Zhang, Li Zhang, Pan Zhang, Peng Zhang, Ruijie Zhang, Shuo Zhang, Songyang Zhang, Wenjian Zhang,
Wenwei Zhang, Xingcheng Zhang, Xinyue Zhang, Hui Zhao, Qian Zhao, Xiaomeng Zhao, Fengzhe Zhou, Zaida Zhou, Jingming Zhuo, Yicheng Zou, Xipeng Qiu, Yu Qiao, and Dahua Lin.
Internlm2 technical report, 2024.
- Cheng et al. (2025)
Zixu Cheng, Jian Hu, Ziquan Liu, Chenyang Si, Wei Li, and Shaogang Gong.
V-star: Benchmarking video-llms on video spatio-temporal reasoning.
*arXiv preprint arXiv:2503.11495*, 2025.
- Chu et al. (2023)
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou.
Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.
*arXiv preprint arXiv:2311.07919*, 2023.
- Chu et al. (2024)
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou.
Qwen2-audio technical report.
*arXiv preprint arXiv:2407.10759*, 2024.
- Comanici et al. (2025)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025.
- Ding et al. (2025)
Shengyuan Ding, Shenxi Wu, Xiangyu Zhao, Yuhang Zang, Haodong Duan, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Dahua Lin, and Jiaqi Wang.
Mm-ifengine: Towards multimodal instruction following, 2025.
URL [https://arxiv.org/abs/2504.07957](https://arxiv.org/abs/2504.07957).
- Dinkel et al. (2025)
Heinrich Dinkel, Gang Li, Jizhong Liu, Jian Luan, Yadong Niu, Xingwei Sun, Tianzi Wang, Qiyang Xiao, Junbo Zhang, and Jiahao Zhou.
Midashenglm: Efficient audio understanding with general audio captions.
*arXiv preprint arXiv:2508.03983*, 2025.
- Drossos et al. (2019)
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen.
Clotho: An audio captioning dataset, 2019.
URL [https://arxiv.org/abs/1910.09387](https://arxiv.org/abs/1910.09387).
- Fonseca et al. (2022)
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra.
FSD50K: an open dataset of human-labeled sound events.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 30:829–852, 2022.
- Ghosh et al. (2025)
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro.
Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities.
In *Forty-second International Conference on Machine Learning*, 2025.
- Goel et al. (2025)
Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, and Bryan Catanzaro.
Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.
*arXiv preprint arXiv:2507.08128*, 2025.
- Gong et al. (2023)
Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass.
Joint audio and speech understanding.
In *2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)*, pp. 1–8, 2023.
doi: 10.1109/ASRU57964.2023.10389742.
- Huang et al. (2025)
Chien-yu Huang, Wei-Chih Chen, Shu-wen Yang, Andy T Liu, Chen-An Li, Yu-Xiang Lin, Wei-Cheng Tseng, Anuj Diwan, Yi-Jen Shih, Jiatong Shi, et al.
Dynamic-superb phase-2: A collaboratively expanding benchmark for measuring the capabilities of spoken language models with 180 tasks.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Jiang et al. (2024)
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al.
Mixtral of experts.
*arXiv preprint arXiv:2401.04088*, 2024.
- KimiTeam et al. (2025)
KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang, Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingcheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Dongchao Yang, Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, and Zaida Zhou.
Kimi-audio technical report, 2025.
URL [https://arxiv.org/abs/2504.18425](https://arxiv.org/abs/2504.18425).
- Kong et al. (2024)
Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro.
Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities.
In *International Conference on Machine Learning*, pp. 25125–25148. PMLR, 2024.
- Kumar et al. (2025)
Sonal Kumar, Šimon Sedláček, Vaibhavi Lokegaonkar, Fernando López, Wenyi Yu, Nishit Anand, Hyeonggon Ryu, Lichang Chen, Maxim Plička, Miroslav Hlaváček, et al.
Mmau-pro: A challenging and comprehensive benchmark for holistic evaluation of audio general intelligence.
*arXiv preprint arXiv:2508.13992*, 2025.
- Li et al. (2025)
Jinsong Li, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Jiaqi Wang, and Dahua Lin.
Beyond fixed: Training-free variable-length denoising for diffusion large language models, 2025.
URL [https://arxiv.org/abs/2508.00819](https://arxiv.org/abs/2508.00819).
- Liu et al. (2024a)
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al.
Deepseek-v3 technical report.
*arXiv preprint arXiv:2412.19437*, 2024a.
- Liu et al. (2024b)
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin.
Mmbench: Is your multi-modal model an all-around player?, 2024b.
URL [https://arxiv.org/abs/2307.06281](https://arxiv.org/abs/2307.06281).
- Liu et al. (2024c)
Ziyu Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Haodong Duan, Conghui He, Yuanjun Xiong, Dahua Lin, and Jiaqi Wang.
Mia-dpo: Multi-image augmented direct preference optimization for large vision-language models, 2024c.
URL [https://arxiv.org/abs/2410.17637](https://arxiv.org/abs/2410.17637).
- Liu et al. (2025)
Ziyu Liu, Yuhang Zang, Yushan Zou, Zijian Liang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang.
Visual agentic reinforcement fine-tuning, 2025.
URL [https://arxiv.org/abs/2505.14246](https://arxiv.org/abs/2505.14246).
- Lu et al. (2025)
Ke-Han Lu, Zhehuai Chen, Szu-Wei Fu, Chao-Han Huck Yang, Sung-Feng Huang, Chih-Kai Yang, Chee-En Yu, Chun-Wei Chen, Wei-Chih Chen, Chien-yu Huang, et al.
Desta2. 5-audio: Toward general-purpose large audio language model with self-generated cross-modal alignment.
*arXiv preprint arXiv:2507.02768*, 2025.
- Ma et al. (2025)
Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al.
Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix.
*arXiv preprint arXiv:2505.13032*, 2025.
- Paul et al. (2022)
Sudipta Paul, Amit Roy-Chowdhury, and Anoop Cherian.
AVLEN: Audio-visual-language embodied navigation in 3d environments.
In *NeurIPS*, 2022.
- Qi et al. (2025)
Zhangyang Qi, Zhixiong Zhang, Yizhou Yu, Jiaqi Wang, and Hengshuang Zhao.
Vln-r1: Vision-language navigation via reinforcement fine-tuning, 2025.
URL [https://arxiv.org/abs/2506.17221](https://arxiv.org/abs/2506.17221).
- Sakshi et al. (2025)
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha.
Mmau: A massive multi-task audio understanding and reasoning benchmark.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Scheibler et al. (2018)
Robin Scheibler, Eric Bezzam, and Ivan Dokmanić.
Pyroomacoustics: A python package for audio room simulation and array processing algorithms.
In *2018 IEEE international conference on acoustics, speech and signal processing (ICASSP)*, pp. 351–355. IEEE, 2018.
- Shimada et al. (2023)
Kazuki Shimada, Archontis Politis, Parthasaarathy Sudarsanam, Daniel A Krause, Kengo Uchida, Sharath Adavanne, Aapo Hakala, Yuichiro Koyama, Naoya Takahashi, Shusuke Takahashi, et al.
Starss23: An audio-visual dataset of spatial recordings of real scenes with spatiotemporal annotations of sound events.
*Advances in neural information processing systems*, 36:72931–72957, 2023.
- Tang et al. (2024)
Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang.
SALMONN: Towards generic hearing abilities for large language models.
In *The Twelfth International Conference on Learning Representations*, 2024.
- Team et al. (2025)
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al.
Gemma 3 technical report.
*arXiv preprint arXiv:2503.19786*, 2025.
- Touvron et al. (2023)
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
*arXiv preprint arXiv:2302.13971*, 2023.
- Wang et al. (2024)
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen.
Audiobench: A universal benchmark for audio large language models.
*arXiv preprint arXiv:2406.16020*, 2024.
- Wei et al. (2025a)
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Jiaqi Wang, Xipeng Qiu, and Dahua Lin.
Sim-cot: Supervised implicit chain-of-thought, 2025a.
URL [https://arxiv.org/abs/2509.20317](https://arxiv.org/abs/2509.20317).
- Wei et al. (2025b)
Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, Xipeng Qiu, and Dahua Lin.
Videorope: What makes for good video rotary position embedding?, 2025b.
URL [https://arxiv.org/abs/2502.05173](https://arxiv.org/abs/2502.05173).
- Wu et al. (2025)
Boyong Wu, Chao Yan, Chen Hu, Cheng Yi, Chengli Feng, Fei Tian, Feiyu Shen, Gang Yu, Haoyang Zhang, Jingbei Li, Mingrui Chen, Peng Liu, Wang You, Xiangyu Tony Zhang, Xingyuan Li, Xuerui Yang, Yayue Deng, Yechang Huang, Yuxin Li, Yuxin Zhang, Zhao You, Brian Li, Changyi Wan, Hanpeng Hu, Jiangjie Zhen, Siyu Chen, Song Yuan, Xuelin Zhang, Yimin Jiang, Yu Zhou, Yuxiang Yang, Bingxin Li, Buyun Ma, Changhe Song, Dongqing Pang, Guoqiang Hu, Haiyang Sun, Kang An, Na Wang, Shuli Gao, Wei Ji, Wen Li, Wen Sun, Xuan Wen, Yong Ren, Yuankai Ma, Yufan Lu, Bin Wang, Bo Li, Changxin Miao, Che Liu, Chen Xu, Dapeng Shi, Dingyuan Hu, Donghang Wu, Enle Liu, Guanzhe Huang, Gulin Yan, Han Zhang, Hao Nie, Haonan Jia, Hongyu Zhou, Jianjian Sun, Jiaoren Wu, Jie Wu, Jie Yang, Jin Yang, Junzhe Lin, Kaixiang Li, Lei Yang, Liying Shi, Li Zhou, Longlong Gu, Ming Li, Mingliang Li, Mingxiao Li, Nan Wu, Qi Han, Qinyuan Tan, Shaoliang Pang, Shengjie Fan, Siqi Liu, Tiancheng Cao, Wanying Lu, Wenqing He, Wuxun Xie, Xu Zhao, Xueqi Li, Yanbo Yu,
Yang Yang, Yi Liu, Yifan Lu, Yilei Wang, Yuanhao Ding, Yuanwei Liang, Yuanwei Lu, Yuchu Luo, Yuhe Yin, Yumeng Zhan, Yuxiang Zhang, Zidong Yang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Xiangyu Zhang, and Yibo Zhu.
Step-audio 2 technical report, 2025.
URL [https://arxiv.org/abs/2507.16632](https://arxiv.org/abs/2507.16632).
- Xiaomi (2025)
LLM-Core-Team Xiaomi.
Mimo-audio: Audio language models are few-shot learners, 2025.
URL [https://github.com/XiaomiMiMo/MiMo-Audio](https://github.com/XiaomiMiMo/MiMo-Audio).
- Xing et al. (2025a)
Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, and Dahua Lin.
Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction, 2025a.
URL [https://arxiv.org/abs/2410.17247](https://arxiv.org/abs/2410.17247).
- Xing et al. (2025b)
Long Xing, Qidong Huang, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Jinsong Li, Shuangrui Ding, Weiming Zhang, Nenghai Yu, Jiaqi Wang, Feng Wu, and Dahua Lin.
Scalecap: Inference-time scalable image captioning via dual-modality debiasing, 2025b.
URL [https://arxiv.org/abs/2506.19848](https://arxiv.org/abs/2506.19848).
- Xu et al. (2025)
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al.
Qwen2. 5-omni technical report.
*arXiv preprint arXiv:2503.20215*, 2025.
- Yang et al. (2025a)
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al.
Qwen3 technical report.
*arXiv preprint arXiv:2505.09388*, 2025a.
- Yang et al. (2024)
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al.
Air-bench: Benchmarking large audio-language models via generative comprehension.
*arXiv preprint arXiv:2402.07729*, 2024.
- Yang et al. (2025b)
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang, Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, et al.
Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents.
*arXiv preprint arXiv:2502.09560*, 2025b.
- Yang et al. (2025c)
Sihan Yang, Runsen Xu, Yiman Xie, Sizhe Yang, Mo Li, Jingli Lin, Chenming Zhu, Xiaochen Chen, Haodong Duan, Xiangyu Yue, et al.
Mmsi-bench: A benchmark for multi-image spatial intelligence.
*arXiv preprint arXiv:2505.23764*, 2025c.
- Yao et al. (2024)
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al.
Minicpm-v: A gpt-4v level mllm on your phone.
*arXiv preprint arXiv:2408.01800*, 2024.
- Zhang et al. (2025a)
Beichen Zhang, Yuhong Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Haodong Duan, Yuhang Cao, Dahua Lin, and Jiaqi Wang.
Booststep: Boosting mathematical capability of large language models via improved single-step reasoning, 2025a.
URL [https://arxiv.org/abs/2501.03226](https://arxiv.org/abs/2501.03226).
- Zhang et al. (2025b)
Zhixiong Zhang, Shuangrui Ding, Xiaoyi Dong, Songxin He, Jianfan Lin, Junsong Tang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang.
Sec: Advancing complex video object segmentation via progressive concept construction, 2025b.
URL [https://arxiv.org/abs/2507.15852](https://arxiv.org/abs/2507.15852).
- Zheng et al. (2024)
Zhisheng Zheng, Puyuan Peng, Ziyang Ma, Xie Chen, Eunsol Choi, and David Harwath.
Bat: Learning to reason about spatial sounds with large language models.
In *International Conference on Machine Learning*, pp. 61454–61469. PMLR, 2024.

## The Use of Large Language Models

We used Gemini-2.5-Pro to assist in expanding and consolidating the taxonomy of tasks in our benchmark. Both DeepSeek-V3 and Gemini-2.5-Pro were utilized for the automated pre-screening of candidate data.
The final task definitions and data samples are verified by humans.
We also used GPT-4o to generate some of the illustrative figures presented in the paper, and used GPT-5 to polish the manuscript text.
Only human-verified revisions are included in the final version.

## Appendix A Related Work

### A.1 Audio Language Models

With the advancements of large language models (LLMs) and multimodal language models (Yang et al., 2025a; Jiang et al., 2024; Achiam et al., 2023; Comanici et al., 2025; Cai et al., 2024; Touvron et al., 2023; Liu et al., 2024c; 2025; Zhang et al., 2025b; Qi et al., 2025; Xing et al., 2025b; a; Ding et al., 2025; Wei et al., 2025a; b; Li et al., 2025; Zhang et al., 2025a), recent research has increasingly focused on integrating audio perception with LLMs to enhance audio understanding and reasoning. Existing methods can be broadly grouped into two categories: Large Audio Language Models(LALMs) and Omni Language Models(OLMs).

Most LALMs combine a pre-trained audio encoder with an LLM backbone, where the two modalities are aligned via large-scale text-audio joint training. Notable models include LTU-AS (Gong et al., 2023), SALMONN (Tang et al., 2024), MidashengLM (Dinkel et al., 2025), Audio Flamingo series (Ghosh et al., 2025; Goel et al., 2025), Qwen-Audio series (Chu et al., 2023; 2024), Step-Audio (Wu et al., 2025) and Mimo-Audio (Xiaomi, 2025). These models have achieved remarkable performance across a wide range of audio understanding tasks, including automatic speech recognition(ASR), spoken question answering(SpokenQA), and automated audio captioning(AAC). In parallel, OLMs extend this paradigm to unify multimodal understanding with representative examples such as Qwen-2.5-Omni (Xu et al., 2025), Ming-Omni (AI et al., 2025),MiniCPM-O (Yao et al., 2024), Phi-4 (Abouelenin et al., 2025), GPT-4o (Achiam et al., 2023), and Gemini 2.5 (Comanici et al., 2025). Notably, they also achieve impressive performance on audio understanding and reasoning, highlighting their potential to bridge multimodal perception and advanced audio intelligence.

### A.2 Audio Benchmarks

Existing audio benchmarks illustrate the rapid progress of multimodal evaluation but also expose limitations. AudioBench (Wang et al., 2024) and AIR-Bench (Yang et al., 2024) primarily focus on tasks such as automatic speech recognition (ASR), spoken question answering (SpokenQA), and audio captioning (AAC). These settings tend to reduce audio understanding to transcription or description, thereby neglecting the broader spectrum of acoustic reasoning. MMAU (Sakshi et al., 2025) and MMAR (Ma et al., 2025) further expand the scope, yet their results reveal an inherent weakness: LLMs with audio captions can achieve comparable performance to advanced LALMs, suggesting that these benchmarks probe little beyond language-level semantics. MMAU-Pro (Kumar et al., 2025) attempts to add temporal and spatial reasoning, but its scope is restricted to single-audio temporal reasoning and single static spatial reasoning.

Beyond the LALMs evaluation, multimodal benchmarks in video question answering (Cheng et al., 2025; Yang et al., 2025c) and embodied AI (Yang et al., 2025b) have emphasized temporal and spatial reasoning. However, these frameworks are predominantly grounded in the visual modality, leaving the audio modality underexplored. Real-world audio understanding frequently requires integrating information across multiple sound streams and reasoning about subtle changes in intensity, phase, or frequency—capabilities that existing benchmarks scarcely capture.

Our benchmark aims to address these gaps by introducing tasks that require multi-audio input and cross-audio reasoning, such as comparing or integrating information across multiple sound inputs, as well as fine-grained spatio-temporal deep reasoning, such as tracking how acoustic patterns evolve with underlying physical changes. Rather than being limited to surface-level semantics, the benchmark is designed to assess whether models can leverage raw audio cues to perform physically grounded reasoning across spatial and temporal dimensions.

## Appendix B Details of Data Annotation

In this section, we present the details of data annotation.

### B.1 Prompts for audio captioning

The prompt for Gemini 2.5 Pro audio captioning: “Please provide a detailed description of the audio, including speech, music, environmental sounds, and any other noticeable elements. Be as specific as possible.”

### B.2 Detail information for foundational acoustic perception

[Tab. 3](https://arxiv.org/html/2510.24693v1#A2.T3) details the ranges and levels used for each acoustic attribute, alongside illustrative examples of our foundational acoustic perception tasks.

**Table 3: Task examples of foundational acoustic perception.**
| Attribute | Range / Level | Example |
| --- | --- | --- |
| Absolute Perception Range |  |  |
| Pitch,<br>Loudness | 125 Hz - 8000 Hz<br>-10dB - 110dB | [Audio]The audio you just heard is divided into two halves. |
| Does a sound appear in the first half, the second half, or is it not present at all? |  |  |
| (A) The first half (B) The second half (C) It is not present at all (D) Unable to determine |  |  |
| Azimuth | 0° - 360° | [Audio] Given that 0° is directly in front and the angle increases clockwise, which azimuth range is the sound most likely coming from? |
| (A) Front-Right (0°–90°) (B) Back-Right (90°–180°) (C) Back-Left (180°–270°) (D) Front-Left (270°–360°) (E) Unable to determine |  |  |
| Elevation | -90° - 90° | [Audio] Where does the sound seem to be coming from in terms of elevation, relative to ear level? |
| (A) Above ear level (B) Below ear level (C) At ear level (D) Unable to determine |  |  |
| Distance | 0 meter - 10 meters | [Audio] How far away does the sound seem to be? |
| (A) Near (within about 0–3 meters) (B) Medium (around 3–8 meters) (C) Far (more than 8 meters) (D) Unable to determine |  |  |
| Relative Discrimination Sensitivity |  |  |
| Pitch | 0, 50, 100, 200, 400, 1200 (cents) | [Audio] Which sound has a higher pitch: the first sound, the second sound, or are they the same? |
| (A) The first sound has a higher pitch (B) The second sound has a higher pitch(C) Both sounds are the same (D) Unable to determine |  |  |
| Loudness | 0, 4, 8, 12, 24, 48 (dB) | [Audio] Which sound is louder: the first sound, the second sound, or are they the same? |
| (A) The first sound is louder (B) The second sound is louder (C) Both sounds are the same (D) Unable to determine |  |  |
| Duration | 0, 20, 50, 100, 150, 200 (%) | [Audio] Which sound is longer: the first sound, the second sound, or are they the same? |
| (A) The first sound is longer (B) The second sound is longer (C) Both sounds are the same (D) Unable to determine |  |  |
| Azimuth | 30, 60, 90, 120, 150, 180 (°) | Audio 1: [Audio$_$1] Audio 2:[Audio$_$2] Are Audio 1 and Audio 2 at the same azimuth? (Consider differences of less than 45° as the same.) |
| (A) Same (B) Different (C) Unable to determine |  |  |
| Elevation | 15, 90, 120, 150 (°) | Audio 1: [Audio$_$1] Audio 2:[Audio$_$2] Which audio has the higher elevation angle? (Consider differences of less than 45° as the same.) |
| (A) Audio 1 is higher (B) Audio 2 is higher (C) Both are at the same elevation (D) Unable to determine |  |  |
| Distance | 1-2, 4-5, 6-7, 8-9 (meters) | Audio 1: [Audio$_$1] Audio 2:[Audio$_$2] Which audio is farther away? (Consider differences of less than 3 meters as the same.) |
| (A) Audio 1 is farther away (B) Audio 2 is farther away (C) Both audios are the same (D) Unable to determine |  |  |

#### B.2.1 Binaural Audio Synthesis

We generated binaural recordings for foundational perception tasks (azimuth, elevation, distance) in Pyroomacoustics (Scheibler et al., 2018) across three rectangular rooms—small (4.0×3.5×2.8 m), medium (8.0×6.0×3.5 m), and large (20×15×8 m)—each with a frequency-independent wall absorption coefficient of 0.25. Image-source reflections were modeled up to order 10 at 44.1 kHz (matched to the HRTF sampling rate). For each room, we evaluated two listener positions (distinct Cartesian coordinates) and oriented the head toward the +x axis. Binaural reception used a co-located two-microphone array at the listener position with ear-specific directivity derived from a measured SOFA HRTF(^1^11[https://sofacoustics.org/data/database/mit/mit_kemar_normal_pinna.sofa](https://sofacoustics.org/data/database/mit/mit_kemar_normal_pinna.sofa)) (MIT KEMAR, “normal pinna”; interpolation order 12, 1000 points), loaded via a local SOFA reader and applied to the left/right channels.

For each condition (room × listener), sources were placed on a sphere centered at the listener (radii 1–10 m; configurable azimuth/elevation), and ear-specific BRIRs were computed. Mono source signals were drawn from three curated audio clips (“alarm,” “applause,” “telephones”), downmixed if necessary. Rendering was performed by convolving each dry signal with the left/right BRIRs after an early/late mix to emphasize distance cues: we preserved the first 80 ms and attenuated the late tail by 0.5. We then applied global peak normalization across the batch to avoid clipping while preserving inter-position level differences.

We discretized each attribute into fixed partitions to control dataset balance.

Absolute azimuth:
Eight angles
$\{30^{\circ},\,60^{\circ},\,120^{\circ},\,150^{\circ},\,210^{\circ},\,240^{\circ},\,300^{\circ},\,330^{\circ}\}$.
For each angle we rendered all combinations of 3 rooms $\times$ 2 listener positions $\times$ 2 source clips, yielding
$8\times(3\times 2\times 2)=96$ utterances.
Absolute elevation:
Six angles
$\{-75^{\circ},\,-45^{\circ},\,-15^{\circ},\,15^{\circ},\,45^{\circ},\,75^{\circ}\}$.
Per angle we rendered 3 rooms $\times$ 2 listener positions $\times$ 2 source clips, for
$6\times(3\times 2\times 2)=72$ utterances.
Absolute distance:
Radii from 1–10 m with a nonuniform allocation to emphasize near-field cues:
for 1–7 m we generated 6 utterances per meter (42 total),
and for 8–10 m we generated 3 per meter (9 total),
giving $42+9=51$ utterances per (room $\times$ listener) set.

Relative azimuth:
Differences were multiples of $30^{\circ}$:
$\{30^{\circ},\,60^{\circ},\,90^{\circ},\,120^{\circ},\,150^{\circ},\,180^{\circ}\}$ (6 levels),
totaling $6\times 20=120$ utterances.
Relative elevation:
Four difference angles $\{15^{\circ},\,90^{\circ},\,120^{\circ},\,150^{\circ}\}$
with 18, 17, 17, 12 utterances respectively (64 total).
Relative distance:
Four difference levels $\{1-2,\,4-5,\,6-7,\,8-9\}$ m with counts per level
$\{12,\,12,\,12,\,9\}$, totaling $45$ utterances.

### B.3 Prompt Used for AI-Assisted Automated Filtering of Temporal Task Data

[Fig. 10](https://arxiv.org/html/2510.24693v1#A2.F10) and [Fig. 11](https://arxiv.org/html/2510.24693v1#A2.F11) present our carefully designed prompts, which leverage LLMs and MLLMs to filter candidate data that meet the requirements of audio segment reordering.

Figure: Figure 10: The prompt for our AI-assisted filtering process on temporal tasks.
Refer to caption: Figure/prompt1.png

Figure: Figure 11: The prompt for our AI-assisted filtering process on temporal tasks.
Refer to caption: Figure/prompt2.png

## Appendix C Robust Evaluation

All questions in STAR-Bench are presented as clear multiple-choice questions with well-formatted options. We adopt classification accuracy as the evaluation metric. To determine the correctness of a response, we employ string matching to extract either the chosen option label (e.g., $<$A$>$) or the full text content of the option from the model’s output.

Furthermore, we implement a robust evaluation strategy to ensure rigorous and reliable results. For perception and spatial tasks, we adopt the CircularEval method from MM-Bench (Liu et al., 2024b). Specifically, each question is presented to the model $N$ times ($N$ is the number of options), with the option order cyclically rotated in each run to mitigate potential positional biases. For temporal tasks, we conduct three runs per question with different temporal segment orders to evaluate the model’s robustness to sequence variations.
Note that due to the significant API costs, GPT-4o Audio was evaluated only once per question.
This strategy yields two key metrics: Average Accuracy (AA), the mean accuracy across all evaluation runs, and All-Correct Rate (ACR), the proportion of questions answered correctly in every single run, which serves as a stronger indicator of model reliability.

For models that do not support multi-audio input (only Audio Flamingo 3 and its Think variant among the models we evaluated), we concatenate the audios with a 2-second silence and specify this in the prompt. In contrast, for models that support multiple audio inputs, we feed them sequentially with textual indices.

To establish a human performance baseline, we conduct a human evaluation on a randomly sampled subset of approximately 10% of the data from each task. This evaluation is performed by 10 university students, from whom we explicitly exclude anyone involved in data annotation or with domain-specific expertise, thereby ensuring a general, non-expert perspective.

## Appendix D Breakdown Results

In this section, we present detailed results for perception, temporal reasoning, and spatial reasoning on STAR-Bench, as shown in [Tab. 4](https://arxiv.org/html/2510.24693v1#A5.T4), [Tab. 5](https://arxiv.org/html/2510.24693v1#A5.T5), and [Tab. 6](https://arxiv.org/html/2510.24693v1#A5.T6).

## Appendix E Further Analysis and Discussion

### E.1 High Output Instability and Concentrated Predictions

The reliability of model outputs on our benchmark is notably low, as evidenced by the stark contrast between their Average Accuracy (AA) and All-Correct-Rate (ACR) scores. Even the top-performing model, Gemini 2.5 Pro, exhibits an average drop of 25.01 percentage points from its AA to its ACR. This issue is even more pronounced for the majority of open-source models, which record an ACR near zero. This score indicates a complete failure to maintain consistent predictions under minor input perturbations. For these models, the instability often manifests as a tendency to concentrate predictions on a specific option, suggesting a reliance on superficial biases rather than genuine understanding.

### E.2 Ablation Study on Spatial Reasoning.

As shown in [Tab. 6](https://arxiv.org/html/2510.24693v1#A5.T6), the results reveal a fundamental limitation of LALMs’ spatial understanding in perception. The native input inherently discards part of the multi-channel information during model preprocessing, which leads to a significant loss of spatial cues that are essential for fine-grained reasoning. On the other hand, the channel-wise input explicitly presents each channel with textual instructions, mitigating some of the information loss. However, as most models are not trained on multi-audio inputs, they struggle to align channel representations and to exploit interaural cues reliably.
Overall, the gap between human and model performance highlights that spatial reasoning in audio remains an unsolved challenge. While channel-wise input can provide partial gains, neither strategy fully captures spatial dependencies, underscoring the need for an audio encoder that natively supports multi-channel audio input.

**Table 4: Results for the foundational perception task. Each cell reports AA / ACR: Average Accuracy (AA; overall accuracy across all runs) / All-Correct Rate (ACR; proportion of samples that are correct on every run). The best model in each category is shown in bold, and the second best is underlined.**
| Model | Size | Absolute Perception Range | Relative Discrimination Sensitivity | MA (%) |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Pitch&Loudness | Azimuth | Elevation | Distance | Pitch | Loudness | Duration | Azimuth | Elevation | Distance |  |  |  |
| Random Guess | — | 25.00 / 0.39 | 20.00 / 0.03 | 25.00 / 0.39 | 25.00 / 0.39 | 25.00 / 0.39 | 25.00 / 0.39 | 25.00 / 0.39 | 33.33 / 3.7 | 25.00 / 0.39 | 25.00 / 0.39 | 25.33 / 0.68 |
| Human | — | 98.67 / — | 73.33 / — | 66.67 / — | 70.00 / — | 83.33 / — | 85.56 / — | 83.33 / — | 83.33 / — | 38.09 / — | 73.68 / — | 75.60 / — |
| SALMONN | 13B | 14.34 / 0.00 | 25.83 / 0.63 | 35.76 / 0.00 | 33.33 / 0.00 | 31.04 / 0.00 | 25.00 / 0.00 | 28.54 / 0.00 | 31.39 / 3.89 | 24.15 / 0.00 | 12.77 / 0.00 | 26.22 / 0.45 |
| Audio Flamingo 3 | 8.4B | 37.59 / 0.00 | 27.92 / 3.13 | 28.82 / 0.00 | 32.84 / 0.00 | 42.50 / 1.67 | 28.96 / 0.00 | 34.79 / 0.00 | 38.61 / 6.67 | 33.90 / 0.00 | 35.56 / 0.00 | 34.15 / 1.15 |
| Audio Flamingo 3 think | 8.4B | 51.75 / 6.99 | 8.75 / 0.00 | 33.33 / 1.04 | 8.33 / 0.00 | 36.04 / 8.33 | 45.63 / 2.50 | 59.38 / 38.33 | 41.11 / 4.17 | 12.29 / 0.00 | 10.00 / 0.00 | 30.66 / 6.14 |
| Qwen2-Audio-Instruct | 8.4B | 35.66 / 1.40 | 22.50 / 0.00 | 48.61 / 10.76 | 12.75 / 0.98 | 35.63 / 0.00 | 16.25 / 0.00 | 26.46 / 0.00 | 35.00 / 8.06 | 21.61 / 1.69 | 23.88 / 0.00 | 27.84 / 2.29 |
| DeSTA2.5-Audio | 8.8B | 16.96 / 0.00 | 21.25 / 0.42 | 45.49 / 1.39 | 35.78 / 1.47 | 11.67 / 0.00 | 11.25 / 0.00 | 22.71 / 0.00 | 33.06 / 7.78 | 10.59 / 0.00 | 29.44 / 0.00 | 23.82 / 1.11 |
| BAT | 7B | 0.00 / 0.00 | 26.04 / 26.04 | 41.67 / 41.67 | 23.53 / 23.53 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 37.50 / 37.50 | 0.00 / 0.00 | 0.00 / 0.00 | 12.87 / 12.87 |
| Phi4-MM | 5.5B | 9.44 / 0.00 | 24.17 / 0.00 | 15.97 / 0.00 | 26.96 / 0.00 | 24.38 / 0.00 | 30.00 / 0.00 | 27.92 / 0.00 | 36.94 / 0.00 | 32.62 / 0.00 | 27.22 / 0.00 | 25.56 / 0.00 |
| Kimi-Audio | 7B | 18.71 / 0.00 | 18.12 / 0.00 | 38.19 / 0.00 | 18.13 / 0.00 | 24.38 / 0.00 | 32.29 / 0.00 | 34.17 / 0.83 | 39.72 / 3.89 | 25.00 / 0.85 | 9.44 / 0.00 | 25.82 / 0.56 |
| MiDashengLM | 7B | 48.95 / 33.57 | 20.63 / 0.00 | 48.26 / 11.81 | 29.90 / 0.98 | 40.00 / 34.17 | 17.08 / 0.83 | 23.54 / 7.50 | 34.72 / 8.61 | 27.12 / 1.69 | 42.22 / 6.11 | 33.24 / 10.53 |
| Step-Audio-2-mini | 7B | 37.59 / 0.00 | 20.00 / 0.00 | 31.60 / 0.69 | 29.41 / 0.00 | 25.00 / 0.00 | 29.17 / 0.00 | 32.29 / 0.00 | 20.00 / 0.00 | 31.36 / 0.00 | 25.00 / 0.00 | 28.14 / 0.07 |
| Gemma-3n-E4B-it | 7.5B | 7.18 / 0.00 | 24.38 / 4.17 | 25.00 / 0.00 | 17.65 / 0.00 | 38.75 / 0.00 | 8.75 / 0.00 | 15.00 / 5.83 | 40.56 / 1.94 | 23.73 / 0.00 | 23.33 / 0.00 | 22.43 / 1.19 |
| Ming-Lite-Omni-1.5 | 18.9B | 28.67 / 0.00 | 20.21 / 0.00 | 27.78 / 0.35 | 30.39 / 3.92 | 16.67 / 16.67 | 16.67 / 16.67 | 16.67 / 16.67 | 41.67 / 0.28 | 32.81 / 0.00 | 36.11 / 0.00 | 26.77 / 5.46 |
| Qwen-2.5-Omni | 7B | 27.45 / 3.50 | 18.33 / 0.21 | 27.57 / 1.47 | 41.67 / 1.47 | 48.13 / 35.00 | 39.79 / 15.00 | 38.33 / 26.67 | 16.11 / 0.28 | 11.02 / 0.00 | 40.56 / 2.78 | 30.90 / 8.64 |
| Xiaomi-MiMo-Audio | 7B | 36.71 / 5.59 | 18.54 / 19.17 | 48.26 / 3.82 | 36.27 / 2.94 | 46.04 / 24.17 | 36.46 / 0.83 | 17.70 / 16.67 | 40.56 / 2.22 | 20.98 / 0.00 | 27.78 / 1.67 | 32.93 / 7.71 |
| Xiaomi-MiMo-Audio-think | 7B | 43.01 / 14.69 | 11.67 / 0.00 | 25.69 / 0.00 | 39.21 / 4.90 | 28.13 / 3.33 | 15.21 / 1.67 | 22.71 / 1.67 | 29.44 / 2.50 | 21.88 / 0.45 | 32.22 / 1.67 | 26.92 / 3.09 |
| MiniCPM-O-v2.6 | 8B | 46.33 / 8.39 | 24.58 / 0.21 | 23.26 / 0.35 | 29.90 / 0.00 | 38.13 / 3.33 | 38.96 / 4.17 | 32.08 / 3.33 | 37.22 / 2.78 | 22.10 / 0.22 | 22.78 / 0.00 | 31.53 / 2.28 |
| GPT-4o Audio | — | 45.28 / — | 16.67 / — | 44.44 / — | 3.92 / — | 43.33 / — | 36.04 / — | 46.46 / — | 29.58 / — | 11.86 / — | 40.00 / — | 31.76 / — |
| Gemini 2.5 Flash | — | 62.59 / 18.19 | 12.50 / 0.00 | 18.06 / 0.35 | 40.69 / 1.47 | 48.54 / 21.67 | 40.83 / 6.67 | 63.13 / 27.50 | 37.08 / 9.17 | 25.42 / 0.85 | 48.33 / 4.44 | 39.72 / 9.03 |
| Gemini 2.5 Pro | — | 86.71 / 62.94 | 25.83 / 1.25 | 5.88 / 0.00 | 41.18 / 5.88 | 63.33 / 52.50 | 33.75 / 15.83 | 78.96 / 68.33 | 37.08 / 13.75 | 29.24 / 6.36 | 64.44 / 12.22 | 46.64 / 23.91 |

**Table 5: Results for the temporal reasoning task. Each cell reports AA / ACR: Average Accuracy (AA; overall accuracy across all runs) / All-Correct Rate (ACR; proportion of samples that are correct on every run). The best model in each category is shown in bold, and the second best is underlined.**
| Model | Size | Continuous Processes | Discrete Event Sequences | OA (%) |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Object Spatial Motion | In-Situ State Evolution | Tool & Appliance Operation | Daily Scene Scripts | Event-Triggered Consequences |  |  |  |
| Random Guess | — | 14.29 / 0.00 | 14.29 / 0.00 | 14.29 / 0.00 | 14.29 / 0.00 | 14.29 / 0.00 | 14.29 / 0.00 |
| Human | — | 91.11 / — | 88.89 / — | 87.88 / — | 83.33 / — | 83.33 / — | 88.00 / — |
| SALMONN | 13B | 13.88 / 0.74 | 16.12 / 0.00 | 13.56 / 1.96 | 13.15 / 1.11 | 12.50 / 0.00 | 14.15 / 0.89 |
| Audio Flamingo 3 | 8.4B | 8.55 / 0.00 | 10.08 / 0.47 | 8.66 / 0.98 | 7.22 / 1.11 | 8.33 / 3.13 | 8.67 / 0.67 |
| Audio Flamingo 3 think | 8.4B | 14.37 / 0.00 | 11.78 / 0.93 | 15.36 / 1.47 | 12.96 / 2.22 | 11.46 / 0.00 | 13.59 / 1.00 |
| Qwen2-Audio-Instruct | 8.4B | 12.89 / 0.00 | 13.80 / 0.93 | 12.09 / 0.00 | 12.22 / 1.11 | 11.46 / 0.00 | 12.74 / 0.44 |
| DeSTA2.5-Audio | 8.8B | 16.98 / 0.37 | 15.97 / 1.40 | 19.93 / 1.47 | 15.56 / 0.56 | 11.46 / 0.00 | 16.93 / 0.89 |
| BAT | 7B | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 |
| Phi4-MM | 5.5B | 17.72 / 0.00 | 15.50 / 0.47 | 16.34 / 0.98 | 17.04 / 3.89 | 20.83 / 3.13 | 16.85 / 1.22 |
| Kimi-Audio | 7B | 18.71 / 1.49 | 21.55 / 2.33 | 18.63 / 0.49 | 15.19 / 2.22 | 14.58 / 0.00 | 18.52 / 1.56 |
| MiDashengLM | 7B | 17.10 / 0.37 | 13.33 / 0.00 | 17.16 / 1.96 | 16.67 / 2.22 | 21.88 / 0.00 | 16.30 / 1.00 |
| Step-Audio-2-mini | 7B | 16.11 / 0.37 | 14.42 / 0.00 | 15.52 / 0.00 | 16.30 / 0.00 | 15.63 / 0.00 | 15.59 / 0.11 |
| Gemma-3n-E4B-it | 7.5B | 17.10 / 0.00 | 16.59 / 0.00 | 17.81 / 0.00 | 13.70 / 0.00 | 20.83 / 0.00 | 16.59 / 0.00 |
| Ming-Lite-Omni-1.5 | 18.9B | 17.47 / 1.12 | 16.59 / 0.47 | 13.89 / 0.00 | 17.59 / 1.11 | 14.58 / 0.00 | 16.37 / 0.67 |
| Qwen-2.5-Omni | 7B | 17.10 / 0.37 | 15.35 / 0.93 | 19.77 / 1.47 | 16.48 / 0.56 | 11.46 / 0.00 | 16.96 / 0.78 |
| Xiaomi-MiMo-Audio | 7B | 18.22 / 0.00 | 18.14 / 0.47 | 17.16 / 0.98 | 20.19 / 2.22 | 26.04 / 3.13 | 18.63 / 0.89 |
| Xiaomi-MiMo-Audio-think | 7B | 16.36 / 0.37 | 17.36 / 0.47 | 19.93 / 1.96 | 18.70 / 2.22 | 19.79 / 0.00 | 18.00 / 1.11 |
| MiniCPM-O-v2.6 | 8B | 16.23 / 0.00 | 14.26 / 0.93 | 17.48 / 0.49 | 17.78 / 0.56 | 14.58 / 0.00 | 16.30 / 0.44 |
| GPT-4o Audio | — | 15.61 / — | 16.28 / — | 24.02 / — | 22.78 / — | 25.00 / — | 19.44 / — |
| Gemini 2.5 Flash | — | 30.86 / 3.35 | 23.41 / 3.72 | 38.07 / 12.75 | 30.19 / 7.22 | 34.38 / 9.38 | 30.70 / 6.56 |
| Gemini 2.5 Pro | — | 63.82 / 38.66 | 43.72 / 17.67 | 69.77 / 46.08 | 57.22 / 38.33 | 48.96 / 28.13 | 58.52 / 34.89 |

**Table 6: Results for the spatial reasoning task using native and channel-wise audio input. Each cell reports AA / ACR: Average Accuracy (AA; overall accuracy across all runs) / All-Correct Rate (ACR; proportion of samples that are correct on every run). The best model in each category is shown in bold, and the second best is underlined.**
| Model | Size | Single-Source Static Localization | Multi-Source Spatial Relation | Dynamic Trajectory Tracking | OA (%) |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Native Input | Channel-wise Input | Native Input | Channel-wise Input | Native Input | Channel-wise Input | Native Input | Channel-wise Input |  |  |
| Random Guess | — | 33.33 / 3.70 | — | 33.33 / 3.70 | — | 33.33 / 3.70 | — | 33.33 / 3.70 | — |
| Human | — | 70.00 / — | — | 80.00 / — | — | 77.00 / — | — | 73.72 / — | — |
| SALMONN | 13B | 26.15 / 3.18 | 26.62 / 3.18 | 28.61 / 4.42 | 29.50 / 5.31 | 39.94 / 0.94 | 38.36 / 0.94 | 29.62 / 2.99 | 29.75 / 3.19 |
| Audio Flamingo 3 | 8.4B | 37.22 / 1.77 | 42.87 / 2.12 | 38.35 / 4.42 | 46.31 / 10.62 | 44.03 / 4.72 | 46.23 / 0.94 | 38.91 / 2.99 | 44.35 / 3.78 |
| Audio Flamingo 3 think | 8.4B | 35.45 / 7.42 | 42.87 / 13.78 | 37.46 / 23.01 | 46.02 / 23.01 | 38.05 / 18.87 | 37.11 / 19.81 | 36.45 / 13.35 | 42.36 / 17.13 |
| Qwen2-Audio-Instruct | 8.4B | 21.32 / 8.48 | 6.36 / 1.77 | 24.78 / 3.54 | 12.09 / 4.42 | 15.09 / 0.94 | 11.64 / 2.83 | 20.78 / 5.78 | 8.76 / 2.59 |
| DeSTA2.5-Audio | 8.8B | 23.67 / 2.83 | 20.38 / 4.59 | 34.81 / 9.73 | 41.30 / 19.47 | 37.74 / 10.38 | 32.08 / 21.70 | 29.15 / 5.98 | 27.56 / 11.55 |
| BAT | 7B | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 | 0.00 / 0.00 |
| Phi4-MM | 5.5B | 33.10 / 0.35 | 32.63 / 0.35 | 27.14 / 0.88 | 29.79 / 0.88 | 34.28 / 0.94 | 33.02 / 0.00 | 32.01 / 0.59 | 32.07 / 0.40 |
| Kimi-Audio | 7B | 27.56 / 3.53 | 16.49 / 3.53 | 38.94 / 15.04 | 22.42 / 8.85 | 44.03 / 7.55 | 40.25 / 8.49 | 33.60 / 6.97 | 22.84 / 5.77 |
| MiDashengLM | 7B | 43.11 / 15.19 | 37.22 / 17.67 | 45.43 / 23.89 | 42.77 / 16.81 | 46.23 / 30.19 | 45.60 / 21.70 | 44.29 / 20.32 | 40.24 / 18.33 |
| Step-Audio-2-mini | 7B | 33.33 / 0.00 | 33.33 / 0.00 | 31.27 / 0.00 | 37.46 / 0.00 | 37.74 / 6.38 | 35.22 / 2.83 | 33.80 / 1.34 | 34.66 / 0.60 |
| Gemma-3n-E4B-it | 7.5B | 23.32 / 1.41 | 28.27 / 6.01 | 41.89 / 15.04 | 36.58 / 7.96 | 33.96 / 5.66 | 40.57 / 8.49 | 29.75 / 5.37 | 32.74 / 6.97 |
| Ming-Lite-Omni-1.5 | 18.9B | 20.14 / 6.36 | 34.63 / 6.01 | 35.10 / 9.73 | 33.04 / 9.73 | 38.36 / 18.87 | 39.94 / 20.75 | 27.35 / 9.76 | 35.39 / 9.96 |
| Qwen-2.5-Omni | 7B | 39.46 / 7.07 | 36.98 / 15.19 | 41.30 / 18.58 | 35.10 / 15.93 | 27.04 / 17.92 | 34.59 / 8.49 | 37.25 / 11.95 | 36.05 / 13.94 |
| Xiaomi-MiMo-Audio | 7B | 36.16 / 0.71 | 41.58 / 5.65 | 41.30 / 5.31 | 38.05 / 4.42 | 45.28 / 9.43 | 44.34 / 9.43 | 39.24 / 3.58 | 41.37 / 6.17 |
| Xiaomi-MiMo-Audio-think | 7B | 34.28 / 7.42 | 25.44 / 2.83 | 44.54 / 14.16 | 37.76 / 7.96 | 36.79 / 7.55 | 27.99 / 3.77 | 37.12 / 8.96 | 28.75 / 4.18 |
| MiniCPM-O-v2.6 | 8B | 29.92 / 3.18 | 27.92 / 2.83 | 43.36 / 11.50 | 39.53 / 12.39 | 38.36 / 26.42 | 35.53 / 17.92 | 34.73 / 9.96 | 32.14 / 8.17 |
| GPT-4o Audio | — | 41.81 / — | 42.76 / — | 43.07 / — | 54.87 / — | 39.94 / — | 42.45 / — | 41.70 / — | 45.42 / — |
| Gemini 2.5 Flash | — | 24.62 / 4.95 | 40.75 / 7.42 | 43.07 / 15.93 | 43.07 / 17.70 | 22.64 / 2.83 | 40.57 / 11.32 | 28.35 / 6.97 | 41.23 / 10.56 |
| Gemini 2.5 Pro | — | 40.87 / 10.95 | 34.98 / 11.66 | 48.97 / 25.66 | 49.26 / 20.35 | 45.28 / 14.15 | 47.17 / 7.55 | 43.62 / 14.94 | 40.77 / 12.75 |

## Appendix F Case Study

In this section, we present several case studies of error analysis, including temporal reasoning ([Figs. 12](https://arxiv.org/html/2510.24693v1#A6.F12), [13](https://arxiv.org/html/2510.24693v1#A6.F13), [14](https://arxiv.org/html/2510.24693v1#A6.F14), [15](https://arxiv.org/html/2510.24693v1#A6.F15), [16](https://arxiv.org/html/2510.24693v1#A6.F16) and [17](https://arxiv.org/html/2510.24693v1#A6.F17)) and spatial reasoning ([Fig. 18](https://arxiv.org/html/2510.24693v1#A6.F18)).

Figure: Figure 12: An error case from the temporal reasoning task.
Refer to caption: x10.png

Figure: Figure 13: An error case from the temporal reasoning task.
Refer to caption: x11.png

Figure: Figure 14: An error case from the temporal reasoning task.
Refer to caption: x12.png

Figure: Figure 15: An error case from the temporal reasoning task.
Refer to caption: x13.png

Figure: Figure 16: An error case from the temporal reasoning task.
Refer to caption: x14.png

Figure: Figure 17: An error case from the temporal reasoning task.
Refer to caption: x15.png

Figure: Figure 18: An error case from the spatial reasoning task.
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How would you generalize the presentation of spatial audio to models, beyond Kemar? How would you accommodate ambisonics? It would have been good to provide some example sounds.
soundness: 3
strengths: A clear way of generating spatial audio and temporal reasoning benchmarks is described and implemented. The results show clear lack of perception of current LALMs on tasks requiring these skills. This is a valuable and distinct addition to the plethora of benchmarks coming out
summary: A very topical benchmark for large audio language models, which adds basic audio perception modalities, and explores categories where linguistic descriptions are hard.
weaknesses: The audio scenes were generated binaurally, but this restricts the range of applications of the benchmark to perhaps humanoid listeners. Also, the HRTF used was a Kemar one, which is somewhat limited; and the scene rendering was also a bit limited. Nonetheless a good beginning. The models are of course set up to fail in this testing.

The spatial audio tasks are relatively hopeless, and analysis of the models was not too much possible To an extent, I would have liked more detailed analysis of the temporal reasoning tasks. An ablation I would like to see is if the models perform better if the  with just single channel presentation of these events to the models.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 4
questions: - Do you think gemini 2.5 pro captioning stage filters the data points to be more biased towards the current capabilities of the models? 
- How easy/difficult is it to hill climb on this benchmark for audio models?
- Do the baselines actually support spatial audio?
soundness: 3
strengths: - Introduces and benchmarks multi-audio segment reordering and stereo spacial reasoning which has been ignored by the previous benchmarks
- Proper coverage of non-spatial attributes (Loudness, Pitch, Duration) and spatial attributes (Azimuth, Elevation, Distance)
- Detailed error analysis on why models dont do well on the proposed benchmark
- Thorough reporting of AA and ACR metrics to test the model's reliability
- Really like the finding "a fundamental inability to effectively compare, ground, and integrate information from multiple audio inputs"
summary: The paper introduces STAR-Bench which measures Audio 4D Intelligence (reasoning over sound dynamics in time and 3D space). They combine a Foundational Acoustic Perception setting with a Holistic Spatio-Temporal Reasoning setting that includes segment reordering for continuous and discrete processes and spatial tasks spanning static localization, multi-source relations, and dynamic trajectories. Unlike prior benchmarks where caption-only answering reduces accuracy slightly, STAR-Bench induces far larger drops (-31.5% temporal, -35.2% spatial), evidencing its focus on linguistically hard-to-describe cues. They also provide a comprehensive evaluation of 19 LALMs/OLMs.
weaknesses: - missing details of AI-Assisted Automated Filtering. What exactly are we filtering using gemini 2.5 pro
- Spatial data is synthetic and does not represent real world use cases.
- Fig 8 not readable
- Not sure if the baselines support spatial audio. Analysis on those models would not provide beneficial information
- Support for spatial audio in these models can lead to huge jumps in model performance on the benchmark which questions the difficulty of the benchmark

---

confidence: 5
contribution: 2
initial_rating: 4
presentation: 3
questions: See the weakness section.
soundness: 2
strengths: - Clear problem framing ('audio 4D intelligence) and a structured task design that disaggregates perception, temporal reasoning, and spatial reasoning.
- Useful diagnostic split: absolute vs. relative perceptual tests; temporal re-ordering; spatial subtasks (single-source localization, multi-source relations, dynamic trajectories).
- Strong curation pipeline with AI filtering, human annotation, and expert validation; explicit use of public datasets + simulated audio for coverage and control. 
- Robustness intent (AA vs. ACR; circular option shuffling; segment order perturbations) is a step forward relative to one-shot MCQ protocols. 
- Empirical evidence that 'caption-only' shortcuts collapse on STAR-Bench more than on MMAU/MMAR, supporting the target of non-linguistic cues. (Claim supported in text/figures.)
summary: This paper introduces STAR-Bench, a benchmark to evaluate 4D Audio Intelligence - reasoning over time (1D) and 3D space—via two levels: (a) Foundational Acoustic Perception (six attributes: pitch, loudness, duration, azimuth, elevation, distance; tested in absolute and relative regimes) and (b) Holistic Spatio-Temporal Reasoning (temporal segment re-ordering for continuous/discrete processes; spatial reasoning over static localization, multi-source relations, and dynamic trajectories). The dataset has 2,353 questions and is built through a four-stage pipeline (AI-assisted filtering, human annotation, expert validation) drawing on synthetic audio and real-world corpora (e.g., FSD50K, Clotho, STARSS23). Evaluation is multiple-choice with “robust” perturbations (AA/ACR). Results show humans outperform all models (e.g., ~88% temporal), while the best model (Gemini-2.5 Pro) reaches ~49.6% MA, and a larger drop occurs when answering from captions only, supporting the claim that STAR-Bench targets linguistically hard cues.
weaknesses: - Everything is framed as multiple-choice with string-match grading. No open-ended QAs, while they are more real world.
- A large portion of audio comes from widely used corpora (FSD50K, Clotho, STARSS23); many models likely pretrained on them. The authors argue the task formulation is novel (re-ordering, spatial relations), but clip-level memorization of events/timbres is still possible.
- The paper doesn't quantify inter-rater agreement, item rejection rates, or ambiguity sources.
- In table 1, the authors claim that previous benchmarks like MMAR and MMAU-Pro, do not have spatial reasoning questions, which is wrong. MMAR also has multi-audio questions.
- MMAU, MMAU-Pro and MMAR also contains Temporal Reasoning questions, what does "Deep" signify in the task name in table 1?
- With the recent advancements in Large Audio Language Models where the models are capable of reasoning over all three modalities of audio - sound, speech and music, this benchmark only focuses on sounds which is limiting. Although the authors explicitly frame STAR-Bench as a move away from traditional ASR/captioning style evaluations toward 4D (spatio-temporal) acoustic reasonin - another signal that speech/music tasks per se are out of scope, I strongly feel there can be multiple tasks framed around speech in spatio-temporal setting.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weaknesses.
soundness: 3
strengths: 1. The paper clearly identifies a weakness in existing audio benchmarks. The "caption-only" experiment (Figure 1) provides empirical evidence that current benchmarks often test text-level semantics rather than fine-grained audio perception.
2. The paper's most significant contribution is identifying and proving the insufficiency of standard audio preprocessing in LALMs. The "additive inverse" experiment in Figure 3 is a simple and effective demonstration that a model's spatial reasoning capabilities are non-existent if it just averages channels to mono.
3. The analysis provides clear takeaways for the community, such as the need for models that natively process multi-channel audio and the severe limitations of open-source models in multi-audio grounding (as shown in Figure 9 ablation).
summary: This paper introduces STAR-BENCH, a new benchmark designed to evaluate what the authors term "audio 4D intelligence", the ability to reason about sound dynamics in 3D space and time. The authors argue that existing audio benchmarks primarily test for semantic content that is easily described by text captions, a claim they support with a "caption-only" evaluation experiment where model performance drops only slightly. STAR-BENCH is proposed to fill this gap by focusing on "linguistically hard-to-describe" cues.
weaknesses: 1. My primary concern is with the "channel-wise input" proposed for spatial reasoning. While this is a clever workaround for the "mono-averaging" problem, it fundamentally changes the task. Instead of evaluating native spatial audio perception, this method tests a model's ability to reason about two separate mono audio streams guided by an explicit textual prompt (e.g., "Audio 1 is the left-ear channel and Audio 2 is the right-ear channel"). This confounds auditory intelligence with text-based reasoning and an understanding of the experimental setup. It does not truly measure the model's ability to process and fuse interaural cues (ITD, ILD) from a single binaural stream. The paper acknowledges models are not trained on multi-audio inputs, but I think this workaround is a significant compromise.
2. The temporal reordering task relies on "strong sequential uniqueness". This is robust for tasks governed by physics (e.g., Doppler effect , fluid dynamics). However, I am less convinced about the "Daily Scene Scripts" and "Event-Triggered Consequences"  subcategories. For example, in the teeth-brushing case study (Figure 12) , the "correct" order is given as brushing -> spit foam -> turn on tap/spit. However, an alternative like GPT-4o Audio's turn on water -> brush -> rinse/spit is just as logical and common. This suggests "logical universality"  is not as high as claimed.
3. The foundational perception tasks use sine waves for non-spatial attributes. While this provides control, its relevance to perceiving pitch or loudness in complex, real-world sounds (the paper's motivation) is unclear. More importantly, the spatial perception tasks are simulated in rooms with simple source sounds ("alarm," "applause," "telephones"). A critical aspect of real-world spatial hearing is localization and tracking in the presence of diffuse background noise and significant reverberation. It is not clear from the paper or appendix if the benchmark tests performance under varying SNRs or reverberation times (RTs), which are standard parameters in auditory scene analysis.
4. Missing ablation on spatial realism: how do models perform if reverberation or stereo cues are removed?
5. Section 5 implies that STAR-Bench measures general “audio reasoning ability,” but all examples rely on synthetic or controlled sounds. Thus, the benchmark may not generalize to naturalistic auditory scenes (e.g., urban soundscapes, conversational dynamics). At least testing transfer performance would strengthen external validity.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: I don't have questions.
soundness: 3
strengths: - Establishes a clear and rigorous formalization of “audio 4D intelligence.”
- I agree on the problem formulation.
- The benchmark is novel and challenging. At the same time, it is useful.
- Benchmark design is systematic, combining physical simulation, human validation, and multiple subtask layers.
- Offers a quantitative diagnostic structure — separating perceptual, temporal, and spatial reasoning.
- Provides valuable model insights (e.g., Gemini’s bottleneck in perception vs. open-source deficits in reasoning and grounding).
summary: This paper introduces STAR-Bench, a benchmark designed to evaluate “4D Audio Intelligence”, defined as reasoning over sound dynamics in both time (1D) and 3D space. The authors argue that existing audio benchmarks (e.g., MMAU, MMAR) largely assess coarse, text-representable semantics rather than fine-grained perceptual reasoning. STAR-Bench aims to fill this gap through two complementary levels: a) Foundational Acoustic Perception — quantitative evaluation of six acoustic attributes (pitch, loudness, duration, azimuth, elevation, distance) under absolute and relative regimes.b) Holistic Spatio-Temporal Reasoning — tests requiring temporal segment reordering (continuous and discrete processes) and spatial reasoning (localization, multi-source relations, dynamic trajectories). The dataset combines synthetic and real-world audio, curated through a four-stage pipeline involving procedural synthesis, AI-assisted filtering, human annotation, and expert validation. The benchmark comprises 2,353 questions across these tasks.
weaknesses: - (Minor) The benchmark’s scale (2.3k samples) is relatively small compared to typical multimodal datasets.
- Evaluation focuses mainly on multiple-choice QA
- I don’t have many questions. The work is impressive. Some similar works have popped up lately but can be considered as parallel.
- Some points about Table 1 I am not sure how the ticks and crosses were made. What does Deep Reasoning meaning and why Spatial DR and Temporal DR are crossed for MMAU-Pro? I think spatial is also covered in MMAU-Pro? Can the authors provide more clarity?
- I have no questions. This looks like a well-rounded paper, and I vote for acceptance.


Decision: Accept (Poster)