AudioMarathon: A Comprehensive Benchmark for Long-Context Audio Understanding and Efficiency in Audio LLMs

## Abstract

Abstract Processing long-form audio is a major challenge for Large Audio Language models (LALMs). These models struggle with the quadratic cost of attention ( 𝒪 ​ ( N 2 ) \mathcal{O}(N^{2}) ) and with modeling long-range temporal dependencies. Existing audio benchmarks are built mostly from short clips and do not evaluate models in realistic long context settings. To address this gap, we introduce AudioMarathon , a benchmark designed to evaluate both understanding and inference efficiency on long-form audio. AudioMarathon provides a diverse set of tasks built upon three pillars: long-context audio inputs with durations ranging from 90.0 to 300.0 seconds, which correspond to encoded sequences of 2,250 to 7,500 audio tokens, respectively, full domain coverage across speech, sound, and music, and complex reasoning that requires multi-hop inference.
We evaluate state-of-the-art LALMs and observe clear performance drops as audio length grows.
We also study acceleration techniques and analyze the trade-offs of token pruning and KV cache eviction. The results show large gaps across current LALMs and highlight the need for better temporal reasoning and memory-efficient architectures.
We believe AudioMarathon will drive the audio and multimodal research community to develop more
advanced audio understanding models capable of solving complex audio tasks. Data & Dataset Card: huggingface.co/datasets/Hezep/AudioMarathon Code Repository: github.com/DabDans/AudioMarathon

## 1 Introduction

Multimodal Large Language Models (MLLMs), benefiting from the powerful understanding and reasoning abilities of large language models, have demonstrated remarkable capabilities in understanding and processing various data modalities (Alayrac et al., 2022; Li et al., 2023; Liu et al., 2023; Chen et al., 2024b; Kang et al., 2025; Zhang et al., 2024a; Wen et al., 2024; Li et al., 2025b). With audio being a key area of advancement, the ability to comprehend spoken language, environmental sounds, and music has opened up new frontiers for applications ranging from advanced speech recognition (Radford et al., 2023) to sophisticated audio-based reasoning (Borsos et al., 2023).

However, a significant and persistent challenge remains: the effective processing of long-form audio inputs. As the duration of audio increases, Large Audio Language Models (LALMs) face a dual challenge of escalating computational and memory costs (Vaswani et al., 2017), coupled with the inherent difficulty of capturing and modeling extended temporal dependencies (Beltagy et al., 2020; Zaheer et al., 2020). This bottleneck severely limits their practical application in real-world scenarios such as analyzing meetings, podcasts, or extended dialogues.

Figure: Figure 1: Overview of the AudioMarathon. AudioMarathon extends short audio clips to long-form audio with a diverse range of task categories, offering a comprehensive and practical assessment of audio intelligence in real-world scenarios.
Refer to caption: x1.png

A major factor hindering progress in this domain is the lack of comprehensive benchmarks designed to evaluate the long audio capabilities of LALMs rigorously. Existing audio benchmarks predominantly consist of short clips, typically only a few seconds long (Weck et al., 2024; Sakshi et al., 2024; Yang et al., 2024; Wang et al., 2024a). While valuable, these benchmarks fail to assess a model’s ability to maintain coherence, reason over long time spans, and manage computational resources efficiently when faced with minute-scale and even hour-scale audio inputs. This gap leaves a critical aspect of model performance unevaluated and obstructs the development of more robust and scalable audio understanding models.

To address this critical gap, we introduce AudioMarathon, a comprehensive audio benchmark meticulously designed to evaluate LALMs on long-context audio understanding and inference efficiency. AudioMarathon is built on three foundational pillars: ❶ Long-form Audio Context, featuring audio durations ranging from 90.0 to 300.0 seconds to simulate realistic scenarios; ❷ Full Domain Coverage, encompassing a diverse range of audio types including speech, environmental sounds, and music, as well as comprehensive task coverage spanning ten representative sub-tasks (ASR, SCR, SER, MC, ASC, SED, ER, SD, SAR, SGR) across Speech Context Understanding, Audio Scene Understanding, and Voice Characteristic Identification; and ❸ Complex Reasoning, incorporating multi-hop inference tasks that require models to connect disparate pieces of information across extended temporal windows.

Beyond just establishing a challenging new benchmark, this work also investigates crucial aspects of inference efficiency for long audio. We systematically evaluate a suite of state-of-the-art Audio LLMs (Chu et al., 2023; Abouelenin et al., 2025; Xu et al., 2025b), analyzing their performance degradation as input length increases.
Furthermore, we explore and quantify the effectiveness and trade-offs of various cost-reduction strategies, including inference-time Token pruning (Chen et al., 2024a; Zhang et al., 2024b; Wen et al., 2025b; d) and KV-cache eviction techniques (Li et al., 2024b).
Our findings reveal substantial performance gaps among current models in long-context scenarios and underscore the pressing need for improved temporal reasoning and memory-efficient processing.

By providing a unified and challenging evaluation suite, we aim to catalyze future research. We release AudioMarathon to the community to foster the development of the next generation of scalable, efficient, and robust LALMs capable of truly understanding the rich, continuous tapestry of the auditory world. Our main contributions are summarized as follows:

- •
AudioMarathon is presented as a comprehensive benchmark for long audio understanding, characterized by extended audio durations, diverse domain coverage, and complex reasoning tasks.
- •
Our work thoroughly evaluates state-of-the-art LALMs on AudioMarathon, revealing the specific challenges encountered when processing long audio inputs.
- •
In addition, we systematically analyze various inference efficiency techniques, such as token pruning and KV-cache eviction, to quantify their effectiveness and trade-offs.

## 2 AudioMarathon

Figure: Figure 2: The six-stage data pipeline for constructing the AudioMarathon
Refer to caption: x2.png

Figure: (a) Per dataset duration in AudioMarathon
Refer to caption: x3.png

Figure: Figure 4: Task composition of AudioMarathon by category
Refer to caption: x5.png

### 2.1 Overview

Existing audio benchmarks predominantly comprise short audio clips, often only a few seconds, thereby failing to capture the complexity of real-world scenarios such as meetings, podcasts, and extended dialogues.
To close the pronounced gap in benchmarks for long-form audio understanding, we present AudioMarathon, a comprehensive suite designed to evaluate the advanced capabilities of LALMs.
The construction of AudioMarathon follows a rigorous six-stage pipeline (Figure [2](https://arxiv.org/html/2510.07293v1#S2.F2)), ensuring diversity, difficulty, and high annotation quality. Figure [4](https://arxiv.org/html/2510.07293v1#S2.F4) and Table [5](https://arxiv.org/html/2510.07293v1#A1.T5) summarize the final composition across task categories, while Table [1](https://arxiv.org/html/2510.07293v1#S2.T1) and Table [2](https://arxiv.org/html/2510.07293v1#S2.T2) compare AudioMarathon against existing benchmarks.

### 2.2 Data Collection and Annotation

We adopt a rigorous multi-stage framework to construct AudioMarathon, detailed below.
Step 1. Source Selection. From 30 candidate datasets, we selected ten subsets according to task coverage and acoustic diversity. The tasks are grouped into three categories: Speech Context Understanding, Audio Scene Understanding, and Voice Characteristic Identification, ensuring both practical relevance and expert-level reasoning challenges.
Step 2. Merge Audio. Considering the characteristics of different tasks, we designed specific concatenation logic to merge individual clips into longer sequences.
Step 3. Tool Design. We developed a custom concatenation script to automate merging process. The tool flexibly supports generating sequences of variable length within the constraints of the source material, allowing to adapt sequence duration for different experimental settings.
Step 4. Data Processing. Each audio file was paired with a task-specific prompt and multiple-choice options. Option generation followed customized strategies tailored to each task, and the implementation has been released as open source. Model predictions were evaluated by two criteria: (i) exact match to a provided choice, or (ii) inclusion of the complete correct option without any extraneous information.
Step 5. Manual Verification. To ensure data quality, 10% (at least 20) samples per sub-dataset were randomly reviewed using the criteria detailed in Appendix [D](https://arxiv.org/html/2510.07293v1#A4). Any dataset failing inspection was reconstructed and revalidated until all checked samples passed.
Step 6. Benchmark Finalization. From the fully annotated QA pairs, 6,567 instances were selected to ensure balanced coverage of all 10 tasks and audio types. The concatenated files had durations from 90.0 to 300.0 seconds, balancing long-context evaluation with computational feasibility.

### 2.3 Comparison with Other Benchmarks

**Table 1: Comparison of audio datasets in terms of duration, size, average audio length, and domain coverage (speech, sound, and music).**
| Tasks | Duration | Size | Domain | Average audio<br>duration |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Speech | Sound | Music |  |  |  |  |
| MuChoMusic (Weck et al., 2024) | 5.1h | 1.1k | $\times$ | $\times$ | ✓ | 25.7 sec |
| BLAB (Ahia et al., 2025) | 833h | 1.6k | ✓ | $\times$ | $\times$ | 51.0 min |
| MMAR (Ma et al., 2025) | 5.5h | 1k | ✓ | ✓ | ✓ | 19.4 sec |
| MMSU (Wang et al., 2025a) | 9.73h | 5k | ✓ | $\times$ | $\times$ | 7.0 sec |
| MMAU (Sakshi et al., 2024) | 28.16h | 10k | ✓ | ✓ | ✓ | 10.1 sec |
| AIR-Bench (Yang et al., 2024) | 251.6h | 21k | ✓ | ✓ | ✓ | 35.2 sec |
| AudioBench (Wang et al., 2024a) | 400h | 100k | ✓ | ✓ | ✓ | 14.0 sec |
| AudioMarathon (ours) | 392h | 6.6k | ✓ | ✓ | ✓ | 212.8 sec |

Long Audio Understanding. Public audio benchmarks mostly use second-level clips (e.g., MMAR 19.4 s, MMAU 10.1 s, MMSU 7.01 s, AudioBench 14 s), which miss minute-scale complexity. BLAB includes long audio ($\sim$51.0 min) but is speech-centric. AudioMarathon targets realistic long-form use with durations ranging from 90.0 to 300.0 seconds and supports flexible duration control.

Full Domain Coverage. Audio spans three domains: speech, sound, and music. Most benchmarks cover one or two, limiting cross-domain robustness. Our proposed AudioMarathon covers all three with balanced sampling for comprehensive evaluation and cross-domain studies.

Multi-Hop Inference. We include an audio version of RACE generated via Text-to-Speech (Kokoro-82M (Nayak, 2025)), preserving RACE’s multi-hop reasoning while adding long-term acoustic dependencies—a stricter test of comprehension, memory, and reasoning.

**Table 2: Comparison of AudioMarathon with existing audio understanding and reasoning benchmarks across key properties and capabilities.**
| Capability | AudioMarathon | MuChoMusic | BLAB | MMAR | MMSU | MMAU | AIR-Bench | AudioBench |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Long Audio Understanding | ✓ | $\times$ | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |
| Full Domain Coverage | ✓ | $\times$ | $\times$ | ✓ | ✓ | ✓ | ✓ | $\times$ |
| Multi-Hop Inference | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ |
| Speaker attribute coverage | ✓ | $\times$ | $\times$ | $\times$ | ✓ | $\times$ | ✓ | ✓ |
| Contain deepfake audio | ✓ | $\times$ | $\times$ | $\times$ | $\times$ | $\times$ | ✓ | $\times$ |
| Complex task hierarchy | ✓ | $\times$ | $\times$ | ✓ | ✓ | ✓ | ✓ | $\times$ |
| Emotional and Semantic Understanding | ✓ | $\times$ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |

## 3 Experiments and Evaluations

Models.
We compare 16 recent Large Audio Language Models (LALMs), including ten open-source models and six closed-source models. The open-source models are Phi-4-Multimodal (Abdin et al., 2024), Qwen2.5-Omni-3B (Xu et al., 2025a), and Aero-1-Audio (Li et al., 2025a). Phi-4-Multimodal and Qwen2.5-Omni-3B are multi-modal large language models, while Aero-1-Audio is a compact audio language model designed for audio-centered tasks. The proprietary models are from the Gemini family: Gemini-2.5-Pro (Comanici et al., 2025a), Gemini-2.5-Flash (Comanici et al., 2025a), Gemini-2.0-Flash (Comanici et al., 2025b), and GPT-4o. All are multi-modal models, with Gemini-2.5-Flash and Gemini-2.0-Flash optimized for faster inference.

Evaluation Metrics.
Our evaluation considers two dimensions: task performance and inference efficiency. For task performance, we adopt standard metrics per task: F1-score for classification and MCQs, Word Accuracy Rate (WAR) for ASR, and macro F1-score for audio event detection to balance precision and recall across classes. Inference efficiency is assessed via latency and peak GPU memory usage. We also report speedup over a vanilla model.

Evaluation Setup.
To conduct the ASR task, evaluations are performed on test subset of LibriSpeech-long (Park et al., 2024) after filtering.
Except for ASR, all tasks are framed as MCQs with a single correct answer. SD and SGR provide two options, SAR provides five, and all other tasks use four.
For each instance, the model receives the full audio along with an instruction-following prompt presenting a question and four labeled options.
The model must select one option, and to mitigate positional bias, the option order is randomized.

**Table 3: Performance comparison of models on AudioMarathon across tasks, grouped into Speech Content Extraction (SER, SCR, ASR), Audio Classification (SED, MC, ASC), and Speaker Information Modeling (SD, ER, SAR, SGR). The Avg. column shows the mean score across all tasks. Best scores are in bold, second-best are underlined.**
| Models | Speech Content Extraction | Audio Classification | Speaker Information Modeling | Avg. |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SER | SCR | ASR | SED | MC | ASC | SD | ER | SAR | SGR |  |  |
| Open-source Audio LLMs |  |  |  |  |  |  |  |  |  |  |  |
| Phi-4-Multimodal | 18.4 | 69.3 | 92.7 | 55.1 | 46.7 | 23.4 | 26.4 | 27.3 | 26.6 | 91.1 | 47.7 |
| Qwen2.5-Omni-3B | 25.2 | 82.3 | 94.7 | 70.2 | 97.4 | 69.3 | 67.3 | 39.6 | 29.1 | 97.2 | 67.2 |
| Qwen2.5-Omni-7B | 26.3 | 85.1 | 98.1 | 78.4 | 100.0 | 72.2 | 72.3 | 53.4 | 21.4 | 98.0 | 70.5 |
| Audio-Flamingo-2 | 26.8 | 39.8 | 1.0 | 27.1 | 66.8 | 29.7 | 45.9 | 13.1 | 20.3 | 85.1 | 35.6 |
| Audio-Flamingo-3 | 21.7 | 78.9 | 94.3 | 59.5 | 97.0 | 54.1 | 33.7 | 54.3 | 40.7 | 96.2 | 63.0 |
| Gemma-3n-E2B-it | 22.5 | 51.6 | 91.3 | 50.2 | 56.8 | 28.2 | 35.1 | 15.2 | 12.2 | 91.6 | 45.5 |
| Gemma-3n-E4B-it | 19.0 | 56.9 | 93.2 | 50.2 | 71.9 | 31.7 | 35.9 | 18.9 | 21.8 | 93 | 49.3 |
| Voxtral-Mini-3B-2507 | 24.3 | 71.1 | 96.8 | 71.0 | 83.8 | 27.2 | 68.0 | 29.7 | 30.7 | 71.0 | 57.4 |
| Baichuan-Omni-1.5 | 12.4 | 11.2 | 86.5 | 45.7 | 52.0 | 25.8 | 49.2 | 18.9 | 10.2 | 81.5 | 39.3 |
| Aero-1-Audio | 17.9 | 56.6 | 43.7 | 55.0 | 83.9 | 39.9 | 33.7 | 32.0 | 17.8 | 47.5 | 42.8 |
| Close-source Audio LLMs |  |  |  |  |  |  |  |  |  |  |  |
| GPT-4o-Audio (Preview 2024-10-01) | 25.8 | 61.4 | 94.4 | 50.7 | 59.5 | 40.8 | 32.5 | 22.5 | 17.2 | 69.2 | 47.4 |
| GPT-4o-Audio (Preview 2024-12-17) | 25.7 | 60.2 | 94.7 | 51.2 | 67.6 | 41.9 | 30.8 | 21.8 | 19.9 | 73.1 | 48.7 |
| Gemini-2.0-Flash-Lite | 23.7 | 65.6 | 97.4 | 60.9 | 86.9 | 43.4 | 34.5 | 17.3 | 19.0 | 82.1 | 53.1 |
| Gemini-2.0-Flash | 30.9 | 71.8 | 96.4 | 68.1 | 88.5 | 54.1 | 32.1 | 20.1 | 39.2 | 93.1 | 59.4 |
| Gemini-2.5-Flash-Lite | 30.3 | 64.0 | 96.5 | 68.0 | 64.8 | 36.8 | 33.9 | 14.6 | 19.6 | 77.9 | 50.6 |
| Gemini-2.5-Flash | 28.1 | 83.6 | 96.8 | 69.2 | 79.3 | 40.8 | 33.1 | 31.9 | 34.3 | 99.3 | 59.6 |
| Human Evaluation | 45.1 | 88.1 | – | 96.2 | 100.0 | 100.0 | 100.0 | 90.8 | 71.4 | 97.0 | 87.6 |

**Table 4: Performance comparison of three open-sourced LALMs across token pruning methods and ratios on AudioMarathon tasks, grouped into Speech Content Extraction (SER, SCR, ASR), Audio Classification (SED, MC, ASC), and Speaker Recognition (SD, ER, SAR, SGR). F1-score (0-100) is the primary metric, except for ASR, where Word Accuracy Rate (WAR) is used. The Avg. column shows the mean score across available tasks. Best scores within each pruning ratio are in bold.**
| Method | Model | Speech Content Extraction | Audio Classification | Speaker Recognition | Avg. |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SER | SCR | ASR | SED | MC | ASC | SD | ER | SAR | SGR |  |  |  |
| Vanilla |  |  |  |  |  |  |  |  |  |  |  |  |
|  | Phi-4-Multimodal | 18.4 | 69.3 | 92.7 | 55.1 | 46.7 | 23.4 | 26.4 | 27.3 | 26.6 | 91.1 | 47.7 |
|  | Aero-1-Audio | 17.9 | 56.6 | 43.7 | 55.0 | 83.9 | 39.9 | 33.7 | 32.0 | 17.8 | 47.5 | 42.8 |
|  | Qwen2.5-Omni-3B | 25.2 | 82.3 | 94.7 | 70.2 | 100.0 | 69.3 | 67.3 | 39.6 | 29.1 | 97.2 | 67.5 |
| Light Token Pruning $\mathbf{\mathcolor{ForestGreen}{(\downarrow 30\%)}}$ |  |  |  |  |  |  |  |  |  |  |  |  |
| Random | Phi-4-multimodal | 18.4 | 67.5 | 49.1 | 31.4 | 39.8 | 30.2 | 31.3 | 31.0 | 24.5 | 93.6 | 41.7 |
| Aero-1-Audio | 15.9 | 53.9 | 43.3 | 56.8 | 79.4 | 40.2 | 34.0 | 32.4 | 10.0 | 38.8 | 40.5 |  |
| Qwen2.5-Omni-3B | 26.5 | 90.3 | 88.4 | 71.1 | 97.5 | 69.7 | 72.0 | 38.4 | 28.6 | 95.7 | 67.8 |  |
| FastV | Phi-4-multimodal | 18.3 | 64.0 | 43.9 | 33.2 | 40.6 | 29.6 | 44.0 | 29.1 | 25.7 | 92.9 | 42.1 |
| Aero-1-Audio | 19.7 | 57.0 | 37.5 | 57.0 | 78.8 | 41.0 | 42.1 | 32.2 | 9.2 | 39.2 | 41.4 |  |
| Qwen2.5-Omni-3B | 18.7 | 68.2 | 76.3 | 61.3 | 98.4 | 57.2 | 38.5 | 31.1 | 17.3 | 97.5 | 56.5 |  |
| DART | Phi-4-multimodal | 16.8 | 67.6 | 57.2 | 54.5 | 46.1 | 31.8 | 23.1 | 28.6 | 27.1 | 91.6 | 44.4 |
| Aero-1-Audio | 20.2 | 57.0 | 16.4 | 56.3 | 78.8 | 41.0 | 34.0 | 32.2 | 9.2 | 39.5 | 38.5 |  |
| Qwen2.5-Omni-3B | 23.2 | 74.2 | 81.4 | 73.1 | 97.6 | 72.5 | 42.2 | 37.1 | 23.0 | 48.7 | 57.3 |  |
| Frame (Ours) | Phi-4-multimodal | 17.7 | 64.4 | 63.4 | 31.4 | 32.6 | 29.0 | 30.6 | 31.0 | 27.4 | 92.4 | 42.0 |
| Aero-1-Audio | 15.6 | 53.7 | 43.4 | 54.3 | 82.5 | 39.8 | 34.4 | 32.1 | 8.1 | 37.3 | 40.1 |  |
| Qwen2.5-Omni-3B | 26.8 | 80.9 | 92.2 | 70.5 | 98.5 | 70.2 | 65.0 | 36.4 | 31.4 | 96.7 | 66.9 |  |
| Medium Token Pruning $\mathbf{\mathcolor{ForestGreen}{(\downarrow 60\%)}}$ |  |  |  |  |  |  |  |  |  |  |  |  |
| Random | Phi-4-multimodal | 18.7 | 61.6 | 7.9 | 30.3 | 27.4 | 30.6 | 36.8 | 29.4 | 20.5 | 91.2 | 35.4 |
| Aero-1-Audio | 12.1 | 49.7 | 34.9 | 54.6 | 78.2 | 41.3 | 42.5 | 34.5 | 8.8 | 34.0 | 39.1 |  |
| Qwen2.5-Omni-3B | 24.2 | 75.3 | 59.7 | 68.7 | 95.8 | 68.3 | 66.6 | 37.9 | 27.2 | 93.5 | 61.7 |  |
| FastV | Phi-4-multimodal | 26.1 | 52.8 | 0.0 | 32.5 | 28.2 | 30.3 | 25.6 | 28.0 | 22.2 | 89.4 | 33.5 |
| Aero-1-Audio | 20.3 | 54.2 | 30.4 | 58.0 | 80.2 | 44.5 | 34.2 | 33.4 | 9.1 | 34.5 | 39.9 |  |
| Qwen2.5-Omni-3B | 18.0 | 63.8 | 39.2 | 60.5 | 97.5 | 57.8 | 44.0 | 28.6 | 17.1 | 95.3 | 52.2 |  |
| DART | Phi-4-multimodal | 18.0 | 61.1 | 23.7 | 53.9 | 44.8 | 25.4 | 26.3 | 29.4 | 24.4 | 88.0 | 39.5 |
| Aero-1-Audio | 20.3 | 54.2 | 14.4 | 58.2 | 80.6 | 44.5 | 34.0 | 33.4 | 9.1 | 34.5 | 38.3 |  |
| Qwen2.5-Omni-3B | 23.1 | 64.6 | 62.8 | 71.9 | 99.1 | 73.4 | 38.3 | 37.6 | 28.1 | 46.0 | 54.5 |  |
| Frame (Ours) | Phi-4-multimodal | 23.8 | 59.0 | 23.3 | 31.1 | 28.5 | 30.0 | 20.6 | 30.1 | 22.3 | 87.8 | 35.6 |
| Aero-1-Audio | 14.0 | 51.7 | 42.5 | 56.4 | 80.9 | 41.1 | 35.0 | 34.5 | 9.1 | 33.3 | 39.9 |  |
| Qwen2.5-Omni-3B | 25.6 | 75.3 | 82.2 | 69.0 | 100.0 | 68.3 | 65.7 | 38.6 | 28.3 | 91.0 | 64.4 |  |
| Extreme Token Pruning $\mathbf{\mathcolor{ForestGreen}{(\downarrow 90\%)}}$ |  |  |  |  |  |  |  |  |  |  |  |  |
| Random | Phi-4-multimodal | 18.7 | 35.3 | 0.0 | 29.3 | 20.6 | 29.6 | 41.9 | 33.4 | 11.1 | 67.5 | 28.7 |
| Aero-1-Audio | 10.1 | 47.6 | 5.1 | 43.4 | 70.3 | 44.9 | 47.5 | 32.2 | 14.6 | 33.3 | 34.9 |  |
| Qwen2.5-Omni-3B | 24.0 | 58.1 | 0.0 | 65.9 | 97.6 | 60.0 | 54.7 | 41.8 | 17.1 | 84.3 | 50.4 |  |
| FastV | Phi-4-multimodal | 23.4 | 43.0 | 0.0 | 27.6 | 30.1 | 29.3 | 46.3 | 24.9 | 17.3 | 82.9 | 32.5 |
| Aero-1-Audio | 18.0 | 50.6 | 8.3 | 55.8 | 69.0 | 45.0 | 38.2 | 26.3 | 16.8 | 33.6 | 36.2 |  |
| Qwen2.5-Omni-3B | 16.8 | 54.9 | 3.5 | 65.2 | 95.9 | 55.9 | 49.5 | 32.7 | 14.8 | 86.5 | 47.6 |  |
| DART | Phi-4-multimodal | 16.8 | 49.3 | 0.0 | 52.0 | 40.2 | 24.4 | 31.9 | 27.4 | 18.6 | 77.2 | 33.8 |
| Aero-1-Audio | 18.0 | 50.6 | 0.0 | 55.8 | 69.0 | 45.0 | 38.1 | 26.3 | 16.8 | 33.6 | 35.3 |  |
| Qwen2.5-Omni-3B | 17.3 | 54.1 | 62.9 | 66.8 | 99.1 | 69.3 | 25.6 | 42.6 | 22.1 | 52.8 | 51.3 |  |
| Frame (Ours) | Phi-4-multimodal | 24.6 | 36.8 | 0.0 | 28.4 | 25.0 | 28.1 | 34.8 | 30.1 | 12.1 | 66.6 | 28.7 |
| Aero-1-Audio | 9.7 | 48.9 | 3.1 | 43.8 | 73.0 | 44.0 | 54.2 | 33.2 | 16.1 | 33.3 | 35.9 |  |
| Qwen2.5-Omni-3B | 22.8 | 58.2 | 0.0 | 64.5 | 95.0 | 60.9 | 51.9 | 41.1 | 18.1 | 87.0 | 50.0 |  |

## 4 Efficiency Optimization for LALMs

Processing extended audio sequences poses significant computational challenges for LALMs. A single 5-minute audio input can generate thousands of tokens, leading to quadratic memory growth and prohibitive inference latency (as shown in Table [8](https://arxiv.org/html/2510.07293v1#A3.T8) of Appendix [C](https://arxiv.org/html/2510.07293v1#A3)).
To address these bottlenecks, we systematically evaluate two complementary efficiency optimization strategies: Token pruning (Liu et al., 2025) during the prefilling stage and KV-cache eviction(^1^11[https://github.com/NVIDIA/kvpress](https://github.com/NVIDIA/kvpress)) during the decoding stage.

### 4.1 Token Pruning and KV Cache Eviction

Processing long-form audio sequences poses substantial memory and latency challenges for LALMs. One-minute audio input is embedded into 1500 tokens, requiring massive KV-cache storage and significantly slow decoding, thus making deployment impractical without compression.
To address these bottlenecks, numerous approaches have emerged that directly reduce the number of tokens to improve inference efficiency.
We evaluate four token pruning methods and four KV cache eviction strategies on our long-audio benchmark.
Experiments are conducted on three open-source LALMs, including Qwen2.5-Omni-3B, Aero-1-Audio, and Phi-4-Multimodal.

Figure: Figure 5: Comparisons of latency and performance trade-off for the Qwen2.5-Omni-3B model under different token pruning strategies across four representative datasets. Frame consistently outperforms other methods across different latency constraints.
Refer to caption: x6.png

Figure: Figure 6: Acceleration effects across token pruning strategies for the Qwen2.5-Omni-3B model under various token pruning strategies across four datasets.
Refer to caption: x7.png

Token pruning. We compare four token pruning strategies on AudioMarathon.
The baseline, Random pruning, discards tokens uniformly at random.
FastV (Chen et al., 2024a) removes low-attention tokens, and DART (Wen et al., 2025b) applies redundancy-guided selection by discarding similar tokens.
However, due to the strongly sequential nature of acoustic signals, naive or purely attention-based pruning can inadvertently remove brief phonetic cues or transient events, leading to degraded recognition.
Unlike vision models, where redundancy often arises from spatial or semantic similarity, audio token redundancy primarily manifests as smooth temporal continuity.
Therefore, we additionally design Frame as a time-aligned token pruning strategy to preserve rare or short-lived acoustic events that other methods may discard, making it a scheme tailored to audio characteristics.

KV-cache eviction. We evaluate four eviction strategies under compression ratios of 30%, 60%, and 90%.
The baseline, Random eviction, uniformly removes cache entries, providing a lower bound on performance under cache pressure.
KNorm (Devoto et al., 2024) evicts tokens according to the L2 norm of their key vectors, based on the intuition that smaller norms contribute less to attention.
TOVA (Oren et al., 2024) greedily discards tokens with minimal attention from the latest query by averaging attention weights across heads at each decoding step.
Finally, SnapKV (Li et al., 2024b) retains high-attention tokens along with their neighbors using cumulative-attention scoring and 1D pooling-based clustering, preserving local semantic coherence while enabling efficient compression.

Figure: Figure 7: Peak GPU memory usage count for KV cache eviction policies applied to the Qwen2.5-Omni-3B model across four datasets, illustrating memory compression benefits during the prefilling stage for long-context audio inference.
Refer to caption: x8.png

## 5 Results and Discussions

Our proposed AudioMarathon provides a realism-oriented evaluation framework for LALMs, focusing on minute-scale recordings, diverse audio domains, and complex reasoning.
Results in Table [3](https://arxiv.org/html/2510.07293v1#S3.T3) reveal clear performance stratification among the 16 evaluated models.
(i) The best-performing model, Qwen2.5-Omni-7B, achieves an average F1-score of 70.5, whereas most open-source models cluster between 30 and 60, highlighting a substantial performance gap.
(ii) Closed-source models perform unevenly: all fail on long-audio emotion recognition and authenticity detection, with Gemini-2.5-Flash being the only one to exceed 30 in emotion recognition.
Even for authenticity detection, Qwen2.5-Omni reaches 72.3, while every closed-source model remains below 35.
(iii) Human evaluation yields an average F1-score of 87.6, clearly surpassing even the strongest models. The gap is most pronounced in Speaker Information Modeling tasks such as speech emotion recognition (SER) and speaker-based entity recognition (ER), where human performance (87.6) remains far above model scores (generally below 65). This pronounced weakness directly reflects the challenges of entity tracking and temporal reasoning discussed in the introduction, highlighting a concrete target for future improvement.
In summary, leading LALMs perform competitively on narrow, single-domain tasks but still struggle with long-form speech understanding.
These weaknesses point to priorities for future work: (i) richer pretraining on long-form and multi-source audio, (ii) improved multi-scale and spatially aware representations, and (iii) benchmarks that explicitly assess long-context extraction, audio scene classification, and efficiency under realistic conditions.

What Challenges Do LALMs Face?
Long-form Speech Entity Recognition (SER) and Speaker-Age Recognition are the most challenging tasks in our benchmark.
The top model achieves an F1-score just above 30 on SER, with only Audio-Flamingo-3 exceeding 40, indicating a substantial capability gap among current LALMs.
Memory consumption and inference latency pose additional challenges.
The Figure [7](https://arxiv.org/html/2510.07293v1#S4.F7) shows that cache eviction reduces peak memory during prefilling, though GPU usage remains high.
Preserving the first output token maintains performance for most MCQs.
Figure [6](https://arxiv.org/html/2510.07293v1#S4.F6) indicates that processing all long-audio tokens at the second decoder layer is computationally expensive; reducing tokens to 10% cuts processing time to 56% of the baseline, achieving a 1.8$\times$ speedup.
Unlike vision tokens, audio tokens encode strong temporal dependencies. Aggressive eviction can disrupt temporal coherence, especially in ASR, where every phoneme matters.
Figure [5](https://arxiv.org/html/2510.07293v1#S4.F5) shows that improper pruning can produce repetitive tokens, increasing latency and lowering word accuracy.
Overall, task-aware audio token compression provides significant runtime and memory savings and is essential for scaling long-audio LLM inference.
Semantic vs. Acoustic: Mapping Capabilities in Long-Audio Tasks.

Figure: Figure 8: Average F1-scores across the three main task categories: Speech Content Extraction, Audio Classification, and Speaker Information Modeling, underscoring the need for enhanced temporal reasoning in extended audio contexts.
Refer to caption: x9.png

Recent LALMs integrate acoustic and linguistic features within a single end-to-end model, enabling joint learning of cross-modal dependencies (Peng et al., 2024).
In AudioMarathon, ASR, Speech Content Reasoning (SCR), and SER are considered semantically sensitive tasks, while the remaining seven tasks are acoustically sensitive.
Figure [8](https://arxiv.org/html/2510.07293v1#S5.F8) shows that all closed-source models achieve around 60 F1-score on semantically sensitive tasks, reflecting their strength in extracting long-form audio content.
The strongest model, Qwen2.5-Omni-7B, reaches 83.5, and the top four LALMs score above 70 on audio classification, indicating extensive training on classification tasks.
In contrast, all models underperform on speaker-related tasks, failing to exceed a 65 F1-score, suggesting that speaker information modeling remains under-emphasized in current LALMs development.

Figure: Figure 9: Performance comparison between Qwen2.5-Omni-3B, Phi-4-Mutimodal and Aero-1-Audio on six-degree capability under varying token pruning ratios.
Refer to caption: x10.png

How do token pruning methods affect performance in LALMs?
We evaluate six-degree capabilities: (i) Speech Understanding (mean of SER and SCR), (ii) Speaker Recognition (mean of SAR and SGR), (iii) Emotion Recognition (ER), (iv) Authenticity Recognition (SD), (v) Automatic Speech Recognition (ASR), and (vi) Multi-Domain Classification (mean of SED and MC), as shown in Figure  [9](https://arxiv.org/html/2510.07293v1#S5.F9). For audio token pruning, the reported score is the maximum F1-score achieved across tested pruning settings.
Qwen2.5-Omni-3B benefits from token pruning, with Speech Understanding F1-score rising from 53.8 to 58.6, Multi-Domain Classification improving modestly (79.8 → 81.9), and Authenticity Recognition gaining about five points (67.3 → 72.0).
Aero-1-Audio is highly pruning-sensitive: its Speech Understanding F1-score loses roughly 5 points under light Frame pruning and over 15 under extreme settings, while DART exacerbates the degradation, resulting in total losses exceeding 25 points under the most aggressive configuration. Multi-Domain Classification remains comparatively robust, with only modest decreases under Frame. Phi-4-Multimodal shows a similar pattern: F1-score of Speech Understanding experiences moderate decline under light Frame pruning and severe degradation under extreme or DART-based pruning, whereas Multi-Domain Classification is better preserved with Frame than with DART.

Table [4](https://arxiv.org/html/2510.07293v1#S3.T4) highlights a task-dependent sensitivity gradient: temporally fine-grained tasks (ASR, Speech Understanding) degrade sharply when selective or attention-driven pruning removes temporally unique phonetic cues, while more global classification tasks (e.g., music detection) remain resilient even under aggressive compression. Mechanistically, DART’s redundancy-focused selection can discard rare temporal segments; Frame’s uniform windowed subsampling preserves coverage; FastV’s attention-based policy can misalign with linear temporal progression. Frame consistently attains the strongest speech-sensitive F1 scores across models and pruning ratios. These results argue for task-aware token pruning: redundancy-based strategies suit over-represented signals, Frame is a safe default for speech-centric workloads, and attention-driven pruning entails a tangible risk of temporal coherence disruption.

## 6 Related Works

Large Audio Language Models.
The development of LALMs follows the broader shift toward multimodal language processing. Early audio models combined ASR and Text-to-Speech with text-based LLMs for audio-to-text tasks, but they suffered from error propagation and weak cross-modal fusion (Ngiam et al., 2011; Hinton et al., 2012; Wang et al., 2017).
Self supervised speech representations, such as wav2vec 2.0 (Baevski et al., 2020) and HuBERT (Hsu et al., 2021), drove major progress and enabled models like Whisper (Radford et al., 2023) and SpeechGPT (Zhang et al., 2023a). Recent instruction-tuned Audio LLMs, such as Phi-4-multimodal (Abouelenin et al., 2025), Freeze-Omni (Wang et al., 2024b), and Qwen2.5-Omni (Xu et al., 2025b), unify audio and language within one framework and support tasks that span ASR, audio question answering, and audio understanding. As context windows grow (Liu et al., 2025), these models are also moving toward longer audio inputs, with some reporting support for hours of audio.

Audio LLM Benchmarks.
Benchmarking has evolved from task-specific datasets to broader frameworks that test multimodal and instruction-following abilities. Early datasets such as AudioSet (Gemmeke et al., 2017), LibriSpeech (Panayotov et al., 2015), ESC 50 (Piczak, 2015), and FSD50K (Fonseca et al., 2021) focused on classification or ASR. SUPERB (Yang et al., 2021) expanded speech evaluation with a broader task set. For audio language understanding, Clotho QA (Drossos et al., 2020) and AudioCaps (Kim et al., 2019) introduced question answering and captioning. More recent datasets, such as MMAU (Sakshi et al., 2024) and AIR Bench (Yang et al., 2024), target instruction following and tri modal reasoning. Despite progress, few benchmarks directly test long audio comprehension or the efficiency of long sequence processing.

Token Compression.
Transformer-based models face memory and compute limits with long context and multimodal inputs. Two practical directions are KV cache eviction and token pruning (Liu et al., 2025; Wen et al., 2025a; Xiong et al., 2025; Yang et al., 2025; Chen et al., 2025; Wang et al., 2025b). For KV cache eviction, SnapKV (Li et al., 2024b) clusters high attention tokens and stores centroids, H2O (Zhang et al., 2023b) balances recent and salient tokens, and StreamingLLM (Xiao et al., 2023) uses fixed attention sinks and a sliding window for unbounded generation. In vision language models, token pruning reduces redundant visual tokens through architectural methods, such as Q-Former context tokens (Li et al., 2024a), and through inference time methods, such as Token Merging (Bolya et al., 2022), FastV (Chen et al., 2024a), SparseVLM (Zhang et al., 2024b), and DART (Wen et al., 2025b). While these methods are effective for vision or text tokens, research on audio token compression remains limited, and it is unknown how well these methods transfer to the audio modality.

## 7 Conclusion

We present AudioMarathon, a comprehensive benchmark for LALMs that targets minute-scale speech, sound, and music inputs, spanning 10 representative audio tasks. Through these long-form scenarios, AudioMarathon exposes fundamental challenges such as long-range dependency, temporal continuity, and source confusion. While existing LALMs perform well on short-range tasks like classification and reasoning, they struggle with long-span speech understanding and speaker analysis, revealing limitations in consistency and entity tracking, which exhibit a significant gap compared to human performance, providing a potential direction for future research. Moreover, the field of large audio models lacks attention to the efficiency of audio encoders, which, in practice, leads to substantial redundancy in audio tokens. Ultimately, AudioMarathon provides a foundation for developing robust and efficient long-audio modeling.

## Ethics Statement

Our research introduces the AudioMarathon to advance long-form audio understanding and inference efficiency in Large Audio Language Models (LALMs). We acknowledge the dual-use potential of this technology, which could be misused for generating deepfake audio or eroding privacy. We justify its public disclosure as a means to foster robustness and safety through transparent benchmarking and to highlight model limitations for proactive risk mitigation. We encourage future work to expand this effort with responsible practices across diverse languages and contexts. We hereby affirm that this work was conducted in strict compliance with academic ethics, with the primary goal of steering technological progress toward beneficial ends; any misuse of this research for unlawful or unethical purposes is unequivocally contrary to our principles.

## Reproducibility Statement

We are committed to ensuring the reproducibility of our results. To this end, we provide comprehensive details about our experimental setup and datasets. Specifically, all hyperparameters, model descriptions, pruning strategies, and evaluation protocols are specified in the main text. Additional analysis, including random baseline results, error analysis, encoding granularity of LALMs, and model details, is presented in the Appendix [E](https://arxiv.org/html/2510.07293v1#A5). We describe all datasets used in our benchmark (AudioMarathon) in Appendix [D](https://arxiv.org/html/2510.07293v1#A4), including their construction and task definitions. Detailed motivation and implementation of pruning at the second layer are provided in Appendix [A](https://arxiv.org/html/2510.07293v1#A1), where we explain its design as an early-stage compression mechanism for audio tokens. We report the performance of different pruning methods, random baselines, acceleration consistency, and model-specific results (e.g., Qwen2.5-Omni-3B) in the appendix for full transparency. Appendix [B](https://arxiv.org/html/2510.07293v1#A2) provides descriptions of all baseline models, including their architectures, training strategies, and modality support. Together, these efforts ensure that all experiments can be independently reproduced.

## References

- Abdin et al. (2024)
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffmann, et al.
Phi-4 technical report.
*arXiv preprint arXiv:2412.08905*, 2024.
- Abouelenin et al. (2025)
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al.
Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras.
*arXiv preprint arXiv:2503.01743*, 2025.
- Ahia et al. (2025)
Orevaoghene Ahia, Martijn Bartelds, Kabir Ahuja, Hila Gonen, Valentin Hofmann, Siddhant Arora, Shuyue Stella Li, Vishal Puttagunta, Mofetoluwa Adeyemi, Charishma Buchireddy, et al.
Blab: Brutally long audio bench.
*arXiv preprint arXiv:2505.03054*, 2025.
- Alayrac et al. (2022)
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al.
Flamingo: a visual language model for few-shot learning.
*Advances in neural information processing systems*, 35:23716–23736, 2022.
- Baevski et al. (2020)
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli.
wav2vec 2.0: A framework for self-supervised learning of speech representations.
*Advances in neural information processing systems*, 33:12449–12460, 2020.
- Beltagy et al. (2020)
Iz Beltagy, Matthew E Peters, and Arman Cohan.
Longformer: The long-document transformer.
*arXiv preprint arXiv:2004.05150*, 2020.
- Bolya et al. (2022)
Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman.
Token merging: Your vit but faster.
*arXiv preprint arXiv:2210.09461*, 2022.
- Borsos et al. (2023)
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Dominik Roblek, Olivier Teboul, David Grangier, Marco Tagliasacchi, et al.
Audiolm: a language modeling approach to audio generation.
*IEEE/ACM transactions on audio, speech, and language processing*, 31:2523–2533, 2023.
- Chen et al. (2025)
Junjie Chen, Xuyang Liu, Zichen Wen, Yiyu Wang, Siteng Huang, and Honggang Chen.
Variation-aware vision token dropping for faster large vision-language models.
*arXiv preprint arXiv:2509.01552*, 2025.
- Chen et al. (2024a)
Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang.
An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models.
In *European Conference on Computer Vision*, pp. 19–35. Springer, 2024a.
- Chen et al. (2024b)
Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, et al.
Mj-bench: Is your multimodal reward model really a good judge for text-to-image generation?
*arXiv preprint arXiv:2407.04842*, 2024b.
- Chu et al. (2023)
Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou.
Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models.
*arXiv preprint arXiv:2311.07919*, 2023.
- Chu et al. (2024)
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, et al.
Qwen2-audio technical report.
*arXiv preprint arXiv:2407.10759*, 2024.
- Comanici et al. (2025a)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025a.
- Comanici et al. (2025b)
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
*arXiv preprint arXiv:2507.06261*, 2025b.
- Devoto et al. (2024)
Alessio Devoto, Yu Zhao, Simone Scardapane, and Pasquale Minervini.
A simple and effective $l_2$ norm-based strategy for kv cache compression.
*arXiv preprint arXiv:2406.11430*, 2024.
- Drossos et al. (2020)
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen.
Clotho: An audio captioning dataset.
In *ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 736–740. IEEE, 2020.
- Fonseca et al. (2021)
Eduardo Fonseca, Xavier Favory, Jordi Pons, Frederic Font, and Xavier Serra.
Fsd50k: an open dataset of human-labeled sound events.
*IEEE/ACM Transactions on Audio, Speech, and Language Processing*, 30:829–852, 2021.
- Gemmeke et al. (2017)
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter.
Audio set: An ontology and human-labeled dataset for audio events.
In *2017 IEEE international conference on acoustics, speech and signal processing (ICASSP)*, pp. 776–780. IEEE, 2017.
- Ghosh et al. (2025)
Sreyan Ghosh, Zhifeng Kong, Sonal Kumar, S Sakshi, Jaehyeon Kim, Wei Ping, Rafael Valle, Dinesh Manocha, and Bryan Catanzaro.
Audio flamingo 2: An audio-language model with long-audio understanding and expert reasoning abilities.
*arXiv preprint arXiv:2503.03983*, 2025.
- Goel et al. (2025)
Arushi Goel, Sreyan Ghosh, Jaehyeon Kim, Sonal Kumar, Zhifeng Kong, Sang-gil Lee, Chao-Han Huck Yang, Ramani Duraiswami, Dinesh Manocha, Rafael Valle, et al.
Audio flamingo 3: Advancing audio intelligence with fully open large audio language models.
*arXiv preprint arXiv:2507.08128*, 2025.
- Hechmi et al. (2021)
Khaled Hechmi, Trung Ngo Trong, Ville Hautamäki, and Tomi Kinnunen.
Voxceleb enrichment for age and gender recognition.
In *2021 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)*, pp. 687–693. IEEE, 2021.
- Heittola et al. (2019)
Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen.
TAU urban acoustic scenes 2019, development dataset.
Zenodo, March 2019.
URL [https://doi.org/10.5281/zenodo.2589280](https://doi.org/10.5281/zenodo.2589280).
Version 1.0.
- Hinton et al. (2012)
Geoffrey Hinton, Li Deng, Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al.
Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups.
*IEEE Signal processing magazine*, 29(6):82–97, 2012.
- Hsu et al. (2021)
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed.
Hubert: Self-supervised speech representation learning by masked prediction of hidden units.
*IEEE/ACM transactions on audio, speech, and language processing*, 29:3451–3460, 2021.
- Hurst et al. (2024)
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al.
Gpt-4o system card.
*arXiv preprint arXiv:2410.21276*, 2024.
- Jin et al. (2025)
Xiangqi Jin, Yuxuan Wang, Yifeng Gao, Zichen Wen, Biqing Qi, Dongrui Liu, and Linfeng Zhang.
Thinking inside the mask: In-place prompting in diffusion llms.
*arXiv preprint arXiv:2508.10736*, 2025.
- Kang et al. (2025)
Hengrui Kang, Siwei Wen, Zichen Wen, Junyan Ye, Weijia Li, Peilin Feng, Baichuan Zhou, Bin Wang, Dahua Lin, Linfeng Zhang, et al.
Legion: Learning to ground and explain for synthetic image detection.
*arXiv preprint arXiv:2503.15264*, 2025.
- Kim et al. (2019)
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim.
Audiocaps: Generating captions for audios in the wild.
In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)*, pp. 119–132, 2019.
- Lai et al. (2017)
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.
Race: Large-scale reading comprehension dataset from examinations.
*arXiv preprint arXiv:1704.04683*, 2017.
- Li et al. (2025a)
Bo Li, Chen Change Loy, Pu Fanyi, Yang Jingkang, Zhang Kaichen, Hu Kairui, Thang Luu Minh, Trung Nguyen Quang, Cong Pham Ba, Liu Shuai, Wang Yezhen, and Liu Ziwei.
Aero: Audio-enhanced large language models.
2025a.
URL [https://www.lmms-lab.com/posts/aero_audio/](https://www.lmms-lab.com/posts/aero_audio/).
- Li et al. (2023)
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
In *International conference on machine learning*, pp. 19730–19742. PMLR, 2023.
- Li et al. (2025b)
Weiya Li, Junjie Chen, Bei Li, Boyang Liu, Zichen Wen, Nuanqiao Shan, Xiaoqian Liu, Anping Liu, Huajie Liu, Youyan Wang, et al.
Tactic: Translation agents with cognitive-theoretic interactive collaboration.
*arXiv preprint arXiv:2506.08403*, 2025b.
- Li et al. (2025c)
Yadong Li, Jun Liu, Tao Zhang, Song Chen, Tianpeng Li, Zehuan Li, Lijun Liu, Lingfeng Ming, Guosheng Dong, Da Pan, et al.
Baichuan-omni-1.5 technical report.
*arXiv preprint arXiv:2501.15368*, 2025c.
- Li et al. (2024a)
Yanwei Li, Chengyao Wang, and Jiaya Jia.
LLaMA-VID: An image is worth 2 tokens in large language models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, 2024a.
- Li et al. (2024b)
Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen.
Snapkv: Llm knows what you are looking for before generation.
*Advances in Neural Information Processing Systems*, 37:22947–22970, 2024b.
- Liu et al. (2023)
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning.
*Advances in neural information processing systems*, 36:34892–34916, 2023.
- Liu et al. (2025)
Xuyang Liu, Zichen Wen, Shaobo Wang, Junjie Chen, Zhishan Tao, Yubo Wang, Xiangqi Jin, Chang Zou, Yiyu Wang, Chenfei Liao, et al.
Shifting ai efficiency from model-centric to data-centric compression.
*arXiv preprint arXiv:2505.19147*, 2025.
- Ma et al. (2025)
Ziyang Ma, Yinghao Ma, Yanqiao Zhu, Chen Yang, Yi-Wen Chao, Ruiyang Xu, Wenxi Chen, Yuanzhe Chen, Zhuo Chen, Jian Cong, et al.
Mmar: A challenging benchmark for deep reasoning in speech, audio, music, and their mix.
*arXiv preprint arXiv:2505.13032*, 2025.
- Nagrani et al. (2017)
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman.
Voxceleb: a large-scale speaker identification dataset.
*arXiv preprint arXiv:1706.08612*, 2017.
- Nayak (2025)
Aryan Nayak.
Kokoro: An accessible text-to-speech application for visually impaired students.
*No, this is the first time. I’m ever publishing a research paper*, 2025.
- Ngiam et al. (2011)
Jiquan Ngiam, Aditya Khosla, Mingyu Kim, Juhan Nam, Honglak Lee, Andrew Y Ng, et al.
Multimodal deep learning.
In *ICML*, volume 11, pp. 689–696, 2011.
- Nie et al. (2025)
Shen Nie, Fengqi Zhu, Zebin You, Xiaolu Zhang, Jingyang Ou, Jun Hu, Jun Zhou, Yankai Lin, Ji-Rong Wen, and Chongxuan Li.
Large language diffusion models.
*arXiv preprint arXiv:2502.09992*, 2025.
- Oren et al. (2024)
Matanel Oren, Michael Hassid, Nir Yarden, Yossi Adi, and Roy Schwartz.
Transformers are multi-state rnns.
*arXiv preprint arXiv:2401.06104*, 2024.
- Panayotov et al. (2015)
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur.
Librispeech: an asr corpus based on public domain audio books.
In *2015 IEEE international conference on acoustics, speech and signal processing (ICASSP)*, pp. 5206–5210. IEEE, 2015.
- Park et al. (2024)
Se Jin Park, Julian Salazar, Aren Jansen, Keisuke Kinoshita, Yong Man Ro, and R. J. Skerry-Ryan.
Long-form speech generation with spoken language models.
*CoRR*, abs/2412.18603, 2024.
- Peng et al. (2024)
Jing Peng, Yucheng Wang, Yu Xi, Xu Li, Xizhuo Zhang, and Kai Yu.
A survey on speech large language models.
*arXiv e-prints*, pp. arXiv–2410, 2024.
- Piczak (2015)
Karol J Piczak.
Esc: Dataset for environmental sound classification.
In *Proceedings of the 23rd ACM international conference on Multimedia*, pp. 1015–1018, 2015.
- Radford et al. (2023)
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever.
Robust speech recognition via large-scale weak supervision.
In *International conference on machine learning*, pp. 28492–28518. PMLR, 2023.
- Sager et al. (2019)
Jacob Sager, Ravi Shankar, Jacob Reinhold, and Archana Venkataraman.
Vesus: A crowd-annotated database to study emotion production and perception in spoken english.
In *Interspeech*, pp. 316–320, 2019.
- Sakshi et al. (2024)
S Sakshi, Utkarsh Tyagi, Sonal Kumar, Ashish Seth, Ramaneswaran Selvakumar, Oriol Nieto, Ramani Duraiswami, Sreyan Ghosh, and Dinesh Manocha.
Mmau: A massive multi-task audio understanding and reasoning benchmark.
*arXiv preprint arXiv:2410.19168*, 2024.
- Shon et al. (2022)
Suwon Shon, Ankita Pasad, Felix Wu, Pablo Brusco, Yoav Artzi, Karen Livescu, and Kyu J Han.
Slue: New benchmark tasks for spoken language understanding evaluation on natural speech.
In *ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)*, pp. 7927–7931. IEEE, 2022.
- Team (2025)
Gemma Team.
Gemma 3n.
2025.
URL [https://ai.google.dev/gemma/docs/gemma-3n](https://ai.google.dev/gemma/docs/gemma-3n).
- Turpault et al. (2019)
Nicolas Turpault, Romain Serizel, Ankit Parag Shah, and Justin Salamon.
Sound event detection in domestic environments with weakly labeled data and soundscape synthesis.
In *Workshop on Detection and Classification of Acoustic Scenes and Events*, 2019.
- Tzanetakis & Cook (2002)
George Tzanetakis and Perry Cook.
Musical genre classification of audio signals.
*IEEE Transactions on speech and audio processing*, 10(5):293–302, 2002.
- Vaswani et al. (2017)
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.
Attention is all you need.
*Advances in neural information processing systems*, 30, 2017.
- Wang et al. (2024a)
Bin Wang, Xunlong Zou, Geyu Lin, Shuo Sun, Zhuohan Liu, Wenyu Zhang, Zhengyuan Liu, AiTi Aw, and Nancy F Chen.
Audiobench: A universal benchmark for audio large language models.
*arXiv preprint arXiv:2406.16020*, 2024a.
- Wang et al. (2025a)
Dingdong Wang, Jincenzi Wu, Junan Li, Dongchao Yang, Xueyuan Chen, Tianhua Zhang, and Helen Meng.
Mmsu: A massive multi-task spoken language understanding and reasoning benchmark.
*arXiv preprint arXiv:2506.04779*, 2025a.
- Wang et al. (2025b)
Shaobo Wang, Jiaming Wang, Jiajun Zhang, Cong Wang, Yue Min, Zichen Wen, Fei Huang, Huiqiang Jiang, Junyang Lin, Dayiheng Liu, et al.
Winning the pruning gamble: A unified approach to joint sample and token pruning for efficient supervised fine-tuning.
*arXiv preprint arXiv:2509.23873*, 2025b.
- Wang et al. (2024b)
Xiong Wang, Yangze Li, Chaoyou Fu, Yunhang Shen, Lei Xie, Ke Li, Xing Sun, and Long Ma.
Freeze-omni: A smart and low latency speech-to-speech dialogue model with frozen llm.
*arXiv preprint arXiv:2411.00774*, 2024b.
- Wang et al. (2017)
Yuxuan Wang, RJ Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, et al.
Tacotron: A fully end-to-end text-to-speech synthesis model.
*arXiv preprint arXiv:1703.10135*, 164, 2017.
- Weck et al. (2024)
Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, and Dmitry Bogdanov.
Muchomusic: Evaluating music understanding in multimodal audio-language models.
*arXiv preprint arXiv:2408.01337*, 2024.
- Wen et al. (2024)
Zichen Wen, Dadi Guo, and Huishuai Zhang.
Aidbench: A benchmark for evaluating the authorship identification capability of large language models.
*arXiv preprint arXiv:2411.13226*, 2024.
- Wen et al. (2025a)
Zichen Wen, Yifeng Gao, Weijia Li, Conghui He, and Linfeng Zhang.
Token pruning in multimodal large language models: Are we solving the right problem?
*arXiv preprint arXiv:2502.11501*, 2025a.
- Wen et al. (2025b)
Zichen Wen, Yifeng Gao, Shaobo Wang, Junyuan Zhang, Qintong Zhang, Weijia Li, Conghui He, and Linfeng Zhang.
Stop looking for important tokens in multimodal language models: Duplication matters more.
*arXiv preprint arXiv:2502.11494*, 2025b.
- Wen et al. (2025c)
Zichen Wen, Jiashu Qu, Dongrui Liu, Zhiyuan Liu, Ruixi Wu, Yicun Yang, Xiangqi Jin, Haoyun Xu, Xuyang Liu, Weijia Li, et al.
The devil behind the mask: An emergent safety vulnerability of diffusion llms.
*arXiv preprint arXiv:2507.11097*, 2025c.
- Wen et al. (2025d)
Zichen Wen, Shaobo Wang, Yufa Zhou, Junyuan Zhang, Qintong Zhang, Yifeng Gao, Zhaorun Chen, Bin Wang, Weijia Li, Conghui He, et al.
Efficient multi-modal large language models via progressive consistency distillation.
*arXiv preprint arXiv:2510.00515*, 2025d.
- Xiao et al. (2023)
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis.
Efficient streaming language models with attention sinks.
*arXiv preprint arXiv:2309.17453*, 2023.
- Xiong et al. (2025)
Minhao Xiong, Zichen Wen, Zhuangcheng Gu, Xuyang Liu, Rui Zhang, Hengrui Kang, Jiabing Yang, Junyuan Zhang, Weijia Li, Conghui He, et al.
Prune2drive: A plug-and-play framework for accelerating vision-language models in autonomous driving.
*arXiv preprint arXiv:2508.13305*, 2025.
- Xu et al. (2025a)
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al.
Qwen2. 5-omni technical report.
*arXiv preprint arXiv:2503.20215*, 2025a.
- Xu et al. (2025b)
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, et al.
Qwen2. 5-omni technical report.
*arXiv preprint arXiv:2503.20215*, 2025b.
- Yang et al. (2024)
Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng, Yuanjun Lv, Zhou Zhao, Chang Zhou, et al.
Air-bench: Benchmarking large audio-language models via generative comprehension.
*arXiv preprint arXiv:2402.07729*, 2024.
- Yang et al. (2021)
Shu-wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Jeff Lai, Kushal Lakhotia, Yist Y Lin, Andy T Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin, et al.
Superb: Speech processing universal performance benchmark.
*arXiv preprint arXiv:2105.01051*, 2021.
- Yang et al. (2025)
Yantai Yang, Yuhao Wang, Zichen Wen, Luo Zhongwei, Chang Zou, Zhipeng Zhang, Chuan Wen, and Linfeng Zhang.
Efficientvla: Training-free acceleration and compression for vision-language-action models.
*arXiv preprint arXiv:2506.10100*, 2025.
- Ye et al. (2025)
Jiacheng Ye, Zhihui Xie, Lin Zheng, Jiahui Gao, Zirui Wu, Xin Jiang, Zhenguo Li, and Lingpeng Kong.
Dream 7b: Diffusion large language models.
*arXiv preprint arXiv:2508.15487*, 2025.
- Yi et al. (2021)
Jiangyan Yi, Ye Bai, Jianhua Tao, Haoxin Ma, Zhengkun Tian, Chenglong Wang, Tao Wang, and Ruibo Fu.
Half-truth: A partially fake audio detection dataset.
*arXiv preprint arXiv:2104.03617*, 2021.
- You et al. (2025)
Zebin You, Shen Nie, Xiaolu Zhang, Jun Hu, Jun Zhou, Zhiwu Lu, Ji-Rong Wen, and Chongxuan Li.
Llada-v: Large language diffusion models with visual instruction tuning.
*arXiv preprint arXiv:2505.16933*, 2025.
- Zaheer et al. (2020)
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al.
Big bird: Transformers for longer sequences.
*Advances in neural information processing systems*, 33:17283–17297, 2020.
- Zhang et al. (2023a)
Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu.
Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities.
*arXiv preprint arXiv:2305.11000*, 2023a.
- Zhang et al. (2024a)
Junyuan Zhang, Qintong Zhang, Bin Wang, Linke Ouyang, Zichen Wen, Ying Li, Ka-Ho Chow, Conghui He, and Wentao Zhang.
Ocr hinders rag: Evaluating the cascading impact of ocr on retrieval-augmented generation.
*arXiv preprint arXiv:2412.02592*, 2024a.
- Zhang et al. (2024b)
Yuan Zhang, Chun-Kai Fan, Junpeng Ma, Wenzhao Zheng, Tao Huang, Kuan Cheng, Denis Gudovskiy, Tomoyuki Okuno, Yohei Nakata, Kurt Keutzer, et al.
Sparsevlm: Visual token sparsification for efficient vision-language model inference.
*arXiv preprint arXiv:2410.04417*, 2024b.
- Zhang et al. (2023b)
Zhenyu Zhang, Ying Sheng, Tianyi Zhou, Tianlong Chen, Lianmin Zheng, Ruisi Cai, Zhao Song, Yuandong Tian, Christopher Ré, Clark Barrett, et al.
H2o: Heavy-hitter oracle for efficient generative inference of large language models.
*Advances in Neural Information Processing Systems*, 36:34661–34710, 2023b.

## Appendix

## Appendix A Additional Results

**Table 5: Core statistics of the AudioMarathon.**
| Statistics | Number |
| --- | --- |
| Total Questions | 6567 |
| Audio Domains | 10 |
| Difficulty (Easy:Medium:Hard) | 24%:61%:15% |
| Speech Content Extraction | 1514 |
| Automatic Speech Recognition (ASR) | 204 (3.10%) |
| Speech Content Reasoning (SCR) | 820 (12.49%) |
| Speech Entity Recognition (SER) | 490 (7.46%) |
| Audio classification | 1519 |
| Audio scene classifier (ASC) | 1145 (17.44%) |
| Music classifier (MC) | 120 (1.83%) |
| Sound event detection (SED) | 254 (3.87%) |
| Speaker Recognition | 3530 |
| Emotion Recognition (ER) | 185 (2.82%) |
| Speech Detection (SD) | 776 (11.82%) |
| Speaker Age Recognition (SAR) | 959 (14.60%) |
| Speaker Gender Recognition (SGR) | 1614 (24.58%) |
| Mutiple Choice Questions | 6452 |
| Transcriptions | 270 |

Token Pruning Details. In multi-modal large language models, the first two layers are regarded as shallow layers, where attention allocation remains relatively balanced, and output tokens mainly attend to preceding outputs while modality-specific tokens (e.g., vision or audio) are not yet fully integrated into semantic reasoning. Prior analysis on vision tokens has demonstrated that pruning at the second layer is particularly effective: it removes redundant tokens while retaining a compact set of representative ones, thereby preventing redundant information from propagating into deeper layers and significantly reducing computational overhead. Motivated by this observation, we apply the same strategy to audio tokens, pruning them directly at the second layer. This early pruning leverages the redundancy of low-level acoustic representations, which often contain overlapping information, and achieves a favorable balance between efficiency and performance. Compared with pruning at the first layer, where feature representations are still unstable and critical information may be lost, the second layer offers a more appropriate trade-off between stability and efficiency. Conversely, deferring pruning to deeper layers would result in repeated computations on redundant tokens, diminishing overall efficiency. Thus, second-layer pruning of audio tokens can be understood as a low-level information compression mechanism, which eliminates ineffective tokens at an early stage to maximize acceleration in subsequent layers while maintaining robust performance.

### A.1 Random Choice Baseline

**Table 6: Random baseline results on nine subsets of the dataset after 100 random selections. General refers to double-choice questions; Four-choice refers to Four-option choice questions.**
| Task | Labels | Type | ACC | Macro F1-score |
| --- | --- | --- | --- | --- |
| MC | 4 | four-choice | 0.2477 | 0.2451 |
| SD | 2 | general | 0.5200 | 0.4700 |
| SER | 4 | four-choice | 0.2525 | 0.2518 |
| ASC | 4 | four-choice | 0.2496 | 0.2493 |
| ER | 4 | four-choice | 0.2436 | 0.2426 |
| SGR | 2 | general | 0.5000 | 0.4798 |
| SAR | 5 | general | 0.1979 | 0.1695 |
| SCR | 4 | four-choice | 0.2490 | 0.2491 |
| ASC | 10 | general | 0.2467 | 0.2450 |
| Four-choice (6 tasks) | 0.2490 | 0.2479 |  |  |
| General (3 tasks) | 0.3999 | 0.3744 |  |  |
| Overall (9 tasks) | 0.2993 | 0.2901 |  |  |

The Table [6](https://arxiv.org/html/2510.07293v1#A1.T6)
illustrates that the F1-scores are lower, directly reflecting the validity of using F1-score as an evaluation metric, highlighting its ability to balance precision and recall.

### A.2 Results of Qwen2.5-Omni-3B on the other six datasets in AudioMarathon

Figure: Figure 10: Comparisons of latency and performance trade-off for the Qwen2.5-Omni-3B model on the SED, SD, and SAR dataset
Refer to caption: x11.png

Figure: Figure 11: Comparisons of latency and performance trade-off for the Qwen2.5-Omni-3B model on the MC, ER, and SER dataset
Refer to caption: x12.png

While prior results demonstrate the robustness and general superiority of the Frame method on representative MCQs, we further investigate the specific advantages and disadvantages exhibited by certain methods on particular tasks. For instance, on simpler tasks like SED and MC, Frame performs stably, surpassing random methods. However, it underperforms on more challenging tasks, such as SER and SAR. In contrast, Frame maintains robust performance, proving its relative reliability.

Figure: Figure 12: Acceleration effects for the Qwen2.5-Omni-3B model on the SED, SD, and SAR dataset
Refer to caption: x13.png

Figure: Figure 13: Acceleration effects for the Qwen2.5-Omni-3B model on the MC, ER, and SER dataset
Refer to caption: x14.png

As shown in Figure [12](https://arxiv.org/html/2510.07293v1#A1.F12) and Figure [13](https://arxiv.org/html/2510.07293v1#A1.F13), different token pruning methods maintain a high degree of consistency in acceleration performance, further illustrating their effectiveness in reducing inference time for long audio MCQ tasks.

**Table 7: Summary of Error Analysis for Phi-4-multimodal on ASR Tasks**
| Method | Error Patterns |
| --- | --- |
| Vanilla | Insertions: ’to’: 12, ’the’: 7, ’a’: 4, ’hundred’: 3, ’is’: 3, ’g.’: 3, ’or’: 3, ’be’: 3, ’with’: 3 |
|  | Deletions: ’the’: 13, ’that’: 5, ’a’: 4, ’percent’: 4 |
|  | Substitutions: ’the’ $\rightarrow$ ’a’: 6, ’in’ $\rightarrow$ ’and’: 3, ’had’ $\rightarrow$ ’has’: 3, ’it’ $\rightarrow$ ’it’s’: 3, ’a’ $\rightarrow$ ’the’: 3 |
| FastV Prune 20% | Insertions: ’to’: 11, ’the’: 8, ’hundred’: 4, ’or’: 4, ’be’: 4, ’and’: 3 |
|  | Deletions: ’the’: 451, ’and’: 152, ’to’: 139, ’of’: 126, ’is’: 122, ’in’: 77, ’a’: 76, ’that’: 45, ’was’: 43, ’are’: 40 |
|  | Substitutions: ’the’ $\rightarrow$ ’a’: 7, ’in’ $\rightarrow$ ’and’: 3, ’is’ $\rightarrow$ ’the’: 3, ’a’ $\rightarrow$ ’the’: 3 |
| Dart Prune 20% | Insertions: ’to’: 12, ’the’: 7, ’a’: 5, ’is’: 4, ’g.’: 3, ’or’: 3, ’be’: 3, ’with’: 3 |
|  | Deletions: ’the’: 52, ’to’: 30, ’is’: 26, ’are’: 13, ’a’: 9, ’and’: 9, ’that’: 8, ’of’: 7, ’was’: 7, ’in’: 6 |
|  | Substitutions: ’the’ $\rightarrow$ ’a’: 6, ’a’ $\rightarrow$ ’the’: 4, ’had’ $\rightarrow$ ’has’: 3 |

## Appendix B Error Analysis

In this section, we further compare the token prune results on ASR task. The result of Table [7](https://arxiv.org/html/2510.07293v1#A1.T7) demonstrates the attention-based selection probably causes the loss of high-frequency words. The model’s substantial omission of high-frequency words in the audio transcription task results in inferior performance under the same pruning ratio.

## Appendix C Encoding Granularity of LALMs

**Table 8: Audio processing capacity of Audio Language Models, including maximum supported audio length, maximum number of encoded audio tokens, and token rate (tokens per second).**
| Model Name | Max Audio<br>Length | Max Encoded<br>Audio Tokens | Token Rate<br>(tokens/s) |
| --- | --- | --- | --- |
| Phi-4-multimodal (Abouelenin et al., 2025) | 30 minutes | 22500 | 12.5 tokens/s |
| Aero-1-Audio (Li et al., 2025a) | 15 minutes | 22500 | 25.0 tokens/s |
| Qwen2-Audio-Instruct (Chu et al., 2024) | 0.5 minutes | 750 | 25.0 tokens/s |
| Qwen2.5-Omni (Xu et al., 2025b) | 21 minutes | 32000 | 25.0 tokens/s |

Table [8](https://arxiv.org/html/2510.07293v1#A3.T8) reports the audio encoding granularity of the LALMs. Except for Phi-4-Multimodal, all models produce about 7,500 tokens for a 5-minute clip, even for straightforward tasks such as gender or age classification, which reveals substantial redundancy in current audio encoding.

## Appendix D Dataset Constitute

SLUE (Shon et al., 2022).
The Spoken Language Understanding Evaluation (SLUE) benchmark is a suite of tasks designed for evaluating speech models on spoken language understanding. It is derived from the full 960 hours of the LibriSpeech corpus and includes various tasks such as named entity recognition (NER), sentiment analysis, and relation extraction. For AudioMarathon, we utilize the sentiment analysis subset, which requires models to comprehend spoken content and infer the underlying sentiment.

RACE (Lai et al., 2017).
The Reading Comprehension from Examinations (RACE) dataset is a large-scale collection of reading comprehension questions from English exams for middle and high-school Chinese students. It consists of over 28,000 passages and nearly 100,000 questions written by human experts to evaluate reading comprehension and reasoning skills. In AudioMarathon, we use an audio-transcribed version of the RACE dataset, transforming the text-based reasoning challenge into a listening comprehension task that tests a model’s ability to process and reason over long spoken narratives.

LibriSpeech-long (Park et al., 2024).
LibriSpeech is a widely used corpus for Automatic Speech Recognition (ASR), containing approximately 1,000 hours of English speech read from public domain audiobooks. The original dataset consists of short audio clips, typically a few seconds long. For AudioMarathon, we created LibriSpeech-long by concatenating multiple short clips from the same speaker and chapter to form continuous, long-form audio files, which are used to evaluate the models’ long-context ASR performance.

DESED (Turpault et al., 2019).
The Domestic Environment Sound Event Detection (DESED) dataset is designed for the task of sound event detection in real-life domestic environments. The dataset is built using a combination of synthesized and real recordings from AudioSet, focusing on 10 common domestic sound classes (e.g., dog bark, blender, speech). It provides strong annotations with precise event start and end times, making it a challenging benchmark for evaluating the temporal localization capabilities of audio models.

GTZAN (Tzanetakis & Cook, 2002).
GTZAN Genre Collection is one of the most widely used datasets for music genre classification. It consists of 1,000 audio tracks, each 30 seconds long, distributed evenly across 10 distinct music genres (e.g., Blues, Classical, Hip-Hop, Jazz, Rock). Each genre is represented by 100 clips. Despite some known issues with label consistency in a small fraction of the data, it remains a standard benchmark for evaluating music information retrieval.

TAU Urban Acoustic Scenes (Heittola et al., 2019).
TAU Urban Acoustic Scenes dataset is a collection of recordings from various acoustic scenes for the task of acoustic scene classification. The 2019 version, which we reference, contains over 40 hours of audio recorded in 10 different European cities. The data is provided as 10-second segments extracted from longer original recordings, capturing diverse urban environments such as airports, public parks, and metro stations. In our benchmark, we utilize these longer source recordings to test scene classification in extended audio contexts.

HAD (Yi et al., 2021).
The Hallym Aging Diacrisis (HAD) dataset is a Korean speech corpus designed for the study of age-related voice characteristics and the diagnosis of pathological voices in the elderly. It contains speech samples from different age groups, including young adults and elderly individuals, performing various speech tasks like reading passages and sustained vowel phonations. The dataset is annotated with speaker age and health status, making it suitable for tasks related to age estimation and vocal health detection.

VESUS (Sager et al., 2019).
The Voice Evaluation for Specific UtteranceS (VESUS) dataset is a corpus for assessing voice pathologies. It contains recordings from speakers with various voice disorders as well as healthy controls. Speakers were recorded producing specific utterances, such as sustained vowels and standard sentences, which are designed to highlight vocal impairments. The dataset is annotated by expert clinicians with labels for overall voice quality and specific perceptual ratings (e.g., roughness, breathiness, strain), serving as a benchmark for automated voice quality assessment systems.

Vox_Age & Vox_Gender (Hechmi et al., 2021).
These tasks are derived from the large-scale VoxCeleb speaker recognition dataset (Nagrani et al., 2017). VoxCeleb consists of hundreds of thousands of "in-the-wild" speech clips extracted from celebrity interview videos on YouTube. While the primary purpose of VoxCeleb is speaker identification and verification, the metadata associated with each celebrity allows for the creation of auxiliary tasks. For AudioMarathon, we use this data to evaluate speaker characteristic identification, specifically age estimation (VoxAge) and gender classification (VoxGender) from long, unconstrained speech segments.

## Appendix E Model Details

Phi-4-Multimodal (Abouelenin et al., 2025).
This model is extended from Phi-4-Mini and integrates three input modalities: text, vision, and speech/audio. Its key innovation lies in the use of the “Mixture-of-LoRAs” technique: while keeping the base language model completely frozen, it introduces modality-specific LoRA adapters and a routing mechanism to enable flexible multimodal reasoning (e.g., vision + language, vision + speech, speech-only) without interference across modalities.

Qwen2.5-Omni (Xu et al., 2025b).
Developed by the Qwen Team, this is an end-to-end multimodal model capable of perceiving multiple modalities, including text, image, audio, and video, and supporting streaming generation of both text and natural speech responses. Its main innovations include: the introduction of TMRoPE (temporally aligned multimodal rotary position embedding) for audio-video timestamp synchronization; the Thinker–Talker architecture, where the Thinker is responsible for text generation and the Talker generates audio tokens based on the hidden states of the Thinker, thereby avoiding interference between text and speech generation; and the use of block-level processing and sliding-window DiT mechanisms to reduce streaming latency.

Audio-Flamingo-2 (AF2) (Ghosh et al., 2025).
This model is an audio language model (ALM) with long audio understanding ability (30 seconds to 5 minutes) and expert-level reasoning capabilities. Its core innovations include: the AF-CLAP audio encoder, trained with an improved contrastive loss on over 8 million audio–text pairs; the AudioSkills dataset, which consists of 4.2 million question–answer pairs covering seven categories of reasoning skills; and a three-stage curriculum training strategy including pretraining, fine-tuning, and long-audio fine-tuning.

Audio-Flamingo-3 (AF3) (Goel et al., 2025).
Jointly developed by NVIDIA and the University of Maryland, this is a leading fully open-source LALM. Its main innovations include: the AF-Whisper unified audio encoder, which enables joint representation learning of speech, environmental sounds, and music; support for on-demand reasoning (e.g., chain-of-thought reasoning), multi-turn multi-audio dialogue, long audio understanding up to 10 minutes, and speech-to-speech interaction.

Baichuan-Omni-1.5 (Li et al., 2025c).
Developed by Baichuan Inc., this is a full-modality model capable of understanding text, image, audio, and video, as well as supporting end-to-end audio generation. Its main strengths include: a data processing pipeline that constructs and cleans approximately 500B high-quality multimodal data; the Baichuan-Audio-Tokenizer, designed to capture both semantic and acoustic features (implemented with an 8-layer RVQ structure and a 12.5 Hz frame rate); and a multi-stage training strategy consisting of image–text pretraining, image–audio–text joint pretraining, full-modality pretraining, and multimodal supervised fine-tuning.

Gemma-3n (Team, 2025).
The Gemma 3n models are optimized for efficient execution on low-resource devices. They support multimodal input (text, image, video, audio) and generate high-quality text outputs. The series provides open weights for both pre-trained and instruction-tuned variants, and covers more than 140 natural languages. The Gemma 3n models employ selective parameter activation technology, which reduces resource requirements and allows the models to operate effectively at sizes of 2B or 4B parameters, although the total number of parameters is larger.

Aero-1-Audio (Li et al., 2025a).
Aero-1-Audio is a compact audio model developed by LMMs-Lab as part of the Aero-1 series, the first generation of lightweight multimodal systems. Built upon the Qwen-2.5-1.5B language model, it achieves strong performance across speech recognition, audio understanding, and instruction-following benchmarks while remaining parameter-efficient. Trained within one day on 16 H100 GPUs with 50k hours of curated data, Aero demonstrates that efficient training is possible with high-quality samples. It further supports continuous audio inputs up to 15 minutes, a challenging setting for most existing audio models.

GPT-4o (Hurst et al., 2024).
GPT-4o, released by OpenAI in August 2024, is an autoregressive universal model supporting arbitrary combinations of text, audio, image, and video as inputs, and text, audio, and image as outputs. All modalities are processed by a single end-to-end trained neural network, enabling seamless multimodal integration and efficient inference across diverse tasks.

Gemini-2.0-Flash-Lite (Comanici et al., 2025a).
Gemini-2.0-Flash-Lite, introduced by Google in April 2025, is the most cost-efficient member of the Gemini 2.0 family. It adopts a sparse Mixture-of-Experts Transformer architecture and leverages Trillium TPUs for training and inference. The model supports text, image, audio, and video inputs with a context window of 1,048,576 tokens, and produces text outputs of up to 8,192 tokens. Its design emphasizes scalability and latency efficiency for high-volume multimodal applications.

Gemini-2.0-Flash (Comanici et al., 2025a).
Gemini-2.0-Flash is a natively multimodal model designed to power next-generation agentic systems. Compared with Gemini 1.5 Flash, it offers higher quality while maintaining comparable inference speed. It accepts text, image, audio, and video inputs with a 1,048,576-token context window and outputs text up to 8,192 tokens, with experimental image generation capabilities. Its architecture refines the sparse Mixture-of-Experts Transformer design with improved stability and optimization efficiency.

Gemini-2.5-Flash (Comanici et al., 2025a).
Gemini-2.5-Flash is Google’s first hybrid reasoning model, allowing developers to toggle reasoning on or off and allocate reasoning budgets for a trade-off between quality, cost, and latency. It supports text, image, audio, and video inputs with a 1M-token context window and generates text outputs up to 64K tokens. Based on a sparse Mixture-of-Experts Transformer with native multimodal support, it significantly outperforms Gemini-1.5-Pro on reasoning and multimodal benchmarks.

Gemini-2.5-Flash-Lite (Comanici et al., 2025a).
Gemini-2.5-Flash-Lite extends the hybrid reasoning family with a cost-efficient design optimized for latency-sensitive tasks such as translation and classification. It provides improvements over Gemini-2.0-Flash-Lite in coding, mathematics, science, and reasoning, while supporting text, image, audio, and video inputs with a 1M-token context window and generating text outputs up to 64K tokens. Its sparse Mixture-of-Experts Transformer architecture balances efficiency with strong performance in large-scale multi-modal applications.

## Appendix F Prompt

Here we present the prompt templates used for various tasks in our AudioMarathon.

Figure: Figure 14: Prompt template for the SED task.
Refer to caption: x15.png

Figure: Figure 15: Prompt template for the MC task.
Refer to caption: x16.png

Figure: Figure 16: Prompt template for the SD task.
Refer to caption: x17.png

Figure: Figure 17: Prompt template for the ASR task.
Refer to caption: x18.png

Figure: Figure 18: Prompt template for the SCR task.
Refer to caption: x19.png

Figure: Figure 19: Prompt template for the SER task.
Refer to caption: x20.png

Figure: Figure 20: Prompt template for the ASC task.
Refer to caption: x21.png

Figure: Figure 21: Prompt template for the SGR task.
Refer to caption: x22.png

Figure: Figure 22: Prompt template for the ER task.
Refer to caption: x23.png

Figure: Figure 23: Prompt template for the SAR task.
Refer to caption: x24.png

## Appendix G Limitations and Future Work

### G.1 Limitations

Our benchmark reflects practical choices in data sources and task design. Some datasets have license limits that restrict redistribution. The benchmark focuses on English and may not reflect cross-language behavior. We rely on automatic pipelines for audio concatenation and option generation, which can introduce bias if the source data has bias. While we test multiple long audio tasks, some domains and tasks are still underrepresented. Our evaluation covers common metrics but does not fully capture human preference or safety risks. Finally, we focus on inference efficiency methods at test time and do not include training time efficiency or energy use.

### G.2 Future Work

We plan to run systematic hyperparameter searches at key encoder and decoder layers to measure sensitivity and find settings that preserve temporal detail while improving efficiency. We will evaluate more compression and acceleration methods, including stronger token selection methods and better cache policies, and test transfer from text and vision methods to audio. We will add more tasks and languages, broaden source datasets, and release tools for reproducible data building and evaluation. We also plan to study human evaluation for long audio tasks and extend metrics that measure temporal continuity and memory. Finally, we will report energy and cost to give a fuller view of efficiency.
In this work, we observe that existing autoregressive audio LLMs still face a dual challenge of accuracy and efficiency when processing long-form audio, as exemplified by ASR. Exploring architectures that depart from the autoregressive paradigm may offer a promising path forward. In particular, the recent surge of diffusion LLMs (Nie et al., 2025; You et al., 2025; Ye et al., 2025; Wen et al., 2025c; Jin et al., 2025), which support parallel decoding, suggests a compelling alternative. Leveraging diffusion LLMs for long-audio understanding and generation could unlock substantial gains in both fidelity and computational throughput.

## Appendix H Use of LLMs

In this study, we utilized large language models (LLMs) to perform grammar checking and to polish certain sentences for improved clarity and fluency, without altering the original meaning of the text. Auxiliary AI coding tools are used for debugging and analyzing code errors, as well as assisting in code implementation, with the main code being constructed and carefully reviewed by humans.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 2
presentation: 2
questions: - What is the FRAME method?
- Why is the proposed FRAME only similar to baseline pruning methods and you list it as a contribution?
- What are some scientific discoveries of the large-scale ablation studies done in section 4-5?
soundness: 2
strengths: The proposed AudioMarathon benchmark is a solid contribution to the audio understanding community. Current long audio understanding benchmarks such as BLAB are limited to speech and other challenging benchmarks are focused on reasoning/skills instead of length. AudioMarathon combines the strengths of both sides and brings the distinct challenging capabilities to long audio. 

The dataset curation pipeline seems quite delicate. Fig 2 makes it very easy to understand and also straightforward for the community to reproduce or curate related training data for research development. There is also manual verification to keep quality high.
summary: This paper introduces AudioMarathon that focuses on evaluation of long audio understanding. AudioMarathon is large-scale, over 200 seconds per sample on average, and across all domains (speech, sound, and music). AudioMarathon is curated with 7 different sub-categories, which covers more capabilities than prior benchmarks. The paper also studies token pruning methods and KV-cache methods in baselines for efficient long audio inference.
weaknesses: Despite the fact that the AudioMarathon benchmark quite solid, it is more of an engineering contribution rather than methodology contribution. Its novelty is limited in that BLAB introduced long audio understanding (despite focused on speech) and Audio Flamingo 2 introduced LongAudioBench (focused on AQA). From this perspective, the proposed AudioMarathon is a complement to existing benchmarks with limited novelty and methodological contribution. 

The baselines are not extensive enough: Table 3 only includes a limited number of models, but there are many more good ones that are publicly available. It is unimaginable that a benchmark paper does not include enough major models. 

In the token pruning part, it is unclear what the proposed FRAME method is. It only has one sentence of description (L267-268). While the paper lists this as contribution, the so-called FRAME does not show any improvements in Table 4. 

I appreciate the large-scale ablations for the token pruning and KV cache methods, but they are just ablation studies and the paper does not seem to reveal any valuable information from these ablations. There is no in-depth analysis on why certain methods are better than others, or what are some newly discovered weaknesses of baseline models that should be addressed. 

In summary, I think the paper proposes a solid benchmark that complements prior ones in terms of audio lengths, capabilities, and domains. However, there is limited novelty and scientific discoveries presented in this paper.

---

confidence: 3
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. Details of Human Participation: Could you provide more details about the human involvement in this study? Specifically, for the benchmark construction (Step 5. Manual Verification), what were the detailed criteria used for the review? I could not find them in Appendix D. Furthermore, for the Human Evaluation scores in Table 3, could you elaborate on the methodology, including the number of participants, whether they were native English speakers, and their level of expertise for the tasks?
2. Random vs. Well-Designed Strategies: I noticed that in both token pruning and KV cache eviction, the Random method performs surprisingly well. What do you believe is the reason for this? Does it suggest that the more sophisticated, well-designed strategies are not well-suited for the audio modality, or are there other factors?
3. ASR Curve in Figure 5: In Figure 5(d), the performance of the Frame method continuously improves as latency decreases from 65s to ~45s. This is an expected trade-off, where less pruning (higher latency) leads to better accuracy. However, the curves for other strategies seem to start at a lower latency point. Could you provide the performance of the other strategies in the 45s-65s latency range for a more direct comparison, or is it that their maximum latency (with minimal pruning) is already below 45s?
4. Details of the Frame Strategy: Could you provide a more detailed technical description of the Frame pruning strategy? The paper describes it as a "time-aligned token pruning strategy," but more specifics on its implementation would be very helpful.
soundness: 4
strengths: 1. Significant Motivation: The motivation for AudioMarathon is highly significant. Most existing benchmarks and datasets for large audio models focus on short audio clips, whereas many real-world applications require the processing of long-form audio. This benchmark effectively addresses that gap.
2. Comprehensive and Diverse Scenarios: The benchmark includes 10 different sub-tasks that cover a wide range of audio-related scenarios, with tasks designed to evaluate both semantic and acoustic capabilities.
3. Robust Data Pipeline: The paper details a well-designed data pipeline that includes source selection, automated construction, and manual verification, which together ensure the reliability and quality of the benchmark.
4. Sufficient Model Evaluation: The study provides a thorough evaluation by testing 16 different audio models and including human evaluation scores as a baseline. This clearly demonstrates the current capabilities of models on this new benchmark.
5. Valuable Exploration of Efficiency Optimization: In the context of long audio, efficiency is critical. The paper's exploration of token pruning and KV-cache eviction demonstrates that while these techniques can reduce memory usage and inference latency, there remains significant room for improvement and further research for the audio modality.
summary: The authors introduce AudioMarathon, a novel and comprehensive benchmark designed to evaluate Large Audio Language Models (LALMs) on long-form audio. The work is motivated by the significant gap in existing benchmarks, which primarily use short audio clips and thus fail to test models on realistic, minute-scale tasks. AudioMarathon features audio inputs from 90 to 300 seconds across 10 diverse tasks covering speech, music, and environmental sounds. Using this benchmark, the paper conducts a large-scale evaluation of 16 state-of-the-art LALMs, revealing significant performance degradation as audio length increases and a substantial gap compared to human performance, particularly on tasks requiring temporal reasoning. A key part of the study is a systematic analysis of inference efficiency techniques like token pruning and KV-cache eviction, highlighting the critical trade-offs between computational cost and model accuracy for long-context audio processing.
weaknesses: 1. The Definition of "Long Audio" Could Be Extended: While the 300-second duration pushes the limits of many current LALMs, real-world scenarios such as meetings, movies, lectures, and podcasts often feature much longer audio, potentially exceeding 30 minutes. It would be more impactful if the paper could demonstrate model performance on even longer audio inputs.
2. Lack of Specific Multi-Speaker Tasks: As noted in the paper, tasks like Speaker Age Recognition (SAR) are performed in multi-speaker contexts. However, more direct multi-speaker tasks, such as speaker counting or diarization, are missing. Future work could benefit from incorporating such tasks.
3. Limited Evaluation Format: With the exception of ASR, all other tasks are framed as multiple-choice questions (MCQs). This format, while easy to evaluate, may not fully capture a model's deep reasoning or summarization capabilities. The benchmark would be more valuable if it included more open-ended question-answering tasks.

Minor comments that did not impact the score:
1. The caption for Figure 6 appears to be incomplete.
2. The abbreviation "MCQ" is used without prior definition.
3. As the authors mention in their limitations, it is hoped that this work can be extended to more languages in the future.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 2
questions: See Weaknesses.
soundness: 2
strengths: 1. Focus on long-form audio and well-thought-out tasks spanning across each domain in audio.
2. Comprehensive experiment results comparing open-sourced and closed-sourced LALMs
3. Analysis on performance trade-off focusing on inference-time efficiency
4. Clear presentation in the first half of the paper for the dataset specification
summary: This paper introduces AUDIOMARATHON, a long-audio benchmark that aggregates 10 tasks spanning speech, environmental audio, and music domains. By studying the performance of open-source and closed-source models on AUDIOMARATHON, the paper shows that current LALMs are still underperforming with long input. Moreover, this paper studies inference-time acceleration techniques for LALMs, including token pruning and KV-cache eviction, and shows their trade-offs to performance.
weaknesses: Formulation of the “long-audio” datasets needs clearer specification.
1. The work appears to add limited new “long-audio” material. Figure 2 suggests that all tasks are long-form and created by concatenation, which is misleading. Appendix D indicates that the only newly introduced long-form set is RACE, and its audio is produced by TTS from a text dataset. For environmental audio and music tasks, it seems the audio is taken directly from the original sources.
2. For the “multi-hop” or “complex reasoning” component, namely RACE, the evaluation largely probes ASR capability and downstream text understanding. An audio-specific, more acoustic-centered multi-hop design would better substantiate the “complex reasoning” claim.

Presentation of experiments and the path from results to conclusions is confusing. The paper would offer more actionable guidance for LALM development with tighter exposition, more focused ablations, and conclusions that are directly supported by the evidence.
1. Frame pruning is under-specified. The text motivates time-aligned pruning and claims that Frame preserves rare short acoustic events, but it does not provide a precise algorithmic description that enables reproduction, for example window size, stride, scoring rule, tie breaking, and any layer-wise schedules. Appendix A mainly explains why pruning occurs at the second layer rather than detailing how Frame operates.
2. KV-cache eviction analysis is limited. Figure 7 shows peak memory across policies, yet the curves look similar and appear driven primarily by the number of tokens. The paper does not report accuracy trade-offs under different eviction settings.
3. Length attribution is missing. The claim that “current models fail at long audios” would be stronger with a length-controlled comparison. The paper aggregates minute-scale results and human baselines, but it does not include short-context baselines for the same tasks or a duration sweep. Without a length-matched ablation, it is difficult to attribute errors to long context rather than task or domain effects.
4. Figure-level clarity issues. The caption of Figure 9 does not make clear which pruning method each panel uses, and the nearby text only mentions “varying token pruning ratios” and “maximum F1 across settings,” which prevents mapping curves to methods. In Figure 5, the caption states that “Frame consistently outperforms other methods,” which seems overstated, since in later segments of ASC and SGR other methods match or surpass Frame at higher token counts.
5. Evidence for “task-aware audio token compression is essential” is insufficient. The Results section makes a broad necessity claim, but the curves mainly show about a 20 percent latency reduction in several regimes. The paper does not provide controlled accuracy-at-fixed-latency comparisons against strong baselines, nor latency-at-fixed-accuracy tables that would justify essentiality rather than usefulness.
6. Actionable request tied to the claim (line 365). Where the paper asserts that token compression is “essential,” please add controlled accuracy-at-fixed-latency or latency-at-fixed-accuracy tables. The current evidence supports that certain pruning methods can maintain similar accuracy with about a 20 percent latency drop. It does not yet support the stronger claim that token compression is essential.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: **Questions**:
- Regarding step 2/3, can the authors elaborate more on the logic behind the concatenation of and merging of clips to have longer sequences and the corresponding “tool design”? The description in Section 2.2 is a bit vague.
- Regarding step 5, why did the authors only review 10% of the data? How many samples needed to be fixed? Unless the failure ratio is extremely low, then I think it would be better to double check all the samples. 
- Why Flamingo-2 achieves only 1% of WAR? It seems a big outlier.
- Why in Table 4, random pruning for Qwen2.5-Omni-3B oftentimes perform better than the other methods which perform more accurate compressions?
- In the paragraph “How do token pruning methods affect performance in LALMs?”, how does the authors explain the fact that Qwen2.5-Omni-3B “benefits from token pruning”? Do they mean that the model benefits when the ratio is 30% or even up to 90%? It seems to deteriorate rather than improve, and this makes sense given the high compression ratio. 
- Can the authors provide some examples of failure cases in some of the considered tasks? Or at least compare when some models fail/succeed. Based on how and where the models make the mistakes, I think it would be easier to devise better techniques to improve their capabilities on long-context scenarios. 


**Typos**:

- Duplicated reference: Qwen2.5-omni technical report by Jin Xu et al., 2025.
- Missing full stops on all image captions.
- Missing acronym definition: MCQ.
- In Table 3: Close-source —> Closed-source.
soundness: 3
strengths: - The paper is well-written and the objectives are clearly stated.
- The paper includes several results.
- The focus of the paper in evaluating LALMs on long-duration clips is of sufficient importance.
summary: This paper introduces AudioMarathon, a comprehensive benchmark for long audio understanding, spanning 10 audio tasks. The authors evaluate several sota LALMs on this benchmark, revealing that the methods struggle more as audio lengths increases.  They also study various acceleration techniques, such as token pruning and KC-cache eviction, to analyze their tradeoffs.
weaknesses: - The papers claims that most of the existing benchmarks entail short audio clips, and therefore they don't capture "complexity of real-world scenarios such as meetings, podcasts, and extended dialogues" (lines 107-108). However, the duration of the clips in AudioMarathon is in the range [90,300]seconds, which in my opinion is quite shorter than the average duration of meetings/podcasts. This is why the benchmark provided in "Orevaoghene Ahia et al., BLAB: Brutally Long Audio Bench, 2025" seems more appropriate as the average audio duration is 51minutes.
- The results about compression and peak memory usage are somewhat expectable and they do not add much value to the paper. 
- The paper lacks multiple details and clarifications --> see questions below.
-  It's not clear to me how the proposed "Frame" compression method works in practice. The description at lines 268-268 is insufficient.
- In Figure 7, it seems like the SnapKV line is never displayed in the 4 plots, and so does it seem for KNorm method on figures (b), (c), and (d). Are they completely overlapping with the other methods or what?