FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing
Authors: Shoutao Guo
Shaolei Zhang
Qingkai Fang
Zhengrui Ma
Min zhang
Yang Feng
Venue: NeurIPS 2025 poster ()
Keywords: Spoken Language Understanding
Large Speech-Language Models
Speech Technologies
Primary area: applications
TL;DR: This paper introduces FastLongSpeech, a novel framework designed to extend LSLM capabilities for efficient long-speech processing without necessitating dedicated long-speech training data.

Abstract: The rapid advancement of Large Language Models (LLMs) has spurred significant progress in Large Speech-Language Models (LSLMs), enhancing their capabilities in both speech understanding and generation. While existing LSLMs often concentrate on augmenting speech generation or tackling a diverse array of short-speech tasks, the efficient processing of long-form speech remains a critical yet underexplored challenge. This gap is primarily attributed to the scarcity of long-speech training datasets and the high computational costs associated with long sequences. To address these limitations, we introduce FastLongSpeech, a novel framework designed to extend LSLM capabilities for efficient long-speech processing without necessitating dedicated long-speech training data. FastLongSpeech incorporates an iterative fusion strategy that can compress excessively long-speech sequences into manageable lengths. To adapt LSLMs for long-speech inputs, it introduces a dynamic compression training approach, which exposes the model to short-speech sequences at varying compression ratios, thereby transferring the capabilities of LSLMs to long-speech tasks. To assess the long-speech capabilities of LSLMs, we develop a long-speech understanding benchmark called LongSpeech-Eval. Experiments show that our method exhibits strong performance in both long-speech and short-speech tasks, while greatly improving inference efficiency.

FogeehngSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing
Shoutao Guo Shaolei Zhang 1.3, Qingkai Fangl Zhengrui Ma 1.3, 1.2.3 Min Zhang', Yang Feng
Key Laboratory of Intelligent Information Processing,
Institute of Computing Technology. Chinese Academy of Sciences (ICT/CAS)
Key Laboratory of AI Safety, Chinese Academy of Sciences
3 University of Chinese Academy of Sciences, Beijing, China
School of Future Science and Engineering. Soochow University
guoshoutao2229ctt.oom ac en, www.an.....aaaaaa...a.aa zhangshaolei20200cccr
cn fengyangdict.
Abstract
The rapid advancement of Large Language Models (LLMs) has spurred significant progress in Large Speech-Language Models (LSLMs), enhancing their capabilities in both speech understanding and generation. While existing LSLMs often concen- trate on augmenting speech generation or tackling a diverse array of short-speech tasks, the efficient processing of long-form speech remains critical yet underex- plored challenge. This gap is primarily attributed to the scarcity of long-speech training datasets and the high computational costs associated with long sequences. To address these limitations, we introduce FastLongSpeech, a novel framework designed to extend LSLM capabilities for efficient long-speech processing without necessitating dedicated Long-speech training data. Sonpeeh incorporates an iterative fusion strategy that can compress excessively long-speech sequences into manageable lengths. To adapt LSLMs for long-speech inputs, it introduces a dynamic compression training approach, which exposes the model to short-speech sequences at varying compression ratios, thereby transferring the capabilities of L.SLMs to long-speech tasks. To assess the long-speech capabilities of LSSLMs, we develop long-speech understanding benchmark called LongSpeech-Evväl Experi- ments show that our method exhibits strong performance in both long-speech and short-speech tasks, while greatly improving inference efficiency
Introduction
Benefiting from the advancement of Large Language Models (LLMs) 133) Large Speech-Language Models (LSLMs) have also made significant strides by extending the speech capabilities of LLMs.
By harnessing the knowledge and reasoning abilities of LLMs, LSLMs can directly comprehend speech signals, perform analysis and reasoning, and achieve superior performance in diverse of tasks such as speech recognition, speech translation, and speech understanding [4-6]. The ability to process and understand diverse speech signals has emerged as a key research focus in L.SLMs 7
To handle speech inputs, traditional methods g typically employ a cascaded pipeline, where speech is first transcribed into and then processed by LLMs. However, these approaches suffer from error propagation and discard valuable paralinguistic information (10). To overcome these drawbacks, recent research 33 has shifted towards an end-to-end paradigm, enabling LSLMs
"Corresponding author: Yang Feng.
"The Code is at htttp:///iitu//
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

to directly process and reason with speech signals. These methods can be broadly divided into two categories. On one hand, some approaches [6) such as Qwen2-Audio [1] align the output spaces of pre-trained audio encoders with the embedding of I.LMs, which allows speech inputs to be accommodated while transferring partial capabilities of the employed L.MMs. the same time, other methods [7] involves discretizing the speech to discrete units, which allows LSLMs to handle speech units similar to text tokens. Despite these advancements, current LSLMs are largely constrained to processing short speech segments, typically under 30 seconds [S [8]. Only a few LSLMs [12] have achieved a processing duration of 30 minutes on speech summarization tasks by relying on the construction of extensive, specialized training datasets.
The processing of long-form speech by LSLMs remains a largely unexplored area, primarily due to two significant challenges. First, unlike the abundance of diverse short-speech datasets (1), there is a scareity of training data specifically for long-speech alignment and instruction, and the generation of such data is costly. Second, long-speech sequences impose substantial computational demands on LSLMs. The sequence of speech representations often more than four times longer than its text equivalents for the same content [7. which, in the context of long-form speech, leads to significantly higher computational costs. Therefore, LSLMs face considerable challenges modeling long-speech sequences, stemming from both a scarcity of training data and increased computational costs. These challenges limit the exploration and application of LSLMs in long-speech processing.
To address the above challenges, we propose FastLongSpeech, a novel framework designed to extend the capabilities of LSLMs long-speech processing. leveraging only short-speech training data. We utilize Qwen2-Audiq [ the currently representative LSLM,, as our foundational speech-languagt model. To enable long- long-speech processing. FongL.ongSpeech incorporates speech extractor module on top of the audio encoder [22]. This module employs our proposed iterative fusion strategy to compress the speech representations output by the audio encoder, preserving essential temporal information while reducing redundancy. The resulting condensed speech representations are then used by LLM for comprehension and reasoning. In our method, the speech extractor significantly reduces the sequence length of the condensed representations, thereby lowering the computational costs for subsequent LLM.
To further adapt LSLM for long-speech processing, FastLongSpeech employs two-stage training approach. In the first stage, CTC loss 23 is introduced within the extractor module to measure the text density of the input speech representations, which is utilized for the iterative fusion strategy. The second stage introduces dynamic compression training method, enabling LLM to effectively adapt to condensed speech representations. This stage leverages existing short-speech data and dynamically adjusts the compression ratios in the iterative fusion strategy to transfer the understanding and reasoning capabilities of LSLMs to long-speech tasks.
To evaluate the long-speect understanding capabilities of LSLMs, we also develop a benchmark, called LongSpeech-Eval. Experiments show that Sageehh achieves efficient speech process- ing on both long-speech and sport-speech benchmarks, and can balance efficiency and effectiveness to meet different requirements.
2 Background
Our method builds upon Qwen2-Audio [5] and incorporates CTC loss 233 We provide brief overview of these key components below.
Qwen2-Audio Qweno-Audio is a large-scale audio-langaage model capable of comprchending various audio signals, performing reasoning, and generating text responses. It consists of three modules: an audio encoder [22), an LLM 244,, and an adaptor. The adaptor serves to align the output of the audio encoder with the embeddings of LLM, enabling the LLM to process speech inputs. Following extensive pre-training, supervised fine-tuning, and DPO (25), pwwn2--Audia demonstrates remarkable performance across wide range of speech tasks. Given raw waveform (S1, ..... 8N) sampled at 16 kHz, the audio encoder produces a sequence of speech representations = (A1, -... 11))
For convenience, Qwen2-Audio refers to QwenZ-Audio- InInttrucc throughout this paper.

---

Speosh
Emieedding
Iterative Fusion
Spseech
0.7
02
Projector
0.74
0.66
0.52
Span
Extractor
llerative Fusion
CTC Decoder
Audio Encoder
Span
Spai
Figure I: Architecture of OastLongSpeech. The left panel illustrates that FastLongSpeech generates a response based the input speech and text instruction. The right panel details the iterative fusion strategy, where numbers between adjacent frames denote similarity scores and numbers below frames represent content density.
with 25 Hz frame rate. The LLMs then generate the text response based on and an instruction x:
ply h) Σ p(u yei, h);
where p( yeis h) denotes probability distribution of the next token.
CTC Connectionist Temporal Classification (CTC) 23 is widely used in Automatic Speech Recognition (ASR), where the input typically longer than the corresponding output (26). To align speech with the transcript, CTC introduces a blank token into the vocabulary and defines the possible output as an alignment a. Each time step this alignment corresponds to either a blank token or non-blank token. The alignment has the same length as the speech sequence and can be reduced to the final transcript through a collapse function r-1, During training, CTC employs an efficient dynamic programming algorithm to maximize the probability of all possible alignments corresponding to the ground-truth transcript c:
Late log Σ
(ate(a h),
aET(c)
where (c) denotes the set of all alignments corresponding to and h denotes the sequence of speech representations produced by audio encoder [22]. this paper, leverage the output distribution of CTC to quantify the text density of speech representations h, which is subsequently utilized in the iterative fusion strategy.
Method
In this section, we introduce the architecture of FastLongSpeech. with particular emphasis on its novel speech extractor module. To empower LSLLM with the ability to learn speech compression techniques and adapt to long-speech processing, we present an innovative two-stage training approach. Additionally, we introduce the LongSpeech- benchmark, designed evaluate the long-speect understanding capabilities of LSLMs.
3

---

3.1 Model Architecture
Figure I jillustrates the model framework of FastLongSpeech. Building upon Qwen2-Audio, we incorporate an advanced extractor module, which features a CTC [23] decoder and employs our pro- posed iterative fusion strategy to condense speech representations. The workflow of FastLongSpeech proceeds as follows. Given waveform s sampled at 16 kHz, the audio encoder converts it into a mee-sppettoggamm and processes it through convolution and transformer layers (2),, yielding a sequence of speech representations h with 25 Hz. frame rate. Subsequently, the extractor module processes h using an iteration fusion strategy to produce the condensed representations h whose length is within the speech window of LLM. The speech window denotes the maximum length of the speech representations h during training, specifically the maximum number of speech frames. LLM then utilizes the condensed representations h along with the instruction x to generate the response y, as described in Eq.
3.2 Iterative Fusion
To facilitate efficient long-speech processing, we introduce an extractor to compress lengthy and sparse speech representations [7 into more compact forms. Our primary objective is to minimize information loss during compression, preserving essential temporal information while reducing redundancy. Thus, our method needs to retain speech representations that contain more textual content, while discarding excessively similar adjacent speech frames. To achieve this, we propose an iterative fusion strategy that incrementally merges selected representations based on content density and similarity between adjacent frames, ultimately yielding condensed representations.
We first define the metries measure content density and frame similarity, followed by an introduction of our iterative fusion strategy. For a given speech frame hy, content density is generally associated
with the amount of textual information it contains [27, [28]. To quantify this, we employ a CTC decoder, whose output distribution provides the probabilities of the speech frame being classified as
either a blank or a non non-blank token. Consequently, the content density of hy is derived from the sum of probabilities for non-blank tokens:
d, Eetee(a hg);
aytt
where denotes the blank token. Frame similarity, on the other hand, captures the overlap in content between adjacent speech frames. We measure this using cosine similarity between hy and hy+1:
h,h [
eji+1
After introducing these measurements, we provide an overview of our iterative fusion strategy in Figure [ For m-th iteration, we first determine the length T(m) of current speech representations and the Tength for the next iteration:
1 ( [T(m))/2,
Tf > 2L
T(m+
(5)
if S 2L
where L denotes the final target length of condensed representations h'. The number of speech frames to be reduced is then calculated as:
r(m) T(m) T(m+1).
Subsequently, we utilize the similarity metric between adjacent frames, as defined in Eq.44) to identify the r(m) most similar pairs of adjacent frames. The consecutive identified frames are grouped into a span, as illustrated in Fi For each span, we employ a weighted fusion approach, leveraging the content density in Eq. compressed speech frame. This process yields a new sequence of speech representations with length T(m+1), comprising both the newly condensed frames and the remaining uncompressed frames. If T(m+1) 1) still exceeds the target length L, we initiate another iteration. Otherwise, the resulting speech representations from this round constitute the final condensed representations h'.
weights to merge all frames within the span into single
This iterative fusion strategy effectively reduces the length of speech representations by half in each iteration, facilitating efficient speech processing.

---

3.3 Training Method
After introducing the iterative fusion strategy, we further present a two-stage training method to enable FastLongSpeech to perform long-speech tasks. To facilitate the generation of condensed representations through the iterative fusion strategy, the first stage of training focuses on the ASR task, allowing FastL LnngSpechh to learn the measurement of content density. Building this, the second stage incorporates our proposed dynamic compression training method, which helps LLMss adapt to short-speech condensed representations with varying compression ratios, thereby transferring short-speech capabilities to long-speech processing.
CTC Training In the first stage, we aim to leverage the ASR task to enable FastLongSpeech to recognize the amount of textual information in speech representations, namely content density. To achieve this, we introduce a CTC decoder in the extractor module, which is trained using the CTC loss [23] as shown in Eq. (2). This allows us to utilize the generation distribution of the CTC decoder to measure the content density of speech representations. In this stage, only train the CTC decoder.
Dynamic Compression Training the second stage, we introduce a novel dynamic compression training method. The introduction of this method is based on two considerations. First, it enables the LLM to adapt to condensed representations h' with different compression ratios. Second, the current <s, y> triplet training data primarily contains short-speech clips, which are typically under 30 seconds in duration [29]. By sampling the length of condensed representations [30], LSLM can maintain its perception of speech sequences corresponding to the length of its speech window, without avoiding excessive bias towards overly condensed speech sequences. The dynamic compression training method is as follows:
Latt
logply x, IF,, L)),
L~>((L)
where is uniformly sampled from L. which is set of hyperparameters. IF [F, L) represents applying the iterative fusion operation on the speech representations h to obtain the condensed representations of length L. After the training process, FastLLoggSpeech can transfer the short-speech capabilities of LSLMs to long-speech tasks.
3.4 LongSpeech-Eii Benchmark
After introducing FastLongSpeech, we aim to evaluate its capacity to handle long-speech inputs. Due to the lack of benchmarks for assessing the long-speech capabilities of LSLMs, we construct LongSpeech- Eval. This benchmark is based on the MultiFieldQA-EE and Narrative subsets of LongBench 311. a comprehensive long context understanding benchmark. This benchmark assesses the ability of LLM to answer questions based on the long document.
To construct LongSpeech-Ever we aim to convert the document into speech. We first employ to filter out samples containing numerous formulas or non-English characters. Subsequently, GPT-4o 32 is utilized to summarize and polish the document into a spoken format, Llama3.1-70B-Instruu is then used answer questions based the spoken-form document, with inappropriate samples manually discarded. For the remaining samples, we then synthesize speech for the spoken-form document using Orca Consequently, each sample in the LongSpeech-EF Eval consists of the synthesized speech along with the corresponding questions and answers. More details are provided in Appendix A
4 Experiments
4.1
Datasets
For the training data the first stage, we utilize the ASR data, which contain 960 hours of LibriSpeech data and 3k hours of data sampled from MLS (34.. In the second training stage, our training data primarily originates from three datasets following the Spoken QA format: OpenASQA (35]
33
708- Instruct
5

---

LibriSQA [3), and Common Voice B77. All employed samples used for training are under 30 seconds in duration. For OpenASQA, we use the Open-Ended Speech AQA subset (5.9k hours), which covers diverse question types including content, speaker style, and emotion. For LibriSQA, we incorporate the 360-hour training set. For Common Voice, we adapt the English subset (1.7k hours) to the spoken QA format by generating transcription instructions via ChatGPT and using the original transcripts as the ground-truth answers.
For evaluation, we employ a diverse set of nine datasets spanning five distinct tasks to comprehensively assess performance:
Short-Speech Spoken QA: We utilize three datasets: the speech QA iemocap
[38,
the LibriSQA test set [36,, and the LibriTTS test subset from OpenASQA [3]. The three datasets contain rich set of QA pairs involving paralinguistic information. We utilize this task to evaluate the effectiveness of various speech fusion methods under different compression ratios.
We leverage our proposed LongSpeech- benchmark to assess the performance of different methods in long-speech understanding scenarios.
Long-Speech Spoken QA:
Spoken Dialogue Understanding: We evaluate the inference efficiency of our method using Bircc__dilogue_a_e QA_fisher subset from AIR-Bench.
Emotion Recognition: We leverage the MELD dataset [39] to benchmark our method against other efficiency method [40] under diverse efficiency scenarios.
ASR: We use the LibriSpeech 33 test-clean and test-other, and GigaSpeech 41 test set to evaluate ASR performance. The ASR task utilizes short-speech samples, which evaluate the capacity to extract complete content from the speech.
Speech Information Retrieval: We introduce SPIRAL-H benchmark, which is introduced in the paper of SpeechPrune 42] for long speech information retrieval.
More details of the training and evaluation dataset are in Appendix B
4.2 System Settings
Given the current scarcity of long-speech LSLMs,, we implement several baseline approaches in addition to our Fastl SoneSh
Random method randomly selects frames from speech representations and arranges them sequentially to serve as conditioning input for LSLM.
AvgPool method sequentially selects fixed number of speech frames to form segments and then apply average operation within each segment.
MostSim method first selects the most similar consecutive speech frames and then applies the average pooling operation to each set of adjacent representations.
NTK-RoPE method modifies the base Rotary Position Embedding (RoPE) 43 of L.MSs to an NTK-Aware Scaled RoPE [44) thereby extending the speech window of Qwen2-Audio to match the context length of its LLM. This method is exclusively utilized for long-speech reasoning tasks.
Cascaded first uses Whisper-Large-V. V3 22 to transcribe the audio, and then pass the resulting context text to Qwen-7B-
(since Qwen2-Audio-77 is based on Liven-BBChhit
Baseline refers to the direct application of vanilla Qwen2-Audio for inference, which is not employed in seeg-speech inference tasks.
Fastl LongSech refers to our proposed method.
We use Qwen2-Audio-P7 A BB-ntrue as the base L.LLM for all methods, with the length of speech window as 750. Besides, we also extend our method to vanilla Qwen2.5-Omni [15] without dynamic compression training to verify the effecctivnness of our method. In the first stage of training for our SastLongSpeech, we only train the CTC decoder. In the second stage, we experimentally set L to (750, 400, 200, 100, 50, 25, 12) and fine-tune the LLM of Owen2-Audin using LoRA 45). For fair comparison, we also fine-tune tune Qwen2-Audhe for all methods except the Baseline and FaseeongSppechh,

---

10
3.-
24
22
Mordo
Nodia
15
Modt
15
20
30
310
(a) Brrsch__AAicmmaane
(b) LibriTTS (OpenASQA)
(c) LibriSQA
Figure 2: Performance of diverse speech fusion methods in the short-speech spoken QA tasks. The score is derived from the I.LM evaluating the quality of responses based on the questions and ground- truth answers. The baseline model utilizes speech window of 750 frames. For the methods other than Baseline, we regulate the compression ratio by adjusting the target length L of the condensed speech representations. In the experiments, smaller value of corresponds to a higher compression ratio. A higher score indicates a better quality of the responses.
using the same training data and implementation settings as used for FastLongSpeech. All methods employ the original prompt template from Qwen2-Audio. For more training details, please refer to the Appendix C
During evaluation, we use greedy search for all methods and control compression ratios by adjusting the target length L. For long-speech inference, we split the input speech into series of 30-second clips, which are processed by the audio encoder and then combined into a complete sequence of speech representations in temporal order. To evaluate the performance, tailored to each task. For the Spoken QA and Spoken Dialogue Understanding task, we use Llama3.1 scale of 5, with the scoring template available in the Appendix D For the ASR task, we use Word Error Rate (WER) to assess the accuracy of the generated transcripts. For Emotion Recognition task, we use the Accuracy (ACC) metric evaluate the performance.
employ various metrics
2 to
responses
4.3 Main Results
We evaluate the performance of our method on short-speech and bong-speech spoken QA tasks.
For the short-speech spoken QA task, Figure 2 illustrates the performance of various methods across three datasets. Our Fast- LongSpeech method consistently outperforms other methods on all three datasets under different speech compression ratios, main- taining high response quality even at 30-fold compression ratio (L = 25). Unlike the Random method, which arbitrarily discards speech representations, other methods consider all temporal in- formation when compressing speech representation, resulting in improved generation quality (29]. Compared AvgPool and MostSim methods, our method more effectively eliminates re- dundant information while preserving highly informative speech representations, leading to better performance across various com-
Table Performance of various speech methods on long-speech spoken QA task.
Method
Score (f)
Random
2.54
Similar
3.08
AvgPool
3.10
NTK RoPE
3.44
FasSeongSpeech
3.55
pression ratios. Notably, when L equals 12, other speech fusion methods exhibit similarly suboptimal performance, while our method maintains substantial performance advantage. We attribute this superiority to our novel iterative fusion strategy and dynamic compression training approach. Fur- thermore, compared vanilla Qwen2-Audio, shorter sequence of speech representations, demonstrating higher efficiency.
method achieves comparable performance with
For the long-speech spoken QA task, our method outperforms other approaches in generation quality, as evidenced Table To handle the long-speech input, methods such as Random, Similar, and AvgPool employ their respective fusion techniques to compress the speech representations within the speech window. However, these approaches yield suboptimal generation quality, primarily due to ineffective fusion strategies and misaligned training methods. In contrast, NTK-RoPE expands the

---

speech window of LSLM to the context length of LLM, thereby preserving more speech information and achieving improved performance. Furthermore, our method leverages more effective speech fusion strategy coupled with a dynamic compression training approach, transferring the short-speech reasoning capabilities I.SLMS to the long-speech domain. Notably, despite utilizing the same speech window size as Qwen2-Audio [5], our method achieves optimal performance in long-speech comprehension tasks with greater efficiency than NTK-RoPE.
Analysis
To provide comprehensive evaluation of our approach, we conduct extensive analyses. We then introduce each analytical experiment in detail.
5.1 Ablation Study
To gain comprehensive understanding of the contribu- tions made by different components in our approach, we conduct detailed ablation experiments. As shown in Ta- ble both the iterative fusion and dynamic compression training strategies proposed FostLooggSeech signif- icantly enhance the performance of LSLMs on long- speech reasoning tasks. First, the dynamic compression training strategy effectively transfers the short-speech ca- pabilities LSLMs to long-speech scenarios, utilizing only short-speech data. This approach enables LLMs to adapt to condensed representations at varying com- pression ratios and mitigate wer-reliance excessively compressed speech representations. Consequently, Fast- LongSpeech can compress long-speech representations to fit within the speech window length, facilitating effi- cient song-speech processing at high compression ratios. Moreover, multiple iterations in the iterative fusion ap- proach lead substantial improvements in generation
The ablation experiments of our method on long-speech benchmark. "w/o DCT" replaces Dynamic Compression Training method with standard fine-tuning approach. "w/o Iterative Fusion" elimi- nates the multiple iterations in the iterative fusion. "w/o Content Density" substitutes the method of merging all speech frames within the same span with an average pool- ing operation.
Table 2:
Method
Score (1)
Fast LongSoohn
3.55
w/o DCT
3.33
w/o Iterate Fusion
3.41
w/o Content Density
3.28
quality. This finding underscores the benefits of progressively expanding the receptive field [46] in
iterative fusion for aggregating semantic information. Furthermore, guided by content density, our iterative fusion strategy tends to retain more informative speech frames [27], resulting in the most significant performance improvement.
5.2 Inference Efficiency
After investigating the impact of various components in FaggSeeng we conduct analyses on the infer- ence efficiency across different methods. To quantify this efficiency, we employ the TFLOPs metric, which measures the average number of floating-poin opera- tions (FLOPs) across the entire dataset and is calculated using calflops" tool. For long-speech scenarios, in- corporate average runtime as an additional efficiency indicator, which is measured in seconds. Table 3 and present the results of inference efficiency experiments, which are obtained on NVIDIA L40.
Table 3: The inference efficiency Lib- riTTS test subset of OpenASQA dataset, where "Ours" denotes Fastl SongSScch...
Method
Score (1)
TFLOPs (L)
Baseline
3.73
9.79
Ours (L=400)
3.80
8.54
Ours (I.=200)
3.87
5.64
Ours (L=100)
3.71
4.17
In short-speech tasks, our approach demonstrates performance comparable to vanilla Qwen2- Audio while requiring only half the computational resources. When the allocated computational resources are increased, we can achieve better results. Notably, the computational costs de- crease as the compression ratio increases. This only demonstrates the better efficiency of our model but also highlights its ability to balance generation quality and inference efficiency.
httpp:///gtaab..c////////aaaaaa
pytorch

---

The advantages of our method become even more pronounced in long-speech tasks, where our method achieves better genera- tion quality than NTK-ROPE, with a 70% reduction in runtime and a 60% decrease in computational costs. Compared to the cas- caded method, it even achieves speedup of more than sevenfold, underscoring its sub- stantial efficiency advantage for processing long-form speech. This further shows the effectiveness of our method in handling long-speech inputs. For spoken dialogue understanding, emotion recognition and speech information retrieval tasks, please refer to the Appendix E and F
Table The efficiency on the long-speech benchmark.
Method
Score (t)
TFL.Pss (L)
Time (4)
NTK-RoPE
3.44
61.21
4.80
Cascaded
3.75
n/a
17.23+1.38
Ours
3,55
26.44
1.47
5.3 Content of Condensed Representations
Beyond the spoken QA, spoken dialogue understanding and emotion recognition tasks, we extend our evaluation to the ASR task, which requires precise transcription of the entire speech content 47 Through this task, we explore variations in condensed representations across different compression ratios. Table 5 demonstrates the ASR performance of Qwen2-Audio and our method. At low compression ratios (L=400), FaseLhngSpeech per- forms comparably to Qwen2-Audio, demonstrating the effectiveness of our dynamic compression training and iterative fusion strategy in preserving speech content. Unlike Qwen2-Audio, our method does not require sub- stantial post-processing to extract the transcript, with strong instruction following abilities. At higher com- pression ratios (L=100), FastLongSppech slightly trails Qwen2-Audio in ASR but maintains comparable results in spoken QA, as illustrated in Figure 2 This indicates that while our approach demonstrates applicability across diverse tasks, the optimal compression ratio is inherently task-dependent. There- fore, achieving an effective balance between efficiency and effectiveness thus necessitates careful calibration and thorough assessment of resource constraints.
Table 5: The performance on the ASR
task, where "Ours" denotes the Fast- LongSpeech. For the dataset, "Clean" and "Other" denote LibriSpeech test-clean and test-other sets. "Giga" denotes the test set of GigaSpeech. The results are evaluated with WER metric.
Method
Clean
Other
Giga
Baseline
3.85
6.70
13.71
Ours (L=750)
4.04
7.02
11.76
Ours (L=400)
4.08
7.17
11.77
Ours (L=200)
4.36
7.40
12.70
Ours (L=100)
27.12
24.61
23.69
6
Related Work
Large Opeech-LLnnguae Models With the advancements in Large Language Models (LLMs), recent research attempts to extend the understanding and reasoning capabilities of LLMs to speech inputs, becoming Large
Models (LSLMs). Early studies (8] 9 employ cascading paradigm, where speech is first transcribed into text before being processed by LLMs. More recently, some works (3) 48
utilize the adaptors to align the output space of speech encoders with the
input space of LLMs, achieving multi-tast task LSLMs. Other approaches [49 II 50 17 utilize speech discretization techniques, converting waveforms into discrete units, enabling LSLMs to process speech in the same way they process text. These approaches allow LSLMs to handle both speech understanding and generation.
Long Sequence Modeling Long sequence modeling presents challenges across diverse domains, including text, video, and speech. The approaches to long-con cong-context modeling vary depending on the type of the inputs. For extended text sequences, researchers explored methods such as position interpolation and extrapolation [51] sliding window [52], continuous fine-tuning on long-text data
[53], and native sparse attention [54] To address long-video processing, recent works leverage frame
selection or merging strategies 55 as well as vision token merging techniques [56]]. In the realm
of speech processing, early methods focus on enhancing the performance of ASR [57] and speech
translation (58) through speech compression techniques. More recently, FastAdaSP (40 mitigates inference overhead by performing token selection within L.MSs. Concurrently, Speechprune [29] employs token selection strategy to extend the effective speech window of Qwen2-Audee to 90

---

seconds for Speech Information Retrieval task. StreamUni 42 achieves real-time speech translation for long speech streams by integrating a segmentation strategy and a policy-decision module.
Conclusion
In this paper, introduce FastLongSpeech, a novel approach that extends the capabilities of LSLMs to efficiently conduct long-speech processing. Experiments show that our method significantly reduces the computational costs and inference time in long-speech tasks, achieving better trade-offs between performance and efficiency.
Limitations
Given the current scarcity of long-speech data, FastL.ongSpeech introduces an innovative dynamic compression training approach. This method leverages short-speech training data to extend the capabilities of LSLMs for long-speech processing. As long-speech training and evaluation data become more abundant in the future, Pegeehh will further enhance its ability to process longer speech inputs using the expanded datasets with lower costs.
Acknowledgement
We gratefully acknowledge all the reviewers for their valuable comments and suggestions. This work was supported by the Natural Science Foundation of Beijing, China (Grant No. L257006).
References
[1] OpenAl, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander Mądry, Alex Baker-Whitcen Whitcobb,,, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, Alex Nichol, Alex Paino, Alex Renzin, Alex Tachard Passos, Alexander Kirillov, Alexi Christakis, Alexis Conneau, Ali Kamali, Allan Jabri, Allison Moyer, Allison Tam, Amadou Crookes, Amin Tootoochian, Amin Tootoonchian, Ananya Kumar, Andrea Vallone, Andrej Karpathy, Andrew Braunstein, Andrew Cann, Andrew Codispoti, Andrew Galu, Andrew Kondrich, Andrew Tul- loch, Andrey Mishchenko, Angela Baek, Angela Jiang, Antoine Pelisse, Antonia Woodford, Anuj Gosalia, Arka Dhar, Ashley Pantuliano, Avi Nayak, Avital Oliver, Barret Zoph. Behrooz Ghorbani, Ben Leimberger, Ben Rossen, Ben Sokolowsky, Ben Wang, Benjamin Zweig, Beth Hoover, Blake Samic, Bob McGrew, Bobby Spero, Bogo Giertler, Bowen Cheng, Brad Lightcap, Brandon Walkin, Brendan Quinn, Brian Guarraci, Brian Hsu, Bright Kellogg, Brydon Eastman, Camillo Lugaresi, Carroll Wainwright, Cary Bassin, Cary Hudson, Casey Chu, Chad Nelson, Chak Li, Chan Jun Shern, Channing Conger. Charlotte Barette, Chelsea Voss. Chen Ding, Cheng Lu, Chong Zhang, Chris Beaumont, Chris Hallacy, Chris Koch, Christian Gibson, Christina Kim, Christine Choi, Christine McLeavey, Christopher Hesse, Claudia Fischer, Clemens Winter, Coley Czamecki, Colin Jarvis, Colin Wei, Constantin Koumouzelis, Dane Sherburn, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Duc Phong Nguyen, Duncan Findlay, Edede Oiwoh, Edmund Wong, Ehsan Asdar, Blizabeth Proehl, Elizabeth Yang. Eric Antonow, Eric Kramer, Eric Peterson, Eric Sigler, Eric Wallace, Hugene Brevdo, Evan Mays, Farzad Khorasani, Felipe Petroski Such, Filippo Raso, Francis Zhang, Fred von Lohmann, Freddie Sulit, Gabriel Goh, Gene Oden, Geoff Salmon, Giulio Starace, Greg Brockman. Hadi Salman, Haiming Bao, Haitang Hu, Hannah Wong, Haoyu Wang. Heather Schmidt, Hcather Whitney, Heewoo Jun, Hendrik Kirchner, Henrique Ponde de Oliveira Pinto, Hongyu Ren, Hui- wen Chang, Hyung Won Chung, Ian Kivlichan, Ian O' O'Connell Ian O' D'Connell, Ian Osband, Ian Silber, Ian Sohl, Ibrahim Okuyucu, Ikai Lan, Ilya Kostrikov, Ilya Sutskever, Ingmar Kanitschei- der, Ishaan Guirajani, Jacob Coxon, Jacob Menick, Jakub Pachocki. James Aung, James Betker, James Crooks, James Lennon, Jamie Kiros, Jan Leike, Jane Park, Jason Kwon, Jason Phang, Jason Teplitz, Jason Wei, Jason Wolfe, Jay Chen, Jeff Harris, Jenia Varavva, Jessica Gan Lee, Jessica Shieh, Lin, Jiahui Yu, Jiayi Weng, Jie Tang, Jieqi Yu, Joanne Jang, Joaquin Quinonero Candela, Joe Beutler, Joe Landers, Joel Parish, Johannes Heidecke, John Schulman, Jonathan
10

---

Lachman, Jonathan McKay, Jonathan Uesato, Jonathan Ward, Jong Wook Kim. Joost Huizinga, Jordan Sitkin, Jos Kraaijeveld, Josh Gross, Josh Kaplan, Josh Snyder, Joshua Achiam, Joy Jiao, Joyce Lee, Juntang Zhuang, Justyn Harriman, Kai Fricke, Kai Hayashi, Karan Singhal, Katy Shi, Kavin Karthik, Kayla Wood, Kendra Rimbach, Kenny Hsu, Kenny Nguyen, Keren Gu-Lemberg, Kevin Button, Kevin Liu, Kiel Howe, Krithika Muthukumar, Kyle Luther, Lama Ahmad, Larry Kai, Lauren Itow, Lauren Workman, Leher Pathak, Leo Chen, Li Jing. Lia Guy, Liam Fedus, Liang Zhou, Lien Mamitsuka, Lilian Weng, Lindsay McCallum, Lindsey Held, Long Ouyang, Louis Feuvrier, Lu Zhang. Lukas Kondraciuk, Lukasz Kaiser, Luke Hewitt, Luke Metz, Lyric Doshi, Mada Aflak, Maddie Simens, Madelaine Boyd, Madeleine Thompson. Marat Dukhan, Mark Chen, Mark Gray, Mark Hudnall, Marvin Zhang, Marwan Aljubeh, Mateusz Litwin, Matthew Zeng, Max Johnson, Maya Shetty, Mayank Gupta, Meghan Shah, Mehmet Yatbaz, Meng Jia Yang, Mengchao Zhong, Mia Glaese, Mianna Chen, Michael Janner, Michael Lampe, Michael Petrov, Michael Wu, Michele Wang. Michelle Fradin, Michelle Pokrass, Miguel Castro, Miguel Oom Temudo de Castro, Mikhail Pavlov, Miles Brundage, Miles Wang, Minal Khan, Mira Murati, Mo Bavarian, Molly Lin, Murat Yesildal, Nacho Soto, Natalia Gimelshein, Natalie Cone, Natalie Staudacher, Natalie Summers, Natan LaFontaine, Neil Chowdhury, Nick Ryder, Nick Stathas, Nick Turley, Nik Tezak, Niko Felix, Nithanth Kudige, Nitish Keskar, Noah Deutsch, Noel Bundick, Nora Puckett, Ofir Nachum, Ola Okelola, Oleg Boiko, Oleg Murk, Oliver Jaffe, Olivia Watkins, Olivier Godement, Owen Campbell-Moore Patrick Chao, Paul McMillan, Pavel Belov, Peng Su. Peter Bak, Peter Bakkum. Peter Deng. Peter Dolan, Peter Hoeschele, Peter Welinder, Phil Tillet, Philip Pronin, Philippe Tillet, Prafulla Dhariwal, Qiming Yuan, Rachel Dias, Rachel Lim, Rahul Arora, Rajan Troll, Randall Lin, Rapha Gontijo Lopes, Raul Puri, Reah Miyara, Reimar Leike, Renaud Gaubert, Reza Zamani, Ricky Wang. Rob Donnelly, Rob Honsby, Rocky Smith, Rohan Sahai, Rohit Ramchandani, Romain Huet, Rory Carmichael, Rowan Zellers, Roy Chen, Ruby Chen, Ruslan Nigmatullin, Ryan Cheu, Saachi Jain, Sam Altman, Sam Schoenholz, Sam Toizer, Samuel Miserendino, Sandhini Agarwal, Sara Culver, Scott Ethersmith, Scott Gray, Sean Grove, Sean Metzger, Shamez Hermani, Shantanu Jain, Shengjia Zhao, Sherwin Wu, Shino Jomoto, Shirong Wu, Shuaiqi, Xia, Sonia Phene, Spencer Papay, Srinivas Narayanan, Steve Coffey, Steve Lee, Stewart Hall, Suchir Balaji. Tal Broda, Tal Stramer, Tao Xu, Tarun Gogineni, Taya Christianson, Ted Sanders, Tejal Patwardhan, Thomas Cunninghman. Thomas Degry, Thomas Dimson, Thomas Raoux, Thomas Shadwell, Tianhao Zheng. Todd Underwood, Todor Markov, Toki Sherbakov, Tom Rubin, Tom Stasi, Tomer Kaftan, Tristan Heywood, Troy Peterson, Tyce Walters, Tyna Eloundou, Valerie Qi, Veit Moeller, Vinnie Monaco, Vishal Kuo, Vlad Fomenko, Wayne Chang, Weiyi Zheng, Wenda Zhou, Wesam Manassra, Will Sheu, Wojciech Zaremba, Yash Patil, Yilei Qian, Yongjik Kim, Youlong Cheng, Yu Zhang, Yuchen He, Yuchen Zhang, Yujia Jin, Yunxing Dai, and Yury Malkov. Gpt-4o system card, 2024. StLpsttps:/ /tapp:///rxivv org harg/abb/24455 21276
[2] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong. Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song. Danielle Pintz, Danny Livshits, Danny Wyatt, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovie, Francisco Guzmán, Frank Zhang. Gabriel Synnaeve, Gabrielle Lee, Georgia I Lewis Anderson, Govind Thattai, Graeme Nail, Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Misra, Ivan Evtimov, Jack Zhang, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Va- suden Alwala, Karthik Prasad, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Kushal Lakhotia, Lauren Rantala- Yearr, Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke
11

---

de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Mannat Singh, Manohar Paluri, Marein Kardas, Maria Tsimpoukelli, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kam- badur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychex, Niladri Chatterji, Ning Zhang. Olivier Duchenne, Onur Çelebi, Patrick A Alrass,, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng. Prajjwal Bhargava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, QingXiao Dong, Ragavan Srinivasan, Raj Ganapathy, Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohan Maheswari, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sumbaly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hos- seini, Sahana Chennabasappa, Samjay Singh, Sean Bell, Seohyun Somia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckhacher, Todor Mihaylov, Tong Xiao, Ujjwal Karn, Vedanuj Goswami, Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do, Vish Vogeti, Vítor Albiero, Vladan Petrovic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaofang Wang, Xiaoqing Ellen Tan, Xide Xia, Xinfeng Xie, Xuchao Jia, Xuewei Wang. Yaelle Goldschlag, Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan. Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aayushi Srivastava, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria. Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Amos Teo, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Dong, Annie Franco, Anuj Goyal, Aparajita Saraf, Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisemman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonhardi, Bernie Huang. Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Brandon Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Ce Liu, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai. Chris Tindal, Christoph Feichtenhofer. Cynthia Gao. Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Duc Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani, Emily Hahn, Emily Wood, Eric-Tuan Le, Erik Brinkman, Esteban Arcaute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Filippos Kokkinos, Firat Ozgenel, Francesco Caggioni, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan. Hakan Inan, Hamid Shojanazeri, Han Zou, Hannah Wang. Hanwen Zha, Haroun Habeeb, Harrison Rudolph. Helen Suk, Henry Aspegren, Hunter Goldman. Hongyuan Zhan, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Ilias Leontiadis, Irina-Elena Veliche. Itai Gat, Jake Weissman, James Geboski, James Kohli, Janice Lam, Japhet Asher, -apn-sae Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres, Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U, Karan Saxena, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kiran Jagadeesh, Kun Huang, Kunal Chawla, Kyle Huang, Lailin Chen, Lakshya Garg, Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Keneally, Miao Liu, Michael L. Seltzer, Michal Valko, Michelle Restrepo, Mihir Patel, Mik Vyatskov, Mikayel Samvelyan. Mike Clark, Mike Maccy, Mike Wang. Miquel Jubert Hermoso, Mo Metanat, Mohammad Rastegari, Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navyata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikhil Mehta, Nikolay Pavlovich Laptev, Ning Dong, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh, Paul Saab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani, Pritish Yuvraj, Qian Liang, Rachad Alao, Rachel Rodriguez. Rafi Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Rangaprabhu Parthasarathy, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Russ Howes, Ruty Rinott, Sachin Mehta, Sachin Siby, Sai Jayesh Bondu, Samyak Datta, Sara Chugh,
12

---

Sara Hunt, Sargun Dhillon, Sasha Sidorov, Satadru Pan, Saurabh Mahajan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lindsay, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shishir Patil, Shiva Shankar, Shugiang Zhang, Shugiang Zhang, Sinong Wang. Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max, Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Summer Deng. Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kochler, Thomas Robinson, Tianhe Li, Tianjun Zhang, Tim Matthews, Timothy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez. Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vlad Ioneseu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang, Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaojian Wu, Xiaolan Wang. Xilun Wu, Xinbo Gao. Yaniv Kleinman, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Ii, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang. Yu Zhao, Yuchen Hao, Yundi Qian, Yunlu Li, Yuzi He, 7ach Rait, Zachary Vito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, Zhiwei Zhao, and Zhiyu Ma. The llama herd of models, 2024. URL http:htttss///aree org abs/ /40/ 21783
[3] DeepSeck-AI, Aixin Liu, Bei Feng, Bing Xue. Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, L. Cai, Jian Liang. Jianzhong Guo, Jiaqi Ni, Jiashi Li, Jiawei Wang. Jin Chen, Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, Junxiao Song, Kai Dong, Kai Hu, Kaige Gao. Kang Guan, Kexin Huang, Kuai Yu, Lean Wang, Lecong Zhang, Lei Xu, Leyi Xia, Liang Zhao, Litong Wang. Liyue Zhang. Meng Li, Miaojun Wang, Mingchuan Zhang, Minghua Zhang. Minghui Tang, Mingming Li, Ning Tian, Panpan Huang. Peiyi Wang. Peng Zhang, Qiancheng Wang. Qihao Zhu, Qinyu Chen, Qiushi Du, J. Chen, R. L. Jin, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang. Runxin Xu, Ruoyu Zhang, Ruyi Chen, S. S. Li, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shaoqing Wu, Shengfeng Ye, Shengfeng Ye, Shirong Ma, Shiyu Wang. Shuang Zhou, Shuiping Yu, Shunfeng Zhou, Shuting Pan, T. Wang, Tao Yun, Tian Pei, Tianyu Sun, W. L. Xiao, Wangding Zeng, Wanjia Zhao, Wei An, Wen Liu, Wenfeng Liang, Wenjun Gao, Wenqin Yu, Wentao Zhang, X. Q. Li, Xiangyue Jin, Xianzu Wang, Xiao Bi, Xiaodong Liu, Xiaohan Wang, Xiaojin Shen, Xiaokang Chen, Xiaokang Zhang, Xiaosha Chen, Xiaotao Nie, Xiaowen Sun, Xiaoxiang Wang, Xin Cheng, Xin Liu, Xin Xie, Xingchao Liu, Xingkai Yu, Xinnan Song, Xinxia Shan, Xinyi Zhou, Xinyu Yang, Xinyuan Li, Xuecheng Su, Xuheng Lin, Y. K. Li, Y. Q. Wang, Y. X. Wei, Y. X. Zhu, Yang Zhang, Yanhong Xu, Yanhong Xu, Yanping Huang, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Li. Yaohui Wang. Yi Yu, Yi Zheng, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Ying Tang, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma. Yiyuan Liu, Yongqiang Guo, Yu Wu, Yuan Ou, Yuchen Zhu, Yuduan Wang. Yue Gong, Yuheng Zou, Yujia He, Yukun Zha, Yunfan Xiong, Yunxian Ma, Yuting Yan, Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Z. F. Wu. Z. Z. Ren, Zehui Ren. Zhangli Sha, Zhe Fu, Zhean Xu, Zhen Huang, Zhen Zhang. Zhenda Xie, Zhengyan Zhang. Zhewen Hao, Zhibin Gou, Zhicheng Ma, Zhigang Yan, Zhihong Shao, Zhipeng Xu, Zhiyu Wu, Zhongyu Zhang, Zhuoshu Li, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song. Ziyi Gao, and Zizheng Pan. Deepseek-v3 technical report, 2024. URL BRLphttps:// http///xxxvvv//// htwpa//.orggab///1/111 19437
[4] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun MA, and Chao Zhang. SALMONN: Towards generic hearing abilities for large language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openrev//
[5] Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv. Jinzheng He, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen2-audio technical report, 2024. URL htRpsttps:/// http://axvvvv.oc wwtp:/gg///aaa/a/ee ht////bb//2/400 10759
[6] Qian Chen, Yafeng Chen, Yanni Chen, Mengzhe Chen, Yingda Chen, Chong Deng, Zhihao Du, Ruize Gao, Changfeng Gao, Zhifu Gao, Yabin Li, Xiang Lv, Jiaqing Liu, Haoneng Luo, Bin Ma, Chongjia Ni, Xian Shi, Jialong Tang, Hui Wang, Hao Wang, Wen Wang, Yuxuan Wang, Yunlan Xu, Fan Yu, Zhijie Yan, Yexin Yang. Baosong Yang, Xian Yang, Guanrou Yang, Tianyu Zhao, Qinglin Zhang, Shiliang Zhang, Nan Zhao, Pei Zhang, Chong Zhang, and Jinren
13

---

Zhou. Minmo: A multimodal large language model for seamless voice interaction, 2025, URL sttpp:://axiv..rr//aa wttpvv...rg/bb///////0 htsp//bb//21111.. 06282
[7] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models, 2023. URL https://arxix> orazziiii.orv//eeeeee orgg/ba///aie hbt///aa./111...// 07919
[8] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face, 2023. URL https
//axxiv.org.. /200. 17580
[9] Rongjie Huang, Mingze Li, Dongchao Yang, Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong. Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanabe. Audiogpt: Understanding and generating speech, music, sound, and talking head, 2023. URL
https://arxiv. httpaaaxivooggr/e///ee// abs 23004. 12995
[10] Chen Wang. Minpeng Liao, Zhongqiang Huang, Jinliang Lu, Junhong Wu, Yuchen Liu, Chengqing Zong, and Jiajun Zhang. Blsp: Bootstrapping language-speech pre-training via behavior alignment continuation writing, 2024. URL httpp://arxiv.orrg
00916
[11] Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. Speechgpt: Empowering large language models with intrinsic cross- moddl conversational abilities, 2023. URL https: ptatps://arxite wwwpx/iivorrgeeoooo absp://bb//// abs/2305 2000.111010000000001 11000
[12] Microsoft, :, Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang. Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Becckr Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang. Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Tetyana Sych, Prancetha Vaddamanu. Shuohang Wang. Yiming Wang. Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang, Ziyi Yang, Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-of-loras 2025. URL https://arxix: org/ abs/ /5033 01743
[13] KimiTeam, Ding Ding, Zeqian Ju, Yichong Leng, Songxiang Liu, Tong Liu, Zeyu Shang, Kai Shen, Wei Song, Xu Tan, Heyi Tang. Zhengtao Wang, Chu Wei, Yifei Xin, Xinran Xu, Jianwei Yu, Yutao Zhang, Xinyu Zhou, Y. Charles, Jun Chen, Yanru Chen, Yulun Du, Weiran He, Zhenxing Hu, Guokun Lai, Qingeheng Li, Yangyang Liu, Weidong Sun, Jianzhou Wang, Yuzhi Wang, Yuefeng Wu, Yuxin Wu, Donghha Yang. Hao Yang, Ying Yang, Zhilin Yang, Aoxiong Yin, Ruibin Yuan, Yutong Zhang, and Zaida Zhou. Kimi-audio technical report, 2025. URL
https://arxiv: org/ abs abs/ 18425
[14] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma. Shaolei Zhang, and Yang Feng. LLaMA- omni: Seamless speech interaction with large language models. In The Thirteenth International Conference on Learning Representations, 2025. URL httph::tt/ //opp:re////peeeeeee www.iiiwwweeetteee nen/f iddPYmrUQmMEN
[15] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang. Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-ommi technical report, 2025. https://s///r: org htrgg/bs//5000 abs// 20215
[16] Qingkai Fang, Yan Zhou, Shoutao Guo, Shaolei Zhang, and Yang Feng. Llama-omni2: Llm- based real-time spoken chatbot with autoregressive streaming speech synthesis, 2025. URL https: / aariiiiiv org/ abs/2505, 02625
14

---

[17] Aohan Zeng, Zhengxiao Du, Mingdao Liu, Kedong Wang, Shengmin Jiang, Lei Zhao, Yuxiao Dong, and Jie Tang. Glm-4-voice: Towards intelligent and human-like end-to-end spoken chatbot, 2024. URI. htt.httts://axxtti
org/abs/ /tabs//2412.. 2412 02612
[18] Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. Llama- omni: Seamless speech interaction with large language models, 2025. URL hattps://arxx
org/abs/22 at//a////249906666
[19] Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. In Interspeech 2018, pages 1086-1090, 2018. doi: 10.21457/miersppcchh211. 1929.
[20] Changhan Wang. Anne Wu, Jiatao Gu, and Juan Pino. Covost 2 and massively multilingual speech translation. In Interspeech 2021, pages 2247-2251, 2021. doi: 10. 10.21457/Intesspecchh 2021-2027.
[21] Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James R. Glass. Joint audio and speech understanding. IEEE Automatic Speech Recognition and Understanding Workshop, ASRU 2023, Taipei, Taiwan, December 16-20, 2023, pages 1-8. IEEE, 2023, doi: 10.1109/ASRUS576442222. 10389742. URLhttps:://ot org/ 1109 ASRU57964 2023 10389742
[22] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. URL https
//arxiv.org/ abs /212. 04356
[23] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pages 369-376, 2006.
[24] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang. Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report, 2023.
URL https: //ap:::///xxxivv org/ abs abs/aa/......////// 16609
[25] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
[26] Shoutao Guo, Shaolei Zhang, and Yang Feng, Simultaneous machine translation with tailored reference, 2023. URL httpp:://aaxi...rr/////////////e a310 13588
[27] Yi Ren, Jinglin Liu, Xu Tan, Chen Zhang, Tao Qin, Zhou Zhao and Tie- Yan Liu. Simul- Speech: End-to-end simultaneous speech to text translation. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3787-3796. Online, July 2020. Associ- ation for Computational Linguistics. doi: 10. 122255//12000...mmaaaa.. //aclanthology. org/ 20200 acl-main. 350/
URL https
[28] Shaolei Zhang and Yang Feng. End-to-end simultaneous speech translation with differentiable segmentation. In Anna Rogers, Jordan Boyd- Graber,, and Naoaki Okazaki. editors, Findings of the Association for Computational Linguistics: ACL 2023, pages (659-7680, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. (0.118///vvl 1/2023/ www.:/.iiiinnnall..cc URI
findinginiiinsii acl. 485/
[29] Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai "Helen" Li, and Yiran Chen. Speechprune: Context-aware token pruning for speech information retrieval,
2024. URL URL https: //tpx//xvvvv///////////////avxv/ org/ http/////22222.... 12009.
15

---

[30] Shoutao Guo. Shaolei Zhang, and Yang Feng. Glancing future for simultaneous machine translation. In ICASSP 2024 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), page 1336- 1 1390. IEEF, April 2024, doi: 10. 100109//ssp44888.. 1024.104 URL. http:ttp::// ortp:////oooooooo/..//oo http:///..1100/AAAAPAA00000
doi
2024. 10446517
[31] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu. Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, Yuxiao Dong, Jie Tang, and Juanzi Li. LongBench: A bilingual, multitask benchmark for long context understanding. In Lun-We wei Ku, Andre Martins, and Vivek Srikumar, editors. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3119-3 5119-3137, Bangkok, Thailand, August 2024. Association for Computational Linguistics, doi: URL
172.
172/
[32] OpenAl. Hello gpt-4o, 2024. URL
htt.a/aai...oonnddee////illlllea gpt-
[33] Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur: Librispeech: An asr corpus based on public domain audio books. 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 5206-5210, 2015. doi: 10.1109/ ICASSP.201..717896
[34] Vineel Pratap, Qiantong Xu, Anuroop Sriram, Gabriel Synnaeve, and Ronan Collobert. Mls: A large-scale multilingual dataset for speech research. In Interspeech 2020. ISCA, October 2020. doi: 10.2 4114tt//neers 202.hh.02222222 2020-2826. httLhhtt////e doi 10. In97/IIntrrppech 2020-2826
[35] Yuan Gong, Alexander H. Liu, Hongyin Luo, Leonid Karlinsky, and James Glass. Joint audio and speech understanding. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pages 1-8, 2023. doi: 10.100/AASRUS59962223...88353
[36] Zihan Zhao, Yiyang Jiang, Heyang Liu, Yanfeng Wang, and Yu Wang. Librisqa: A novel dataset and framework for spoken question answering with large language models, 2024. URL
https://arxiv org/ abs/ 10390
[37] Rosana Ardila, Megan Branson, Kelly Davis, Michael Kohler, Josh Meyer. Michael Henretty, Reuben Morais, Lindsay Saunders, Francis Tyers, and Gregor Weber. Common voice: A massively-mulilinguaal speech corpus. In Nicoletta Calzolari, Frédéric Bécher, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 4218-4222. Marscille, France, May 2020. European Language Resources Association. ISBN 979-10-95546-34-4 URL https: /attts://aclanthooooo
org/ 20///0220.... 1rec- 1.520/
[38] Qian Yang, Jin Xu, Wenrui Liu, Yunfei Chu, Ziyue Jiang, Xiaohuan Zhou, Yichong Leng. Yuanjun Lv, Zhou Zhao, Chang Zhou, and Jingren Zhou. AlR-bench: Benchmarking large audio-language models via generative comprehension. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annuul Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1979-1998, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v 11118655/111222...2 aal.244.l-ooon...... 109. URL www.hhtts///// btattps://acaantholl.ooo org/ 2024. acl long. 109/
[39] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. MELD: A multimodal mulli-party dataset for emotion recognition in conversa- tions. In Anna Korhonen, David Traum, and Lluis Mårquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 527-536. Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 111111661//11-1111. 1050. URL https: ftttp:://aclantho..e
org 1050/
[40] Yichen Lu, Jiaqi Song, Chao-Han Huck Yang, and Shinji Watanabe. Fastadasp: Multitask- adapted efficient inference for large speech language model, 2024. URI. httttttts///aiii org htg/abss/211..0000
16

---

[41] Guoguo Chen, Shuzhou Chai, Guanbo Wang, Jiayu Du, Wei-Qiang Zhang, Chao Weng, Dan Su, Daniel Povey, Jan Trmal, Junbo Zhang, Mingjie Jin. Sanjeev Khudanpur, Shinji Watanabe, Shuaijiang Zhao, Wei Zou, Xiangang Li, Xuchen Yao, Yongqing Wang, Yujun Wang. Zhao You, and Zhiyong Yan. Gigaspeech: An evolving, multi-domain asr corpus with 10,000 hours of transcribed audio, 2021. URL
org abtp///s...//////////////// 22t.//000060009999
[42] Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai "Helen" Li, and Yiran Chen. Speechprune: Context-aware token pruning for speech information retrieval, 2025. URL https::/axxii...ttt//// http:/iw..rrgbbb/242122 12009
[43] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding, 2023. URL https:///arxiiiii orgg/ abs 2ats/214.000 09864
[44] bloc97. Ntk-aware scaled rope allows Ilama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023.
[45] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[46] Xiaohan Ding, Xiangyu Zhang, Jungong Han, and Guiguang Ding, Scaling up your kernels to 31x31: Revisiting large kernel design in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1 1993--11975 June 2022.
[47] Rohit Prabhavalkar, Takaaki Hori, Tara N. Sainath, Ralf Schlüter, and Shinji Watanabe. End-to- end speech recognition: A survey, IEEE/ACM Transactions on Audio, Speech, and Language Processing. 32:325-351, 2024. doi: 10.1 10.1009TAALLL2223338835.
[48] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Meng Zhao, Yifan Zhang, Shaoqi Dong. Xiong Wang, Di Yin, Long Ma, Xiawu Zheng, Ran He, Rongrong Ji, Yunsheng Wu, Caifeng Shan, and Xing Sun. Vita: Towards open-source interactive omni multimodal 11m, 2024. URL https: //axxaaavvvvv.////////aa//aaaa/a org/
05211
[49] Paul K. Rubenstein, Chulayuth Asawaroengchai, Duc Dung Nguyen, Ankur Bapna, Zalán Borsos, Félix de Chaumont Quitry, Peter Chen, Dalia El Badawy, Wei Han, Eugene Kharitonov, Hannah Muckenhirn, Dirk Padfield, James Qin, Danny Rozenberg. Tara Sainath, Johan Schalk- wyk, Matt Sharifi, Michelle Tadmor Ramanovich, Marco Tagliasacchi, Alexandru Tudor, Miha- jlo Velimirović, Damien Vincent, Jiahui Yu, Yongqiang Wang, Vicky Zayats, Neil Zeghidour, Yu Zhang, Zhishuai Zhang, Lukas Zilka, and Christian Frank. Audiopalm: A large language model that can speak and listen, 2023. URL https://aaxii......./a orgz/iiivorrrre abs/ 2206. 12925
[50] Alexandre Défossez, Laurent Mazaré, Manu Orsini, Amélie Royer, Patrick Pérez, Hervé Jégou, Edouard Grave, and Neil Zeghidour. Moshi: a speech-text foundation model for teme-time dialogue, 2024.
org/ aaa/ab/s/22210 20t./2410.0000 00037
[51] Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian. Extending context window of large language models via positional interpolation, 2023. URL htttps/////a org/abs/2300 15595
[52] Nir Ratner, Yoav Levine, Yonatan Belinkov, Ori Ram, Inbal Magar, Omri Abend, Ehud Karpas, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. Parallel context win- dows for large language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61 Annual Meeting of the Association for Computational Lin- guisties (Volume 1: Long Papers). pages ciation for Computational Linguistics. doi: 10.18663////2223.acl acl-long.332.. URL https // //aclanthology.ooom org
Toronto, Canada, July 2023, Asso-
www.//23....../.... 2023. acl 1ong.loog..3 352/
[53] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre. Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défosse7, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code llama: Open foundation models for code, 2024. URL : aariv.iivva...ii//aaaaaaaaavvaam org/ 20../ 12950
17

---

[54] Jingyang Yuan, Huazuo Gao, Damai Dai, Junyu Luo, Liang Zhao, Zhengyan Zhang, Zhenda Xie, Y. Wei, Lean Wang. Zhiping Xiao, Yuqing Wang. Chong Ruan, Ming Zhang, Wenfeng Liang, and Wangding Zeng. Native sparse attention: Hardware-aligned and natively trainable sparse attention, 2025. URL htt.httss//axii.orgg/gaoa/
11089.
[55] Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang. Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye. Yanting Zhang, Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding, 2024. URL https://ares
org/ abs 16449
[56] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. prumerge Adaptive token reduction for efficient large multimodal models, 2024. URL/https://arre
abs/
2403. 15388
[57] Emiru Tsunoo, Hayato Futami, Yosuke Kashiwagi, Siddhant Arora, and Shinji Watanabe. Decoder-only architecture for speech recognition with ctc prompts and text data augmentation, 2024. URL BRRthttps:/// pettps://arxiv.comm wttp:/iwwrrrraaa/////e/// 2339
[58] Marco Gaido, Mauro Cettolo, Matteo Negri, and Marco Turchi. Cte-based compression for direct speech translation, 2021. URL htlphtts:///axii..rrr wrtp./.rrrraaas/12200177777
A Description of LongSpeech-Eval
LongSpeech-Evall is novel benchmark we propose for evaluating the long-speech understand- ing capabilities of Large Speech-Language Models (LSLMs). This benchmark presents a spoken Question-Answert (QA) task, challenging LSLMs to answer questions based on the extended speech inputs. The dataset comprises 164 samples, with an average speech duration of 132.77 seconds and maximum duration reaching 1000 seconds.
The foundation for LongSpeech- Eval is the MultiField-En and NarrativeQA subsets from LongBench, an established long-context understanding benchmark. MultiFiel-EEn is a single-document QA dataset encompassing diverse domains, with questions and answers meticulously annotated by Ph.D. students. NarrativeQA consists of long stories along with questions posed to test reading comprehension. Our methodology for creating LongSpeech-Eve involves a rigorous multi-step process.
We first employ Llama3.1 1--11----nstruut to filter out samples containing numerous formulas or non-English characters, ensuring the dataset's 's suitability for speech synthesis and comprehension. GPT-4o B2 is utilized summarize and polish the documents into more natural spoken forms, enhancing their suitability for speech synthesis. We then reapply Llama3. Insttrrt to eliminate any samples where questions could not be adequately answered based on the spoken- form documents, ensuring the validity of the samples. Finally, we leverage the Text-to-Speech (TTS) model Orca" 2 to synthesize speech from the refined spoken-form documents.
The resulting dataset combines synthesized speech with corresponding questions and answers, forming comprehensive spoken QA benchmark.
B Details of Dataset
In this section, we provide a detailed description of the training and testing data.
B.1 Training Dataset
Our training method is divided into two stages.
In the first stage, we train the CTC decoder using the CTC loss [22]. During this stage, only ASR data are used, including 960 hours of LibriSpeech B3 data and 3k hours of data sampled from MLS dataset (3].
httpe///auginnffaae.co//ttttttt 1lamaamaaamaaaaaaa-ammm 3.1 13.1-700 70----7----- Instruct
httpss//ggtcab..cc///..cooooooo/
18

---

In the second stage, we utilize our proposed dynamic compression training approach to train the LLM. For this stage, we use spoken QA datasets, which come from three datasets: OpenASQA [35]. LibriSQA [36], and Common Voice [37] For OpenASQA, we select the Open-Ended Speech AQA subset, which contains 5.9k hours of speech data. The questions and answers in this dataset are generated by GPT-3.5-Turbo and cover aspects such as spoken text, speaker gender, age, style, and emotion. For LibriSQA, we use the complete training set, which contains 360 hours of training data. The questions and answers in this dataset are generated by ChatGPT, with the speech data sourced from the LibriSpeech train-clean-3e subset [36.. For the Common Voice ASR dataset, we transform it into spoken QA format to enhance training set. First, we use ChatGPT generate 200 diverse speech transcription instructions. For each ASR sample, we randomly select one instruction as the question and use the ground-truth transcription as the answer, resulting in 1.7k hours of training data.
B.2 Evaluation Dataset
For testing, we evaluate our method on short-speech spoken QA, long-speech spoken QA, and ASR tasks. The long-speech spoken QA task corresponds to the LongSpeech- benchmark, which is introduced in Appendix
For short-speech spoken QA, we utilize three test sets: the speech_QAt jemocapp (AIR-Bench) [8], the LibriSQA test set [36, and the LibriTTS test subset from OpenASQA (35) The speech_Q2. _e__eoocppp dataset comes from the The LibriSQA test set includes 2620 samples. For the LibriTTS test subset, we select samples corresponding to the LibriTTS test-clean set from OpenASQA, keeping only the 417 samples with a speech duration longer than 15 seconds
benchmark and contains 200 samples.
test set. All test sets are under 30s in duration.
For spoken dialogue understanding, we evaluate the inference efficiency of our method using speech_ Lirrechh_illuue_____ QA_eisheeeeeeaeee subset (38) from AIR-Bench. This subset contains 200 samples. For this task, our method is directly applied to vanilla Qwen2-Audio, which has only undergone the first training phase of our method. This setup allows to assess the effectiveness of our approach without requiring the training of LSLMs.
For the ASR task, we use the LibriSpeech [33] test-clean, test-other, and GigaSpeech 41 test set as our evaluation datasets. For convenience evaluation, we convert these datasets into the spoken QA format, where the instruction for each sample is: "Transcribe the speech to text without explanation:"
For emotion recognition task, We leverage the MELD dataset 33 to benchmark our method against other efficiency method [40] under diverse efficiency scenarios. FastAdaSP lowers the inference costs of Qwen2-Audio through the layer-wise dynamic reduction of speech representations within the LLM's architecture. We compare our method with FastAdaSP to demonstrate the advantage of our method in retaining information. Since we could not find the specific prompt in FastAdaSP [40], we utilize the following prompt: "Given the Choices: [Anger, Disgust, Fear, Joy, Neutral, Sadness, Surprise]. What is the emotion in the audio?"
C Experimental Details
In this section, we introduce the NTK-RoPE method in greater detail and outline the system configu- ration of Fastl LongSSch....... Our Fastl onngSpehh primarily leverages Qwen2-Audio. Additionally, we apply our approach to a vanilla Qwen2.5-Ome phase. This serves to validate that our method can achieve competitive performance without altering the inherent capabilities of the model, while also demonstrating its generalizability.
[15] model that has only undergone the first training
NTK-RoPE extends the speech window of Qwen2-Audio to match the context length of its LLM by adjusting the Rotary Position Embedding (RoPE). However, some samples in our LongSpeech- Eval may still exceed this extended context length. To handle these special cases, we apply our iterative fusion strategy to reduce the sequence of speech representations to fit within the prescribed context length.
We then delineate the configuration of Fastl LongSeeh........ The training process in two stages. In the first stage, we utilize ASR data to train the CTC Decoder, which is feed-forwar network with
19

---

Table 6: Settings of FastLonggSpeech.
Hyperparameters
Settings
hidden_ding dim output output_dim dim
4096
Model
10000
CTC Decoder
per_device_batch.
16
Training Details
tearning_rate
2e-5
Ir schheculler
cosine
Quen2-Audio-7B-Innttrre
lora r
128
(ora_alphi
256
LoRA
lora Iora_drop dropout
0.05
LSLM
Cora_target_a trrget_modules
p_pooj k proj, v v_proj, o o_proj
per_device_batce tearning_rate rate Ir scheduler
16
Training Details
2e-4
cosine
one hidden layer. We use the SentencePiece toolkit to construct the vocabulary for the training of the CTC decoder. This vocabulary is extracted from the ASR dataset. The second stage focuses on training the LLM within the LSLM using Spoken QA data. Both training stages leverage DeepSpeed ZeRO-2 for optimization. Table 6 provides additional training and configuration details.
D Evaluation Template
In this section, present the prompt template used for evaluating LSLMs. As shown in Figure 3 the template will be employed by the LLM to score the responses generated by the LSLMs. This scoring template is used to evaluate long-speech and short-speech spoken QA tasks.
E
Applicability of Our Method to Vanilla LSLMs
In our FastLongSpeech framework, we extend LSLMs for long-speech processing by adopting iterative fusion strategy and dynamic compression training approach. As highlighted in subsection 5.2 our method not only excels in long-speedl tasks but also achieves good balance between per- formance and efficiency in short-speech scenarios. Therefore, this prompts us to investigate whether vanilla LSLMs benefit from method effectively balance computational efficiency and generation quality, thus meeting diverse requirements across various speech processing applications. To this end, apply the iterative fusion strategy directly to the vanilla Qwen2-Aude Audio and vanilla Qwen2.5-Ommi model.
To demonstrate the effectiveness and robustness of our method, we first extend our experiments to spo- ken dialogue understanding task. For this task, conduct experiments on vanilla Qwen2-Audio using speech dialogue QA fisher of AIR-Bench (38). As shown in Table our method effectively balances per- formance and inference efficiency. Notably, lower compression ratios (I = 200), our approach demon- strate comparable performance the vanilla Qwen2- Audio model with a 50% reduction in computational costs. Moreover, even higher compression ratio of 15x (L=50), our method still maintains robust per-
Table 7: The experiment results on the speech_diieg SecchhddlluueeO_S1 _Q__sshher "Baseline" denotes vanilla Qwen2-Audio and "Ours" denotes applying iterate fusion strategy to vanilla Qwen2-Audio.
fisher subset, where
Method
Score (1)
TFLOPs (L)
Baseline
3.95
11.76
Ours (L=400)
4.13
8.25
Ours (L=200)
3.92
5,46
Ours (L=100)
3.62
4.06
Ours (L=50)
3.16
3.35
20

---

need your help evaluate the performance of several models in the speech interaction scenario. Given segment of speech and a related text question, the model needs to understand the speech and the question, and provide a text answer. Your task to rate the model's responses based on the question [Question], ground-truth response [Ground-truth] and the model's response [Responsel. Please evaluate the faithfulness of the model's responses, and provide score for each on scale of to 5.
Faithfulness (1-5 points):
5 points: The model's response is completely faithful to the ground boound-truth response and answer the questions, covering all key information and expressed clearly and accurately.
points: The model's response is generally faithful to the ground-truth response and answer main questions, missing few non-essential details, but the overall meaning is correct.
3 points: The model's response is somewhat faithful to the ground-truth response and answers the question partially, but the expression not accurate enough or some important information omitted, which may lead misunderstandings.
2 points: The model's response differs significantly from the ground-truth response and doesn't answer the questions, with key content missing or expressed vaguely, affecting comprehension. point:
The model's response completely unfaithful to the ground-truth response and doesn't answer the questions all, containing errors being entirely irrelevant, failing convey the required information.
Below are the question, ground-truth response and model's response:
{question] ### (Ground-truth): (ground _
### [Question]:
### |Response|:
[response]
After evaluating, please output the scores in JSON format: (f"fatthfulnessi": faithfulness score). You don't need to provide any explanations.
Figure 3: The prompt template for the LLM to evaluate the response of LSLMs.
formance. These findings underscore the efficacy and versatility of our iterative fusion strategy.
We further extend our approach to vanilla Qwen2.5- Omni [15], a model exhibiting superior capabili- ties compared to Qwen2-Audio. Speecfficaly,, we benchmark the performance of Qwen2.5-Omni against Qwen2-Audio on the speech_QQ OA_jemocap subset of AIR-Bench. The results in Table g indicate that Qwen2.5-0mmt owing its stronger speech capabili- ties, demonstrates superior performance across various compression ratios. This demonstrates that method achieves superior performance more capable LSLMs, highlighting its generalizability.
Table 8: The experiment results on the speech QA iemocap subset.
L
Qwen2-Audio
Qwen2.5-Omni
750
3.68
3.82
400
3.69
3.82
200
3.67
3.75
F Applicability of Our Method to Other Tasks
Additionally, we extend our experimental evaluation to the emotion recognition task and employ MELD dataset [3]] For this task, we benchmark our method against FastAdaSP 44), an approach designed for enhancing inference efficiency of Qwen2-Audio, We adopt the iden- tical experimental setup as in the FastAdaSP, comparing performance under the same inference reduction settings. As depicted in Table 9 our method not only achieved superior performance but also reduced inference cost by 50% compared to SearAAdSP-pparse This underscores
Table 9: The experiments the MELD dataset, where the results are reported in the configuration with a 50% reduction in inference cost. The performance is mea- sured with accuracy metric.
Method
Accuracy (%) (t)
FastAdaSP Langeongg
52.14
52.95
21

---

Table 10: The experiments on the SPIRAL-H dataset. The performance is measured with accuracy metric.
Method
Prune Rate (1)(T)
Accuracy (%) (1)
SpeechPrune FasteLongSpeech (L = 750)
60.00
63.77
65.88
76.81
Table 11: The performance of SastLLngSSeecch with varying context lengths.
L
200
400
750
1200
4000
Score (t)
2.47
2.90
3.55
3.66
3.59
the effectiveness of approach in preserving crucial information. Furthermore, our method is complementary to the method [40] and holds potential for further improving inference efficiency through integration, prospect leave for future investigation.
Beyond emotion recognition, we also conduct additional experiments on the SPIRAL-H dataset, benchmark designed for long speech information retrieval. On this dataset, we follow the experimental setup [2 and compare model performance under similar speech embedding pruning rates. As shown in the TO our method achieves better performance with fewer speech embeddings, demonstrating its superiority and efficiency in modeling long speech inputs.
G Extending Maximum Context Length
We also explore the performance of FastLongSpeechh the LongSpeech-Evar dataset with varying context lengths L. The results are shown in Table T1 When T. is less than 750, the model exhibits increasing performance with longer context, is trained under dynamically varying compression ratios this range. When equals 1200, although the model is not explicitly trained for this length, still achieves strong performance, indicating good generalization beyond the training regime. When increases to 4000, performance slightly declines. This is expected, as the model is not exposed such long contexts during training, despite having larger speech context window. We think our SpneSeecch can achieve better performance with longer effective context length as the long-speect training data becomes more available.
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the Abstract and Introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In the Limitations section.
23

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
In the Section 4 and Appendix A. D.
Guidelines:
The answer NA means that the paper does not include experiments.

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
In the supplemental material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ /ttppp.cc/pbllc/guideeee http://oeeubiiissioooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
25

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: In the Appendix C.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
Guidelines: We don't 't provide the experiments for statistical significance.
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the compute resources of GPU.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
26

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ltttps:/neurips..
Answer: [Yes]
Justification: The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This is not involved with the impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
27

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: They are mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
28

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy /ettps://neurr cc/ //onfffrencee/22255555 for what should or should be described.
29


Reviews:
[{"review_id": "qo6OgaNE1n", "reviewer": "Reviewer_BctC", "summary": "This paper introduces FastLongSpeech, a framework designed to extend Large Speech-Language Models (LSLMs) for efficient processing of long-form speech without requiring dedicated long-speech training data. The approach builds on Qwen2-Audio and incorporates two main technical contributions: (1) an iterative fusion strategy that compresses long speech sequences by merging similar adjacent frames based on content density measured via CTC output, and (2) a dynamic compression training method that adapts models to varying compression ratios using only short-speech data. The authors also contribute LongSpeech-Eval, a benchmark for evaluating long-speech understanding capabilities. Experimental results demonstrate that FastLongSpeech achieves competitive performance on both short and long speech tasks while significantly reducing computational costs compared to baseline approaches.", "questions": "Comparison with Established Methods: Why weren't established long-sequence modeling techniques like chunking with overlap, hierarchical attention, or other compression methods compared as baselines? A comparison with these approaches would strengthen the paper's claims about the effectiveness of the proposed method.", "limitations": "The authors provide a brief limitations section but could expand on several critical points.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n- **Practical Significance**: The work addresses a genuine limitation in current LSLMs - the inability to efficiently process long-form speech. This is practically important for applications like lecture transcription, long-form audio analysis, and extended conversations.\n\n- **Technical Innovation**: The iterative fusion strategy is well-motivated, using content density from CTC outputs to guide compression decisions. The dynamic compression training approach cleverly leverages short-speech data to transfer capabilities to long-speech scenarios without requiring expensive long-speech datasets.\n\n- **Comprehensive Evaluation**: The experimental setup is thorough, covering multiple tasks (spoken QA, ASR, emotion recognition) and providing both performance and efficiency metrics. The creation of LongSpeech-Eval benchmark is a valuable contribution.\n\n- **Clear Methodology**: The paper clearly explains the iterative fusion algorithm and training procedure, making the approach reproducible.\n\n**Weaknesses:**\n\n- **Limited Baseline Comparisons**: The paper lacks comparison with other established long-sequence modeling techniques beyond NTK-RoPE. Methods like sliding window attention or other compression strategies from the broader literature could provide stronger baselines.\n\n- **Synthetic Benchmark Limitations**: LongSpeech-Eval relies on synthesized speech from text documents, which may not capture the complexities and variabilities of natural long-form speech. The benchmark's validity for real-world scenarios is questionable.\n\n- **Content Density Assumption**: The approach assumes CTC-based content density is a reliable indicator of information importance, but this may not hold across all speech types (e.g., prosodic information, speaker characteristics, emotional content may be important but have low text density).\n\n- **Limited Scale Evaluation**: Experiments are conducted only on 7B parameter models. The scalability and effectiveness of the approach on larger, more capable models remains unclear.", "was_revised": true, "final_justification": "Thank you for the response. I've read the authors' responses and appreciate their reflection on the paper. I will keep my score and maintain my recommendation.", "rebuttal": "Thank you very much for your insightful comments and the time you spent on reviewing.\n\n>Consider comparing with methods like chunking with overlap, sliding window attention, to provide more comparisons? \n\nThank you for your valuable comments. There are two main reasons why our comparisons primarily focus on compression-based and expansion-based methods for extending speech LLM input windows, rather than the mentioned established text-based methods.\n\nFirst, the complexity of speech inputs poses unique challenges. Unlike text, the speech inputs are inherently more sparse in information density, and sentence boundaries are often ambiguous and difficult to determine. **When directly applying traditional long-sequence modeling methods involved chunking, the segmentation process can lead to semantic discontinuity and misinterpretation**.\n\nSecond, the traditional long-context methods are designed on top of already strong language models, aiming to further extend their ability to handle ultra-long textual sequences. In contrast, the long-speech capabilities of current speech LLMs are still limited, which capture much less information per unit input than their text counterparts. Therefore, **our work instead focuses on enhancing the inherent speech capability of speech LLMs themselves**. Furthermore, **our approach is complementary to your proposed techniques and can potentially be combined with them in future work to further improve long speech modeling capabilities**.\n\nFollowing your suggestions, **we also implement two additional variants to provide a broader set of baselines**:\n\n**Word-Bound**, which uses CTC to detect word boundaries and then aggregates representations belonging to the same word.\\\n**Blank-First**, which prioritizes merging segments with blank (non-linguistic) predictions.\n\nAs shown in the table below, our method achieves superior performance, demonstrating its effectiveness in improving long-context modeling for speech.\n| LongSpeech-Eval | | | |\n|:--:|:--:|:--:|:--:|\n| **Method** | FastLongSpeech | Word-Bound | Blank-First |\n| **Score** ($\\uparrow$) | 3.55 | 2.21 | 3.067 |\n\n>Synthetic Benchmark may not capture the complexities and variabilities of natural long-form speech?\n\nThank you for your insightful question. To ensure the quality of the speech in our proposed LongSpeech-Eval benchmark, we take several measures to guarantee the quality of the benchmark.\n\nOn the one hand, we apply a filtering process using LLMs to remove samples that are unsuitable for speech synthesis. For the remaining samples, we leverage GPT-4o, an advanced language model, to rewrite and improve the textual content. Finally, we conduct manual verification to further ensure the quality and fluency of the text.\n\nOn the other hand, we employ a high-quality commercial TTS system, Orca, to perform speech synthesis. Orca supports long-form speech generation and allows fine-grained control over the speech. Therefore, we can obtain good complexities and variabilities of long-form speech.\n\n>Content density assumption may not hold across all speech types (e.g., prosodic information, speaker characteristics, emotional content)?\n\nThank you for your insightful comments. Compared to paralinguistic features such as prosody, textual content is inherently more information-dense and quantifiable. This characteristic motivates our design choice to develop a speech compression strategy guided by textual content density.\n\n**It is worth emphasizing that while our approach leverages content-aware compression, it fundamentally differs from traditional cascaded pipelines by preserving significant paralinguistic information**. The experimental results of FastLongSpeech on speech emotion understanding tasks (speech_QA_iemocap and MELD) demonstrate superior performance compared to alternatives, confirming that our method effectively retains and utilizes crucial paralinguistic cues.\n\nLooking forward, as speech processing research advances, our framework can be readily extended to incorporate more diverse and advanced speech information indicators. This flexibility creates opportunities for further improvements in capturing both linguistic content and paralinguistic elements in long-form speech processing.\n\n>Experiments are conducted only on 7B parameter models? \n\nThanks for your suggestions. Given the real-time and efficiency requirements of speech processing, current open-source speech LLMs are typically around the scale of 7B parameters. However, our proposed method is not limited to a specific model. While we primarily conduct experiments on Qwen2-Audio, we have also demonstrated its effectiveness on Qwen2.5-Omni in Table 8, which supports the generalizability of our approach.\n\nAs computing resources continue to advance, we expect larger-scale speech LLMs to become available. We plan to extend and validate our method on these larger models in future work."}, {"review_id": "oVLyuCHtG6", "reviewer": "Reviewer_CJSP", "summary": "This paper proposes an extractor of speech embeddings by compressing lengthy and sparse speech representations into more compact forms using a iterative fusion based on CTC scores. With the compact representations, speech language models are enabled to handle long speech context.", "questions": "The effect of long context is not shown, how will different lengths of speech context affect the performance? It is expected that longer context would show better advantage of the proposed FastLongSpeech.\n\nThe distributions of lengths of speech context in LongSpeech-Eval should be provided so that the audience have better understanding of effectiveness of the proposed method.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) The proposed FastLongSpeech outperforms other compared baselines on short-speech and long-speech spoken QA tasks in terms of scores using LLM as a judge. Different lengths of compact representation consistently achieve better scores.\n2) A long-speech understanding benchmark is developed using synthetic speech and used to evaluate the propose approach.\n\nWeaknesses:\n1) The evaluation on long speech context is based on synthetic speech, then how to ensure the synthetic questions and corresponding answers are reflecting correct paralinguistic characteristics? How does LongFastSpeech model the paralinguistic information in the long speech context. It is unclear how does the developed LongSpeech benchmark control the paralinguistic information during generation.\n2) Sec.3.2 is the core part of the paper, but it is quite confusing to me. Eq. 5 is wrong? if T(m)=2L-1, then T(m+1)=$\\lfloor{(2L-1)/2}\\rfloor$ but not T(m+1)=L, according to L132-139. Also since the compression is based on the content density as defined by Eq 3, T(m+1) is not always $\\lfloor T(m)/2\\rfloor$ when T(m)>2L?\n3) The LongSpeech-Eval benchmark is developed based on long document-based QA, however, in practice, such kind of human-LLM interaction scenarios may be limited.\n4) There is a lack of comparison to a baseline that use ASR transcription of the long speech context as input, and comparison to FastAdaSP [37] and Speechprune [27] on the LongSpeech-Eval dataset.", "was_revised": true, "final_justification": "Thanks for addressing my previous questions, with clear explanation and new experimental results to support the rebuttal. I have increased the score.", "rebuttal": "Thanks for your valuable feedback and suggestions.\n\n>How does the developed LongSpeech benchmark control the paralinguistic information during generation? \n\nYour question is insightful. Our LongSpeech-Eval is derived from LongBench [1], a long-text understanding benchmark that requires LLMs to answer questions based on extended context. To construct LongSpeech-Eval, we retain the original questions and answers from LongBench while filtering, rewriting, and subsequently synthesizing the context into speech.\n\nCompared to state-of-the-art TTS models such as CosyVoice 2.0 [2], Orca natively supports long-form speech synthesis and offers advantages in semantic consistency and text accuracy. In contrast, CosyVoice 2.0 often omits parts of the text when synthesizing from long input passages. Moreover, Orca also supports fine-grained control over speech, and its performance has been validated in the paper [3]. During inference, Orca leverages contextual information to identify and incorporate paralinguistic cues. Therefore, we choose Orca as our speech synthesis model.\n\n[1] Bai et al. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. In ACL 2024.\\\n[2] Du et al. CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models. In arXiv:2412.10117.\\\n[3] Fang et al. LLaMA-Omni: Seamless Speech Interaction with Large Language Models. In ICLR 2025.\n\n> How does the FastLongSpeech model the paralinguistic information in the long speech context?\n\nIn the context of speech LLMs, the audio encoder extracts speech embeddings that inherently contain rich paralinguistic information, such as prosody and emotion. Our proposed method is designed to enable efficient long-context modeling of speech inputs. During the iterative fusion strategy, we perform content-guided integration of the speech embeddings based on their textual relevance, but crucially, we do not discard paralinguistic information. **This process results in compressed representations that preserve essential textual content while still retaining paralinguistic features**.\n\nAs shown in Tables 8 and 9, our method achieves superior performance on speech_QA_iemocap and MELD, which are benchmarks focused on emotion recognition and emotional question answering. These results demonstrate that **although our method is guided by textual content, it effectively operates on speech embeddings, thereby maintaining a substantial amount of paralinguistic information and contributing to the model’s strong performance on affective tasks**.\n\n>Section 3.2 may be confusing to you, such as Eq. (5)?\n\nWe appreciate the opportunity to provide a more detailed explanation. Section 3.2 of our paper describes the proposed iterative fusion strategy, which progressively compresses the long-form speech embeddings output by the audio encoder over multiple iterations. The goal is to reduce the sequence length step by step until it reaches the target length $L$.\n\nIn each iteration, we first determine the input length $T(m)$ and output length $T(m+1)$ of the speech embeddings using Equation (5). **The value of $T(m+1)$ depends solely on $T(m)$ and is not determined by Equation (3) or the discussion in Lines 132–139**. **Once $T(m+1)$ is computed, the number of speech embeddings to be reduced in this iteration is calculated as $r(m) = T(m) - T(m+1)$**. Next, we select $r(m)$ pairs of adjacent frames for merging based on frame-wise similarity, as defined in Equation (4). The selected adjacent frames are grouped into spans, and each span is compressed into a single frame via weighted pooling using the content density in Equation (3). **After processing all spans, the number of remaining frames becomes $T(m+1)$**.\n\nTherefore, **when $T(m) = 2L - 1$, the current iteration is guaranteed to be the final one, and $T(m+1)$ must equal $L$, ensuring that the output compressed representation has the desired final length $L$**.\n\n>The human-LLM interaction scenarios in LongSpeech-Eval are potentially limited?\n\nThe primary goal of LongSpeech-Eval is to assess the long-form speech understanding capability of speech LLMs, with an emphasis on broad domain coverage.\n\nTo this end, we build LongSpeech-Eval upon the widely adopted LongBench [1] benchmark for long-text understanding, specifically utilizing its Single-Document QA subset. **This subset spans a diverse range of domains, including legal documents, government reports, encyclopedia entries, academic lectures, and narratives**. In practical applications, these types of content correspond to realistic scenarios such as academic presentations, public speeches, and multi-party meetings, where quick comprehension and response to long-form speech is essential.\n\nWe acknowledge the importance of continuously expanding coverage and will further enhance the diversity of scenarios in future iterations of LongSpeech-Eval.\n\n[1] Bai et al. LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding. In ACL 2024.\n\n>Comparison to a baseline that uses ASR transcription of the long speech context as input?\n\nThank you for your suggestions. We implement a cascaded approach as a baseline: specifically, we first use Whisper-Large-V3 [1] to transcribe the audio, and then pass the resulting context text to Qwen-7B-Chat (since Qwen2-Audio-7B is based on Qwen-7B-Chat) for question answering.\n\nWe compare this cascaded baseline with our proposed FastLongSpeech method on LongSpeech-Eval, and the results are shown in the table below:\n\n| LongSpeech-Eval | | |\n|:--:|:--:|:--:|\n| **Methods** | **Score** ($\\uparrow$) | **Time (s)** ($\\downarrow$) |\n|  Cascaded Method | 3.75 | 17.23 (ASR) + 1.38 (LLM) | \n|  FastLongSpeech ($L$=750) | 3.55 | 1.47 |\n| FastLongSpeech ($L$=1200) | 3.66 | 2.52 | \n\nAs shown above, while our method slightly underperforms the cascaded approach in response quality, it achieves over a 7× speedup, highlighting its significant efficiency advantage for long speech processing. This quality result is not surprising, as it has also been observed that current speech LLMs tend to lag behind cascaded pipelines in certain content-based speech understanding tasks [2].\n\nWe further evaluate both methods on the speech_QA_iemocap dataset, which involves emotional question answering. The results are as follows:\n\n| speech_QA_iemocap | |\n|:--:|:--:|\n| **Methods** | **Score** ($\\uparrow$) | \n| Cascaded Method | 3.59 |\n| FastLongSpeech ($L$=100) | **4.15** |\n\nIn this case, our method significantly outperforms the cascaded approach, highlighting the strength of an end-to-end framework in capturing emotional and paralinguistic cues from speech.\n\n[1] Radford et al. Robust Speech Recognition via Large-Scale Weak Supervision. arXiv:2212.04356.\\\n[2] Wang et al. AudioBench: A Universal Benchmark for Audio Large Language Models. In NAACL 2025.\n\n>Comparison to FastAdaSP [37] and Speechprune [27] on the LongSpeech-Eval?\n\nThe FastAdaSP method does not extend the long-speech capabilities of speech LLMs; instead, it focuses solely on improving inference efficiency. Therefore, we did not include it as a baseline on LongSpeech-Eval.\n\nAs for SpeechPrune, we are unable to conduct experiments on LongSpeech-Eval since the method is not open-sourced. **Therefore, to provide a comparison, we conduct experiments on the SPIRAL-H benchmark [1], which is introduced in the paper of SpeechPrune for long speech information retrieval**. The results are presented in the table below, where Pruning Rate indicates the proportion of speech embeddings removed compared to the original inputs.\n\n| SPIRAL-H | | |\n|:--:|:--:|:--:|\n| **Methods** | **Prune Rate** ($\\uparrow$) | **Accuracy** ($\\uparrow$) | \n| SpeechPrune |  60 | 63.77 |\n| FastLongSpeech ($L$=750) | **65.88** | **76.81** |\n\nAs shown in the table above, our method achieves better performance with fewer speech embeddings, demonstrating its superiority and efficiency in modeling long speech inputs. We will include these results in our paper.\n\n[1] Lin et al. SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval. In ICME 2025.\n\n>How will different lengths of speech context affect the performance?\n\nTo investigate how our method performs under different context lengths, we conduct experiments on LongSpeech-Eval with varying values of $L$. The results are shown below:\n\n| LongSpeech-Eval | | | | | |\n|:--:|:--:|:--:|:--:|:--:|:--:|\n| $L$ | 200 | 400 | 750 | 1200 | 4000 |\n| **FastLongSpeech** | 2.47 | 2.90 | 3.55 | 3.66 | 3.59 |\n\nWhen $L \\leq 750$, the model exhibits increasing performance with longer context, as it is trained under dynamically varying compression ratios in this range.\\\nWhen $L$ equals 1200, although the model is not explicitly trained for this length, it still achieves strong performance, indicating good generalization beyond the training regime.\\\nWhen $L$ increases to 4000, performance slightly declines. This is expected, as the model is not exposed to such long contexts during training, despite having a larger speech context window.\n\n>The distributions of lengths of speech context in LongSpeech-Eval should be provided?\n\nThank you for your suggestion. The distribution of speech durations, measured in seconds, is shown in the table below:\n\n| LongSpeech-Eval | | | | | | |||\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| **Duration Range** | [50, 70) | [70, 80) | [80, 90) | [90, 100) | [100, 110)  | [110, 130) | [130, 170) | $\\geq$170 |\n| **Percent** | 9.15 | 21.39 | 17.68 | 12.80 | 9.76 | 9.76 | 8.54 | 7.93 |\n\n**If our response addresses your questions and concerns, we would greatly appreciate it if you could consider raising your score.**"}, {"review_id": "LT3yL7jOZg", "reviewer": "Reviewer_jHca", "summary": "This paper proposes a frame merging strategy to compress the time dimension of the speech embeddings in a large speech LLM. The LLM is based on Qwen2 -audio model. The compression algorithm is based on the CTC decoder predictions and neighboring frame similarity. For a given audio input, a context density is computed from the non-blank CTC probability and a cosine similarity score between adjacent frames. In an iterative way, depending on the target number of frames, the most similar adjacent frames are merged into groups and the group representation is obtained by a weighted average of the embeddings in that group where the weights are based on the content density. These time-reduced embeddings then pass through the adapter and the LLM layers. Experiments investigate the effectiveness of the approach on spoken question answering (SQA) for both short and long speech signals, and ASR as well as they compare the inference cost in terms of FLOPs. The proposed method is compared against the Qwen2-audio, a random downsampling of frames, an averaging based scheme, and a ROPE based approach. On both the short-form and long-form SQA tasks, the proposed method outperforms the baselines in terms of answer quality while providing reduction in FLOPs and inference time. On the ASR task, however, the WER gets significantly higher as the compression rate increases (the target length decreases) which is expected as the task itself does not require content compression.", "questions": "1) In Table 5, it is not clear how the shorter audios are counted. For instance, what does L=750 mean for the utterances much shorter than 30 seconds? \n\n2) In Table 5, for the L=100 case, it could have been better to report the WER versus test audio length (in number of embedding frames before compression) instead of an overall average because mapping a 4s audio versus a 30s-long audio with the same L would lead to a high variance in the WER per utterance.\n\n3) The limitations section does not really mention limitations but it mentions about the potential use cases of the proposed method.", "limitations": "Please see the question (3) above regarding the limitations section in the paper. \n\nTo my understanding, the model still chunks the audio longer than 30s into 30s chunks. It could have been beneficial to show the effectiveness of the method when we indeed provide a 30 minutes or an hour long audio as input during training.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths: \n\nQuality: Mathematical formulation and the experimental design seems to be correct. Ablation studies in Section 5.1 show the effectiveness of the design choices. Experiments show both quality improvements and reduction in inference run time. The conclusion regarding the best target length (or compression ratio) being dependent on the particular task is an expected result. \n\nClarity: Mostly clear.\n\nSignificance: Handling long-form speech has been an open problem even though there are several approaches to handle them successfully. Hence, this paper is relevant to the spoken LLM community as well as long-form ASR community. \n\nOriginality: Even though the technical details of the compression algorithm is not very novel (weighted average of similar consecutive), its investigation for long-form speech LLMs is novel enough.\n\n\nWeaknesses:\n\nQuality: Regarding the compression ration discussion, it might have been useful to include some other speech tasks (e.g. speech summarization) to further support the claims. \n\nClarity: In Table 5, it is not clear how the shorter audios are counted. For instance, what does L=750 mean for the utterances much shorter than 30 seconds? Especially, for the L=100 case, it could have been better to report the WER per test audio length (in number of embedding frames before compression) instead of an overall average.", "was_revised": true, "final_justification": "+++ Post-rebuttal:\n\nIt is good to see the performance breakdown based on the utterance length in the rebuttal. I am increasing my score to accept.\nThe observations about the compression rate make sense. I am increasing my score to accept.", "rebuttal": "Thanks for your valuable suggestions and insights.\n\n>Conduct some other speech tasks (e.g. speech summarization) to further support the claims?\n\nFollowing your suggestion, we conduct additional experiments on the SPIRAL-H dataset, a benchmark designed for long speech information retrieval. On this dataset, we follow the experimental setup in [1] and compare model performance under similar speech embedding pruning rates. The results are presented in the table below, where Prune Rate indicates the proportion of speech embeddings removed compared to the original inputs. SpeechPrune, RAP, and RAC are baselines introduced in the referenced paper [1].\n\n| SPIRAL-H | | |\n|:--:|:--:|:--:|\n| **Methods** | **Prune Rate** ($\\uparrow$) | **Accuracy** ($\\uparrow$) | \n| RAP | 60 | 21.45 |\n| RAC | 60 | 35.41 |\n| SpeechPrune | 60 | 63.77 |\n| FastLongSpeech ($L$=750) | **65.88** | **76.81** |\n\nAs shown in the table above, our method achieves better performance with fewer speech embeddings, demonstrating its superiority and efficiency in modeling long speech inputs. We will include these results in our paper.\n\n[1] Lin et al. SpeechPrune: Context-aware Token Pruning for Speech Information Retrieval. In ICME 2025.\n\n\n>In Table 5, it is not clear how the shorter audios are counted? It could have been better to report the WER per test audio length (in number of embedding frames before compression)?\n\nThank you for your constructive feedback. In Table 5, when the number of original speech embeddings is shorter than the window $L$, our method does not apply compression, but instead directly feeds the original embeddings into the LLMs for understanding and inference.\n\nFollowing your suggestions, we conduct an additional experiment to explore the relationship between the number of pre-compression speech frames and WER, under speech window sizes of $L = 50$ and $L = 100$. The results are shown below.\n\n| LibriSpeech test-clean ($L$=50) | | || ||||\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| **Number of Frames** | [1,100) | [100,200) | [200,300) | [300,400) | [400,500) | [500,600) | $\\geq$ 600 |\n| **WER** ($\\downarrow$) | 4.33 | 7.45 | 22.30 | 30.67 | 45.12 | 57.92 | 70.28 |\n\n| LibriSpeech test-clean ($L$=100) | | || ||||\n|:--:|:--:|:--:|:--:|:--:|:--:|:--:|:--:|\n| **Number of Frames** | [1,100) | [100,200) | [200,300) | [300,400) | [400,500) | [500,600) | $\\geq$ 600 |\n| **WER**  ($\\downarrow$) | 3.68 | 3.24 | 6.09 | 8.96 | 14.90 | 20.70 | 23.83 |\n\n\nThese results are based on an improved system trained with a larger dataset after our paper submission.\nFrom the results above, we observe that as the number of original speech frames increases, the compression ratio also increases, which leads to a progressive degradation in transcription quality. However, increasing the window size ($L$) can significantly improve ASR performance, mitigating the negative effects of compression on longer inputs.\n\n>In the future, it could have been beneficial to show the effectiveness of the method when we indeed provide a 30-minute or an hour-long audio as input during training?\n\nThanks for your suggestion. In the future, as more long-form speech training data becomes available, our method holds promise for further enhancing the speech processing capabilities of speech LLMs. By leveraging longer speech inputs during training, we aim to improve both the long-context speech and reasoning abilities of these speech LLMs, while maintaining high efficiency during inference.\n\nIf our response addresses your questions and concerns, we would greatly appreciate it if you could consider raising your score."}, {"review_id": "LuiOcjqMU5", "reviewer": "Reviewer_tmJM", "summary": "This paper studies modelling long-input in audio/speech enabled language models (I’ll refer to audio/speech language model as ALM). The motivation is that various ALMs (e.g., Qwen2-Audio, SALMONN) aligns continuous audio representations with the textual representations of the base LLM. However, audio representations usually have much higher frame rates (e.g., 30-second audio chunk accounts for 750 tokens in Qwen2-Audio). This limits the ability of ALMs to handle long-input audios (another limitation being limited long-audio training data).\n\nThis work proposes an extractor module on top of the audio representations (before passing them to the shared representation space). The extractor module iteratively compresses the number of audio tokens, while maintaining the information. This extractor module is inspired by the CTC loss, and hence is trained in two stages where the first stage adopts the CTC loss (it identifies less informative audio frames from the CTC predictions). The proposed method is designed to extend current ALMs so that they can operate with fewer audio tokens in the shared representation space.\n\nExperiments compare the proposed method against other audio representation compression methods, and show that this proposed method is effective. However, it should be noted that the paper doesn’t achieve extending the maximum audio context length of ALMs (e.g., beyond 30 seconds for Qwen2-Audio), but provides an effective audio representation compression could potentially allow the extension (albeit future work is required).", "questions": "Q1) [see weakness] Have you considered experimenting with actually extending the maximum context length? (\nQ2) Would this approach be applicable to other architectures (e.g discrete token-based audio LLMs?)", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The idea of the iterative compression method is novel \n- The audio representation compression is shown to be effective\n- The paper provides LongSpeech-Eval, which could be a useful resource \n\n**Weakness**\n- The paper has not explored extending the maximum audio input context of ALMs. The authors stated that this is due to limited long audio data being available. However, I believe that obtaining such data (e.g., 60-second or say 5-minute chunks from existing raw audio data such as Yodas2) could allow the authors to validate the proposed method for actual long contexts.  \n\nSuggestion: Some work that has tried long audio input [Audio Flamingo2](https://arxiv.org/pdf/2503.03983) achieves 30 secs to 5 mins", "was_revised": true, "final_justification": "I decided to keep my original assessment as the main title of this paper is about \"long speech\" -- although it has shown how this could be achieved, the paper itself hasn't actually delivered a model with a long-speech capability yet. I understand that data is one limitation; however, one could prepare/propose such long data and perhaps test it at a smaller scale. Overall, my original assessment was also leaning accept.", "rebuttal": ">Obtain such data (e.g., 60-second or say 5-minute chunks from existing raw audio data) could allow the authors to validate the proposed method for actual long contexts? \n\nThank you for your valuable suggestion. In the future, as more long-form speech training data becomes available, our method stands to benefit from it by further enhancing the speech context window and long speech processing capabilities of speech LLMs in an efficient manner. In addition, we appreciate the reference [1] you provided and will cite it in the next version of our paper.\n\n[1] Ghosh et al. Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities. In ICML 2025.\n\n>Experiment with actually extending the maximum context length?\n\nFollowing your suggestion, we explore the performance of FastLongSpeech on the LongSpeech-Eval dataset with varying context lengths $L$. The results are shown below:\n\n| LongSpeech-Eval | | | | | |\n|:--:|:--:|:--:|:--:|:--:|:--:|\n| $L$ | 200 | 400 | 750 | 1200 | 4000 |\n| **FastLongSpeech** | 2.47 | 2.90 | 3.55 | 3.66 | 3.59 |\n\nWhen $L \\leq 750$, the model exhibits increasing performance with longer context, as it is trained under dynamically varying compression ratios in this range.\\\nWhen $L$ equals 1200, although the model is not explicitly trained for this length, it still achieves strong performance, indicating good generalization beyond the training regime.\\\nWhen $L$ increases to 4000, performance slightly declines. This is expected, as the model is not exposed to such long contexts during training, despite having a larger speech context window.\n\nWe think our FastLongSpeech can achieve better performance with longer effective context length as the long-speech training data becomes more available.\n\n>Would this approach be applicable to other architectures, such as discrete token-based audio LLMs?\n\nWe think it is feasible. In discrete token-based audio LLMs such as GLM-4-Voice [1] and Kimi-Audio [2], discrete speech units are introduced into the vocabulary to enable both understanding and generation of speech. Given that such models have demonstrated strong performance on ASR and TTS tasks, the speech embeddings in the vocabulary corresponding to speech units are also capable of encoding rich information such as **content**, prosody, emotion, and speakers. Therefore, **our content-density-guided iterative fusion strategy and dynamic compression training hold promise for application to the embedding sequences of discrete speech units as well**. Due to the limited time available during the rebuttal period, we have not been able to explore this direction yet, but we plan to investigate it further in future work based on your suggestion.\n\n[1] Zeng et al. GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot. In ICLR 2025.\\\n[2] KimiTeam et al. Kimi-Audio Technical Report. In arXiv:2504.18425.\n\nIf our response addresses your questions and concerns, we would greatly appreciate it if you could consider raising your score."}]

Author rebuttal:

Meta-review: This paper introduces FastLongSpeech, a framework that extends LSLMs to long-form speech without dedicated training data. It combines CTC-based iterative fusion for sequence compression with dynamic compression training using short-speech data, and contributes LongSpeech-Eval for evaluation. Experiments show competitive performance with lower computational cost.

In rebuttal, the authors addressed reviewers’ concerns by adding results for Word-Bound/Blank-First, reporting on additional datasets (SPIRAL-H), and analyzing performance by sequence length. All reviewers found the method novel, practical, and publication-worthy.

Due to all of the above I recommend acceptance. I strongly encourage the authors to incorporate the reviewers’ suggestions—particularly the additional results and clarifications—into the final version, doing so will significantly enhance the quality and impact of the paper.
Decision: Accept (poster)