Title: FastLongSpeech: Enhancing Large Speech-Language Models for Efficient Long-Speech Processing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Spoken Language Understanding

Large Speech-Language Models

Speech Technologies
Abstract: The rapid advancement of Large Language Models (LLMs) has spurred significant progress in Large Speech-Language Models (LSLMs), enhancing their capabilities in both speech understanding and generation. While existing LSLMs often concentrate on augmenting speech generation or tackling a diverse array of short-speech tasks, the efficient processing of long-form speech remains a critical yet underexplored challenge. This gap is primarily attributed to the scarcity of long-speech training datasets and the high computational costs associated with long sequences. To address these limitations, we introduce FastLongSpeech, a novel framework designed to extend LSLM capabilities for efficient long-speech processing without necessitating dedicated long-speech training data. FastLongSpeech incorporates an iterative fusion strategy that can compress excessively long-speech sequences into manageable lengths. To adapt LSLMs for long-speech inputs, it introduces a dynamic compression training approach, which exposes the model to short-speech sequences at varying compression ratios, thereby transferring the capabilities of LSLMs to long-speech tasks. To assess the long-speech capabilities of LSLMs, we develop a long-speech understanding benchmark called LongSpeech-Eval. Experiments show that our method exhibits strong performance in both long-speech and short-speech tasks, while greatly improving inference efficiency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies modelling long-input in audio/speech enabled language models (I’ll refer to audio/speech language model as ALM). The motivation is that various ALMs (e.g., Qwen2-Audio, SALMONN) aligns continuous audio representations with the textual representations of the base LLM. However, audio representations usually have much higher frame rates (e.g., 30-second audio chunk accounts for 750 tokens in Qwen2-Audio). This limits the ability of ALMs to handle long-input audios (another limitation being limited long-audio training data).\n\nThis work proposes an extractor module on top of the audio representations (before passing them to the shared representation space). The extractor module iteratively compresses the number of audio tokens, while maintaining the information. This extractor module is inspired by the CTC loss, and hence is trained in two stages where the first stage adopts the CTC loss (it identifies less informative audio frames from the CTC predictions). The proposed method is designed to extend current ALMs so that they can operate with fewer audio tokens in the shared representation space.\n\nExperiments compare the proposed method against other audio representation compression methods, and show that this proposed method is effective. However, it should be noted that the paper doesn’t achieve extending the maximum audio context length of ALMs (e.g., beyond 30 seconds for Qwen2-Audio), but provides an effective audio representation compression could potentially allow the extension (albeit future work is required).\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The idea of the iterative compression method is novel \n- The audio representation compression is shown to be effective\n- The paper provides LongSpeech-Eval, which could be a useful resource \n\n**Weakness**\n- The paper has not explored extending the maximum audio input context of ALMs. The authors stated that this is due to limited long audio data being available. However, I believe that obtaining such data (e.g., 60-second or say 5-minute chunks from existing raw audio data such as Yodas2) could allow the authors to validate the proposed method for actual long contexts.  \n\nSuggestion: Some work that has tried long audio input [Audio Flamingo2](https://arxiv.org/pdf/2503.03983) achieves 30 secs to 5 mins\n\nQUESTIONS:\nQ1) [see weakness] Have you considered experimenting with actually extending the maximum context length? (\nQ2) Would this approach be applicable to other architectures (e.g discrete token-based audio LLMs?)\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a frame merging strategy to compress the time dimension of the speech embeddings in a large speech LLM. The LLM is based on Qwen2 -audio model. The compression algorithm is based on the CTC decoder predictions and neighboring frame similarity. For a given audio input, a context density is computed from the non-blank CTC probability and a cosine similarity score between adjacent frames. In an iterative way, depending on the target number of frames, the most similar adjacent frames are merged into groups and the group representation is obtained by a weighted average of the embeddings in that group where the weights are based on the content density. These time-reduced embeddings then pass through the adapter and the LLM layers. Experiments investigate the effectiveness of the approach on spoken question answering (SQA) for both short and long speech signals, and ASR as well as they compare the inference cost in terms of FLOPs. The proposed method is compared against the Qwen2-audio, a random downsampling of frames, an averaging based scheme, and a ROPE based approach. On both the short-form and long-form SQA tasks, the proposed method outperforms the baselines in terms of answer quality while providing reduction in FLOPs and inference time. On the ASR task, however, the WER gets significantly higher as the compression rate increases (the target length decreases) which is expected as the task itself does not require content compression.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nQuality: Mathematical formulation and the experimental design seems to be correct. Ablation studies in Section 5.1 show the effectiveness of the design choices. Experiments show both quality improvements and reduction in inference run time. The conclusion regarding the best target length (or compression ratio) being dependent on the particular task is an expected result. \n\nClarity: Mostly clear.\n\nSignificance: Handling long-form speech has been an open problem even though there are several approaches to handle them successfully. Hence, this paper is relevant to the spoken LLM community as well as long-form ASR community. \n\nOriginality: Even though the technical details of the compression algorithm is not very novel (weighted average of similar consecutive), its investigation for long-form speech LLMs is novel enough.\n\n\nWeaknesses:\n\nQuality: Regarding the compression ration discussion, it might have been useful to include some other speech tasks (e.g. speech summarization) to further support the claims. \n\nClarity: In Table 5, it is not clear how the shorter audios are counted. For instance, what does L=750 mean for the utterances much shorter than 30 seconds? Especially, for the L=100 case, it could have been better to report the WER per test audio length (in number of embedding frames before compression) instead of an overall average.\n\nQUESTIONS:\n1) In Table 5, it is not clear how the shorter audios are counted. For instance, what does L=750 mean for the utterances much shorter than 30 seconds? \n\n2) In Table 5, for the L=100 case, it could have been better to report the WER versus test audio length (in number of embedding frames before compression) instead of an overall average because mapping a 4s audio versus a 30s-long audio with the same L would lead to a high variance in the WER per utterance.\n\n3) The limitations section does not really mention limitations but it mentions about the potential use cases of the proposed method.\n\nLIMITATIONS:\nPlease see the question (3) above regarding the limitations section in the paper. \n\nTo my understanding, the model still chunks the audio longer than 30s into 30s chunks. It could have been beneficial to show the effectiveness of the method when we indeed provide a 30 minutes or an hour long audio as input during training.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an extractor of speech embeddings by compressing lengthy and sparse speech representations into more compact forms using a iterative fusion based on CTC scores. With the compact representations, speech language models are enabled to handle long speech context.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The proposed FastLongSpeech outperforms other compared baselines on short-speech and long-speech spoken QA tasks in terms of scores using LLM as a judge. Different lengths of compact representation consistently achieve better scores.\n2) A long-speech understanding benchmark is developed using synthetic speech and used to evaluate the propose approach.\n\nWeaknesses:\n1) The evaluation on long speech context is based on synthetic speech, then how to ensure the synthetic questions and corresponding answers are reflecting correct paralinguistic characteristics? How does LongFastSpeech model the paralinguistic information in the long speech context. It is unclear how does the developed LongSpeech benchmark control the paralinguistic information during generation.\n2) Sec.3.2 is the core part of the paper, but it is quite confusing to me. Eq. 5 is wrong? if T(m)=2L-1, then T(m+1)=$\\lfloor{(2L-1)/2}\\rfloor$ but not T(m+1)=L, according to L132-139. Also since the compression is based on the content density as defined by Eq 3, T(m+1) is not always $\\lfloor T(m)/2\\rfloor$ when T(m)>2L?\n3) The LongSpeech-Eval benchmark is developed based on long document-based QA, however, in practice, such kind of human-LLM interaction scenarios may be limited.\n4) There is a lack of comparison to a baseline that use ASR transcription of the long speech context as input, and comparison to FastAdaSP [37] and Speechprune [27] on the LongSpeech-Eval dataset.\n\nQUESTIONS:\nThe effect of long context is not shown, how will different lengths of speech context affect the performance? It is expected that longer context would show better advantage of the proposed FastLongSpeech.\n\nThe distributions of lengths of speech context in LongSpeech-Eval should be provided so that the audience have better understanding of effectiveness of the proposed method.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces FastLongSpeech, a framework designed to extend Large Speech-Language Models (LSLMs) for efficient processing of long-form speech without requiring dedicated long-speech training data. The approach builds on Qwen2-Audio and incorporates two main technical contributions: (1) an iterative fusion strategy that compresses long speech sequences by merging similar adjacent frames based on content density measured via CTC output, and (2) a dynamic compression training method that adapts models to varying compression ratios using only short-speech data. The authors also contribute LongSpeech-Eval, a benchmark for evaluating long-speech understanding capabilities. Experimental results demonstrate that FastLongSpeech achieves competitive performance on both short and long speech tasks while significantly reducing computational costs compared to baseline approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- **Practical Significance**: The work addresses a genuine limitation in current LSLMs - the inability to efficiently process long-form speech. This is practically important for applications like lecture transcription, long-form audio analysis, and extended conversations.\n\n- **Technical Innovation**: The iterative fusion strategy is well-motivated, using content density from CTC outputs to guide compression decisions. The dynamic compression training approach cleverly leverages short-speech data to transfer capabilities to long-speech scenarios without requiring expensive long-speech datasets.\n\n- **Comprehensive Evaluation**: The experimental setup is thorough, covering multiple tasks (spoken QA, ASR, emotion recognition) and providing both performance and efficiency metrics. The creation of LongSpeech-Eval benchmark is a valuable contribution.\n\n- **Clear Methodology**: The paper clearly explains the iterative fusion algorithm and training procedure, making the approach reproducible.\n\n**Weaknesses:**\n\n- **Limited Baseline Comparisons**: The paper lacks comparison with other established long-sequence modeling techniques beyond NTK-RoPE. Methods like sliding window attention or other compression strategies from the broader literature could provide stronger baselines.\n\n- **Synthetic Benchmark Limitations**: LongSpeech-Eval relies on synthesized speech from text documents, which may not capture the complexities and variabilities of natural long-form speech. The benchmark's validity for real-world scenarios is questionable.\n\n- **Content Density Assumption**: The approach assumes CTC-based content density is a reliable indicator of information importance, but this may not hold across all speech types (e.g., prosodic information, speaker characteristics, emotional content may be important but have low text density).\n\n- **Limited Scale Evaluation**: Experiments are conducted only on 7B parameter models. The scalability and effectiveness of the approach on larger, more capable models remains unclear.\n\nQUESTIONS:\nComparison with Established Methods: Why weren't established long-sequence modeling techniques like chunking with overlap, hierarchical attention, or other compression methods compared as baselines? A comparison with these approaches would strengthen the paper's claims about the effectiveness of the proposed method.\n\nLIMITATIONS:\nThe authors provide a brief limitations section but could expand on several critical points.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}