SSR: Alignment-Aware Modality Connector for Speech Language Models

Authors: Weiting Tan, Hirofumi Inaguma, Ning Dong, Paden Tomasello, Xutai Ma
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: modality fusion, speech language model, spoken language understanding

Abstract:
Fusing speech into pre-trained language model (SpeechLM) usually suffers from inefficient encoding of long-form speech and catastrophic forgetting of pre-trained text modality. We propose SSR-connector (Segmented Speech Representation Connector) for better modality fusion. Leveraging speech-text alignments, our approach segments and compresses speech features to match the granularity of text embeddings. Additionally, we introduce a two-stage training pipeline that includes the distillation and fine-tuning phases to mitigate catastrophic forgetting. SSR-connector outperforms existing mechanism for speech-text modality fusion, consistently achieving better speech understanding (e.g., $+10$ accuracy on StoryCloze and $+20$ on Speech-MMLU) while preserving pre-trained text ability.

Decision: reject
Reviewer scores: 8, 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1730212671890

thread_type:
official_review

<thread_object>
creator: reviewer_j1u2

<content>
summary:
This paper presents a method to improve speech langauge models by using speech-text alignment. Experiments have been conducted to evaluate the performance of the proposed method.

strengths:
1. The idea of utilizing the alignement between speech and text sequences to build speech LMs is novel to me.
2. The paper is written clearly and easy to follow.

weaknesses:
Since the speech-text alignement is considered in the proposed method, it has similarities with the ASR+LLM (cascade) implementation of speech LMs. For an ASR model, its task is to derive the aligned text from given speech. In Table 4, the performance of the cascade system was quite good. So, some discussions on comparing the proposed method with cascade systems should be emphasized.

questions:
1) In Section 3.1, what is the unit of the transcription sequence y for alignement? characters, phonemes or words?
2) In Eq. (4), why are both similarity loss and MSE loss used? How to determine the weight for combining them?
3) In Section 5, why is the Char-CTC model used for evaulation since it didn't perform as good as UnityY2 in Section 4?
4) In Table 4, LLaMa2 was used for the cascade system while LLaMa3 was used in the proposed method. This seems unfair.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474060896

thread_type:
official_review

<thread_object>
creator: reviewer_bk9j

<content>
summary:
The paper presents SSR-Connector, an innovative approach to integrating speech data into pre-trained LLMs by aligning and compressing speech representations to match text embeddings, enhancing speech-text modality fusion. This technique tackles two main challenges: inefficient processing of long speech sequences and the risk of catastrophic forgetting of text-based capabilities. SSR-Connector employs a two-stage training process: an initial distillation phase aligns speech representations with text embeddings, followed by fine-tuning to improve cross-modal understanding. The approach incorporates alignment-aware segmentation of speech features, optimized to work with text-compatible granularity, and achieves some gains in some speech understanding benchmarks, surpassing prior frameworks such as SpiritLM and VoxtLM.

strengths:
1. The paper is well-structured and systematically describes each component of the SSR-Connector. Additionally, it clearly explains the methodology used to prevent catastrophic forgetting and offers comparisons with prior models.
2. The SSR-Connector introduces a novel "alignment-aware" approach to speech-text modality fusion.
3. The paper provides a comprehensive analysis of the proposed SSR-Connector across a variety of benchmarks, including StoryCloze, sWUGGY, and Speech-MMLU.

weaknesses:
1. One significant aspect of SpeechLMs is their ability to capture information in speech beyond just the content. This has been analyzed in many related studies; for instance, in SALMONN [1] and Qwen2-Audio [2], paralinguistic information, such as emotion, is treated as an important evaluation aspect, and SpiritLM also examines capabilities related to understanding paralinguistic information. In addition, as the authors mention in the introduction, cascade systems are unable to learn paralinguistic information in speech, which is an advantage of end-to-end SpeechLMs. However, this paper lacks an analysis of the proposed model’s understanding of paralinguistic information and focuses solely on performance from a content comprehension perspective. If the model lacks the ability to understand paralinguistic information, then even if it addresses the issues of long speech sequences and catastrophic forgetting, it still wouldn't make for an ideal SpeechLM.

2. From the perspective of model methodology, it is challenging for the approach to capture paralinguistic information from speech. The methodology focuses on understanding speech content while overlooking other information within the speech. For instance, in stage 1, the target is text embedding, and achieving the best result would mean that the model could directly predict text embedding from speech but ignore other information. In stage 2, the SSR-Connector is fixed, and only the LLM is trained, so the LLM is also unable to learn information beyond content. Therefore, the entire training process essentially resembles a speech recognition model’s functionality by learning to extract textual information from speech. This limits the novelty of the paper, as the method does not introduce a more comprehensive SpeechLM capable of learning beyond content and instead functions more like an end-to-end model which integrates a speech recognition model with an LLM.

3. Another concern might be that the model’s best performance relies on the UNITY2 model for alignment, which itself requires both speech and transcription for alignment results. Given that a transcript is already available, why not use it directly as the LLM’s input instead of processing it through the SSR-Connector? In my view, using a model that doesn’t rely on transcription for alignment would better align with the scenario proposed in the paper.

4. As far as I know, the test sets used in the paper, such as sWUGGY and sBLIMP, are those employed in the Zero Resource Speech Challenge, where one of the key functions of the models is speech continuation. All the baseline models compared in the paper include speech generation capabilities. I would suggest comparing more models that, like the proposed model, take speech as input and output text, such as SALMONN [1], Qwen-Audio [3], and Qwen2-Audio [2].

[1] Tang, Changli, et al. "Salmonn: Towards generic hearing abilities for large language models." arXiv preprint arXiv:2310.13289 (2023).

[2] Chu, Yunfei, et al. "Qwen2-audio technical report." arXiv preprint arXiv:2407.10759 (2024).

[3] Chu, Yunfei, et al. "Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models." arXiv preprint arXiv:2311.07919 (2023).

questions:
I have outlined my main concerns in the Weaknesses part. Here are the corresponding questions:
1. How does the proposed model account for paralinguistic information (e.g., emotions) in speech, and why has the analysis of this aspect been omitted from the paper, despite its significance in related studies?
2. Given the methodology’s focus on content comprehension, how does the model address the capture of paralinguistic information?
3. Why does the model rely on the UNITY2 alignment method, which uses both speech and transcription, rather than directly using the available transcription as input for the LLM? Would a model that does not depend on transcription for alignment be more aligned with the paper's goals?
4. Could you include more baseline models, such as SALMONN and Qwen2-Audio?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730564936189

thread_type:
official_review

<thread_object>
creator: reviewer_98pj

<content>
summary:
This paper tackles the issue of catastrophic forgetting in the Language Model (LM) during speech fusion. The authors introduce the SSR-Connector method to enhance modality fusion. This approach utilizes a speech-text aligner and a decoder-only compressor to produce speech representations that align with the granularity of text embeddings.

strengths:
The proposed method outperforms existing speech fusion mechanisms in terms of speech understanding.

weaknesses:
The cost-performance ratio of the proposed method is lower when compared to a cascaded ASR+LLM system. 
Please refer to the questions section regarding my queries.

questions:
At the very beginning, it is mentioned that integrating speech often results in catastrophic forgetting of the primary pre-training. Have you quantified the severity of this issue? During the second stage of training where you fine-tune the Large Language Model (LLM), is there a risk of forgetting within the LLM itself? Specifically, I am referring to the LLM's performance (across various NLP tasks) before and after incorporating the speech modality.

Could you provide the specific figure for the original LaLLM3 as listed in Table 6?

Could you provide further explanation for the statement "lack the ability to support text or speech generation unless further instruction-finetuned" in the introduction section?

Given the option to use a decoder-only compressor and considering the overall model complexity, why not opt for a cascade system like Whisper+LLM? My assumption is that post the entire training process, the adapted representation z (as shown in Figure 3, right-hand side) will closely resemble the input embedding of the corresponding text (or sub-word units). This scenario might make the entire SSR-Connector resemble an Automatic Speech Recognition (ASR) system. In that case, why not simply use an ASR with text (or sub-word units) output directly?

Does the Speech Signal Representation (SSR) convey any audio/voice information beyond text? If so, have you conducted an analysis on this aspect? This is crucial as the community is progressing towards general audio Large Language Models as opposed to solely speech-based models. If not, similar to the previous question, what is the rationale behind incorporating an ASR system?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730689130810

thread_type:
official_review

<thread_object>
creator: reviewer_izxp

<content>
summary:
This paper proposes a new architecture with the corresponding loss design to connect speech embeddings to LLM. Thorough experiments are conducted to demonstrate the values of the proposed method.

strengths:
The proposed method has some connections with the previous methods in speech downsampling and model distillation but is novel enough. The evaluation is very thorough.

weaknesses:
Although the paper acknowledges the three categories of connector designs in Figure 1, it does not compare enough to Figure 1(a). Compared with the proposed 2-stage method, a single-stage or 2-stage baseline with a much simpler 1(a) design can motivate the more complex design in this paper.
 
Some experiment comparisons and explanations can be improved (see questions).

questions:
1. need to explain why unitY2 based method is much better than others
2. "Even our weakest system (CHAR-CTC (Unit-based)) can outperform SPIRITLM ’s 10-shot result." the aligner in the proposed method is trained with paired ASR speech-text data, which is unfair to compare with SPIRITLM. 
3. missing description of speech-MMLU
</content>
</thread_object>

</thread>
<thread>
creation date:
1731983416922

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to all reviewers

comment:
We sincerely thank all the reviewers for their constructive feedback on our work. We are particularly pleased that several reviewers (izXP, Bk9J, j1u2) recognized the novelty of our modality fusion approach. 


## Paralinguistic Information
We noted a common concern from reviewers 98Pj and Bk9J regarding the ability of our SpeechLM to leverage information beyond semantics. We would like to address this concern in our general response.
Firstly, we acknowledge the importance of utilizing paralinguistic information in end-to-end SpeechLM systems, and we agree that this is an area that requires further development. However, similar to most existing works on SpeechLM, our primary focus has been on semantic information. This is because **addressing catastrophic forgetting and improving speech understanding in end-to-end SpeechLM is a challenging and significant problem in its own right**. Our modality fusion approach has demonstrated superior performance compared to several strong baselines (such as SpirtLM, TWIST, VoxtLM, etc.) in both speech and text understanding tasks. Therefore, we believe it would be an overly stringent criterion to dismiss our work on the basis of not incorporating non-semantic information at this stage.

We still appreciate the reviewers for highlighting concerns about the non-semantic aspects of our work, which motivated us to conduct additional experiments. These experiments aimed to determine whether our SSR-based SpeechLM retains paralinguistic information, particularly in the context of in-context learning. Interestingly, we found that **our model, despite not being explicitly trained to use paralinguistic cues, can learn to differentiate between speech styles through in-context learning**.
To investigate this, we used the Expresso dataset [1], which features speeches in different styles (e.g., happy, sad, whispering, laughing). We conducted two sets of experiments:

- Whisper vs. Laugh: We tasked the model with identifying whether a speech was whispered or laughed. The prompt used was: 
*"You are given speeches from two styles. Your task is to judge if the speech is a whisper or laugh. Here are some example speeches: [Speech]: {speech} [Style]: {whisper/laugh}..."*

- Happy vs. Sad: We asked the model to determine if the speech was delivered happily or sadly. The prompt was: 
*"Listen to the following speech and judge if the speaker is happy or sad. Here are some examples: [Speech]: {speech} [Emotion]: {happy/sad}..."*

For each experiment, we provided our SpeechLM (trained with an SSR-connector based on the UnitY2 aligner) with 0, 1, 5, and 10 examples, and reported the accuracy (averaged over 10 runs) in the table below (We also benchmarked a cascaded system, Whisper + Llama3, for comparison):


|Task | Model  | 0-shot |1-shot| 5-shot | 10-shot |
|--------|---------|-------|-------|--------|---------|
|Whisper v.s. Laugh | Cascaded System | 51.6  | 52.1 | 52.2 | 54.7 |
| | Ours         | 49.6 | 62.4 |64.0 |75.9  |
| |
|Happy v.s. Sad | Cascaded System | 50  | 51.4 | 51.8 | 51 |
| | Ours         | 51.6 | 52.1 |52.2 | 54.7 |

With zero-shot prompting, the model generates random predictions because it hasn't been trained to use paralinguistic information. However, when given a few-shot examples, the model can learn from the prompts to make decisions, achieving surprisingly good results in distinguishing between whispering and laughing speech. In-context learning also helps to some extent in differentiating happy and sad speech, though not as effectively as distinguishing between whispering and laughing. We believe this is because the style difference between whispering and laughing is more pronounced. **The results indicate that paralinguistic information is still present in our model's representations and can be utilized**. Additionally, our approach complements previous methods that enable models to consider paralinguistic information, such as using expressive tokens from SpiritLM or emotion-relevant instruction tuning from SALMONN.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces SSR-Connector, an approach designed to integrate speech data into pre-trained LLMs. It achieves this by aligning and compressing speech representations to match text embeddings, enhancing the fusion of speech and text modalities. SSR-Connector employs a two-stage training process: an initial distillation phase to align speech representations with text embeddings and fine-tuning to enhance cross-modal understanding. This approach demonstrates improvements in certain speech understanding benchmarks, surpassing prior frameworks like SpiritLM and VoxtLM.

The novelty of the paper is limited. The concepts of connector distillation and text-to-speech alignment have been explored in prior studies. Although these previous works differ from the current one, the authors have not compared their approach with these earlier methods to demonstrate its advantages. The training of the SSR connector is also similar to ASR, with the primary distinction being that LLM embeddings are used as the target. Consequently, reviewers have expressed concerns regarding the benefits of the proposed approach compared to an ASR+LLM cascade model. Furthermore, reviewers pointed out the absence of certain baseline comparisons.