SPARQ: Outlier-free SpeechLM with Fast Adaptation and Robust Quantization

Authors: Shang Wu, Yen-Ju Lu, Haozheng Luo, Maojiang Su, Jerry Yao-Chieh Hu, Jiayi Wang, Jing Liu, Najim Dehak, Jesus Villalba, Han Liu
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Multi-modal foundation model; Speech Language Model; Low-rank adaptation; Post-training quantization

Abstract:
We propose SpARQ (outlier-free SpeechLM for Fast Adaptation and Robust Quantization) to address the outlier problem in Speech and Language multi-modal Models (SpeechLMs). Our primary observation is that outliers stemming from cross-modal (speech and text) low-rank adaptation and post-training quantization stages affect the performance of the current SpeechLMs. Methodologically, SpARQ leverages a pretrained language model as its foundation, substituting the traditional attention layer with a novel stabilized outlier-free layer. This modification eliminates outliers typically arising during cross-modal low-rank adaptation and post-training quantization. The model is then fine-tuned on multi-modal data using this outlier-free architecture, allowing it to handle textLM, speechLM, ASR, and TTS tasks through a unified interface while maintaining compatibility with parameters adapted from standard pretrained LLMs. Consequently, on the OPT-1.3b model, the proposed framework achieves relative performance improvements: 41\% in cross-modal low-rank adaptation and 45\% in post-training quantization, along with a 1.33x training speedup. We benchmark it against state-of-the-art low-rank adaptation and post-training quantization methods.

Decision: reject
Reviewer scores: 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730334701516

thread_type:
official_review

<thread_object>
creator: reviewer_h2gv

<content>
summary:
This paper proposes to use the outlier-free layer within the attention to reduces the occurrence of weight outliers for quantization and LoRA adaptations with a SpeechLM model. The outlier-free layer is constructed by replacing the first softmax layer with a layer that uses S = S - max(S) as the input, where S is the original input vector. The authors conducted three sets of experiments to validate the proposed approach: quantization, LoRA, and ablation studies / analyses.

strengths:
* Although the novelty is a combination of existing works (SpeechLM + outlier-free layer), the combination seems to be novel. The authors also provided systematic theory support for the proposed approach.
* The organization of paper is good. Paper writing is good. References are enough.

weaknesses:
* Quantization experiments: A common approach to remove outliers in quantization is to do clipping. For example, ignore the 5% tail weight values during computing the quantization scale. It’d be helpful to include this as a baseline to correctly validate the effectiveness of the proposed work.
* LoRA experiments: rank=128 seems to be large. Please add the percentage of adapter size compared to the full model.
* LoRA experiments: I would also suggest to separate the experiments to two fold, if LoRA+Quantization is one of the focus on this work. In a first experiment, we can investigate the impact of LoRA (with FP 32/16). In a second experiment, we investigated the confounding effect between LoRA and quantization (should be more aggressive than just INT8).
* Table 2: Why are the ASR WEB numbers of vanilla -125M model much better than 350B and 1.3B. Is it because the rank is set to 128 across the board? If so, the experimental setup is problematic. When applying LoRA, the rule of thumb is to control the adapter size to be under a certain percentage.
* Table 3: where does the training efficiency improvement come from? Does it require fewer steps to converge? Or is it quantization?

questions:
See weaknesses section. I'm happy to increase my ratings if my concerns can be addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730515201619

thread_type:
official_review

<thread_object>
creator: reviewer_pffh

<content>
summary:
The paper introduces SpARQ, an outlier-free framework designed to address performance degradation in SpeechLMs by mitigating outliers. SpARQ replaces the standard attention mechanism with a stabilized, outlier-free layer that improves the model's adaptability in cross-modal tasks and enhances post-training quantization robustness. Built on the OPT model, SpARQ demonstrates resilience to outliers and offers performance gains across several tasks, such as textLM, speechLM, ASR, and TTS.

strengths:
1. The paper addresses an important challenge induced by the outliers in multimodal attention mechanisms, particularly in SpeechLMs.
2. The stabilized outlier-free layer demonstrates effective mitigation of outliers during cross-modal low-rank adaptation and quantization, showing promise for real-world SpeechLM applications.

weaknesses:
1. The primary contribution of this paper is the integration of the Outlier-Efficient Hopfield Layer with max-shift normalization, which limits the originality of SpARQ. While this is the first application of outlier mitigation specifically within SpeechLMs, the Outlier-Efficient Hopfield Layer itself was previously introduced by Hu et al.
2. Techniques like Clipped Softmax and Gated Attention are potential alternatives but were not included in comparisons. Experimental results of adapting them to the SpeechLM setup would further validate SpARQ's relative effectiveness.
3. It remains unclear how much of SpARQ's improvement is attributable to the outlier-free Hopfield layer itself versus the max-shift stabilization. An additional experiment applying the stabilization module to a standard Transformer would help clarify the effectiveness of SpARQ's contributions.
4. The author states that outliers negatively impact LoRA and quantization performance. Yet, in some cases (e.g., With OPT-350m), QLoRA performs better than LoRA. Clarification on this result would improve understanding.
5. Previous studies (e.g., “Robust multimodal models have outlier features and encode more concepts” by Crabbe) mention that outlier features increase robustness for out-of-domain (OOD) samples, but OOD evaluations of SpARQ are missing. Demonstrating SpARQ’s performance on OOD data would add value to this paper.
6. There is insufficient analysis in the ablation study to explain why the max-shift stabilization uniquely works in SpARQ.
7. In Equation 3.2, the notation "S = S - max(S)" should clearly distinguish between the normalized and original versions of S.

questions:
1. Why does fully fine-tuned SpARQ generally show worse performance than fully fine-tuned Vanilla, despite the claim that multimodal samples have more outliers that degrade performance?
2. What is the number of trainable parameters for LoRA in the tables presented in the paper?
3. Do Table 1 and Table 2 reflect experiments with only PTQ and LoRA, respectively? If so, what are the results when applying PTQ on the LoRA-trained SpARQ? Including these results would clarify the model’s performance when combining the techniques.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730757564854

thread_type:
official_review

<thread_object>
creator: reviewer_pk8x

<content>
summary:
The paper, "SpARQ: Outlier-Free SpeechLM with Fast Adaptation and Robust Quantization," tackles a major hurdle in Speech and Language Models (SpeechLMs): the disruptive effects of outliers in multi-modal settings. To address this, the authors present SpARQ, a framework that replaces traditional attention layers with a mechanism specifically designed to handle outliers. This innovation helps overcome challenges in cross-modal low-rank adaptation and post-training quantization, resulting in a more efficient and accurate approach to tasks like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). The goal is to streamline training while improving performance across these applications.

strengths:
Innovative Approach: SpARQ introduces an outlier-free layer within the SpeechLM architecture, a novel solution to the inefficiencies caused by outliers that often hamper model performance and computational effectiveness.

Enhanced Performance: The framework brings notable improvements to Automatic Speech Recognition (ASR) and Text-to-Speech (TTS) tasks, showing a remarkable 41% boost in cross-modal adaptation and a 45% gain in post-training quantization accuracy.

Training Efficiency: With a 1.33x increase in training speed, SpARQ is well-suited for environments with limited computational resources, making advanced applications more accessible.

Thorough Evaluation: The model’s effectiveness is rigorously tested across various quantization and adaptation techniques, establishing a strong empirical basis for its success.

weaknesses:
Testing Constraints: The paper notes that due to computational limitations, the model hasn’t been tested on extremely large models, such as OPT 6.7B. This restriction could affect the generalizability of results for larger-scale applications.

Architectural Complexity: The architecture includes several layers and a specialized stabilization method, which could make practical implementation challenging without advanced technical expertise.

Risk of Bias Amplification: Although not deeply explored, the framework’s emphasis on reducing computational demands might unintentionally increase biases inherited from pre-trained models, given the reduced resources available for extensive fine-tuning.

questions:
Given the computational limitations that prevented testing with larger models, how might SpARQ perform when scaled up to models with billions of parameters, such as OPT-6.7B? Are there specific architectural adjustments needed to maintain stability and performance at this scale?

Several stabilization methods are compared in the paper, with max-shift normalization performing best. What were the primary reasons for choosing max-shift, and are there conditions under which other methods (like L1 or mean-centering) might be more effective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730932460177

thread_type:
official_review

<thread_object>
creator: reviewer_rfse

<content>
summary:
Authors propose a way to integrate text and speech into a one single model where outliers do not have a significant impact on the modeling.

strengths:
Good empirical performance and reasonable model structure. 
Theoretical analysis presented in the appendices is also appreciated, even though most are from previous papers. But for completeness it is good to have them there.

weaknesses:
I am puzzled but this paper. It is well written, experimented and mathematized. But the central claim about the outliers is not clearly presented and it is not theoretically pursued! As an example, I would have expected that at least following things would be easy to find in the paper, taking into account that the keyword "outlier" is in the title: 
1. Definition of the outlier in the context of the paper. Especially considering that classical definition of outlier in statistics and ML is that observation is _not_ from the distribution that is being modeled. 
2. Explaining theoretically why Eq. (3.2) is resistant to outliers as defined in the 1. 
3. Show in experiments explicitly that this indeed happens in practice also. Noting that this is different to showing general performance of the model in some downstream tasks.

questions:
- What is meant by "Given this challenge, we would like to develop a transformer architecture that can efficiently solve
the challenge in Equation (3.1). "
</content>
</thread_object>

</thread>
<thread>
creation date:
1732427829928

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Rebuttal / Revision Response

comment:
Dear Reviewers,

  

We thank the reviewers for the insightful questions and reviews. Your time and effort dedicated to improving our work are truly appreciated.

  
  

We have done all the experiments suggested and answered all the questions. All modifications are marked in **blue** color.

  

Major revisions include:

  

### Summary of Experiments and Results

-   **New Experiment: Clipped Softmax and Gated Attention (Appendix G.1)**  `reviewer pfFh`
    - We conducted an ablation study comparing SpARQ with alternative techniques under different quantization levels.  
    - Results: At 8-bit quantization, all methods performed similarly with performance drops under 0.5%, indicating that outlier mitigation strategy is less critical at moderate quantization. However, at 4-bit quantization, SpARQ showed significantly lower performance drop (130.71%) compared to Gated Attention (215.03%), Clipped Softmax (223.72%), and the vanilla approach (233.36%). This validates SpARQ’s superior performance under aggressive quantization due to its architectural outlier mitigation approach, critical for tasks requiring precise attention patterns.  
      
    
-   **New Experiment: Stabilization Module to a Standard Transformer (Appendix G.2)** `reviewer pfFh`
    - We conducted an experiment to assess the isolated effect of max-shift stabilization on a standard Transformer architecture.  
    - **Results**: Minimal improvement in validation accuracy (34.4% vs. 34.3%) was observed with max-shift stabilization alone. SpARQ’s significant gains stem from the synergistic combination of the outlier-free Hopfield layer and max-shift stabilization rather than either component individually. Max-shift stabilization addresses numerical stability issues, highlighting the complementary roles of SpARQ's components.  
      
    
-   **New Experiment: Applying PTQ to LoRA-trained Models (Appendix G.3)** `reviewer pfFh`
    - We evaluated SpARQ's performance when combining LoRA training with post-training quantization.  
    - **Results**: SpARQ achieved significantly lower performance drops (155.95% vs. 396.34% at 8-bit and 186.66% vs. 417.52% at 4-bit) compared to the vanilla framework. ASR performance remained stable with SpARQ (18.52% to 28.32% WER), while the vanilla framework degraded severely (93.91% to 97.24% WER). These results validate SpARQ’s robustness when combining low-rank adaptation with quantization, making it suitable for practical compression applications.  
      
    
-   **New Experiment: Weight Clipping During PTQ (Appendix G.4)** `reviewer H2gv`
    - We conducted experiments applying weight clipping during PTQ on both SpARQ and vanilla baselines.  
     - **Results**: SpARQ exhibited significantly lower performance drops compared to the vanilla framework, further emphasizing its resilience under aggressive quantization conditions.  
      
    
-   **New Experiment: QLoRA with Lower Bits (Appendix G.5)** `reviewer H2gv`
    - We investigated the impact of varying quantization levels on LoRA-adapted models.  
    - **Results**: SpARQ reduced performance degradation significantly, from 330.47% to 136.07% at 8-bit and from 387.50% to 176.52% at 4-bit quantization. SpARQ showed more gradual degradation across quantization levels, indicating better stability and mitigating the confounding effects of combining LoRA and quantization, as evidenced by smaller performance gaps between 8-bit and 4-bit settings.  
      
    
-   **New LoRA Trainable Parameter Table (Appendix H)** `reviewer pfFh` 

    A detailed table summarizing the trainable parameters in LoRA configurations has been included for reference.
    

- **Revised limitation in Sec 5** `reviewer pfFh`
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work addresses a critical bottleneck in Speech and Language Models (SpeechLMs), namely the disruptive influence of outliers in multi-modal environments. To overcome this challenge, the authors propose SpARQ, an innovative framework that replace traditional attention mechanisms with a specialized design to effectively mitigate the impact of outliers. This approach enables seamless cross-modal low-rank adaptation and robust post-training quantization, ensuring both efficiency and precision. By tackling these long-standing hurdles, SpARQ not only streamlines training but also delivers superior performance across key applications like Automatic Speech Recognition (ASR) and Text-to-Speech (TTS). 

The key claim of this work is the introduction of a novel outlier-free framework aimed at mitigating outliers to address performance degradation in SpeechLMs. The primary strengths of the proposed work are:  
(i) a well-written and well-organized paper with strong references, and  
(ii) significant improvements demonstrated on selected models for Automatic Speech Recognition (ASR) and Text-to-Speech (TTS).  

The major weaknesses include:  
(i) the solution has not been tested on extremely large models, such as OPT 6.7B, due to computational limitations, which may impact the generalizability of results for larger-scale applications;  
(ii) the risk of bias amplification; and  
(iii) the lack of comparative experiments with standard clipping approaches for outlier removal.  

A minor weakness is the incremental novelty, as the contribution primarily lies in the combination of existing techniques. However, it is worth acknowledging that the assembly and integration of these solutions present a novel approach.

On one hand, the idea is interesting, and the authors have expanded the experimental evaluation of their approach during the rebuttal + discussion phase. On the other hand, significant concerns remain regarding generalizability to larger models and the risk of bias amplification. Additionally, Reviewer H2gv, while acknowledging the new experiments on clipping, explicitly stated the need for further experiments. They noted that, in the quantization literature, clipping typically refers to limiting outlier values when computing the quantization scale, rather than completely removing outliers as done by the authors. This difference might explain the significant performance drop reported and raises some doubts about the validity of those experiments.