A Variational Approach for Generative Speech Language Modeling

Authors: Li-Wei Chen, Takuya Higuchi, Zakaria Aldeneh, Ahmed Hussen Abdelaziz, Alexander Rudnicky
Primary area: generative models
Keywords: Generative Spoken Language Modeling;Speech Language Model

Abstract:
The success of large language models in text processing has inspired their adaptation to speech modeling. However, because speech is continuous and complex, it is often discretized into tokens derived from self-supervised speech models. These speech tokens typically focus on the linguistic aspects of speech and neglect its paralinguistic content. As a result, autoregressive models trained on these tokens may generate speech with suboptimal naturalness. Previous methods attempted to address this limitation by adding pitch features to speech tokens prior to autoregressive modeling. However, pitch alone cannot fully represent the range of paralinguistic attributes, and selecting the right features requires careful hand-engineering. To tackle this issue, we propose a variational approach that automatically learns to encode these continuous speech attributes to enhance the speech tokens. Our proposed approach eliminates the need for manual paralinguistic feature selection and extraction. Moreover, we demonstrate that our proposed approach maintains or improves speech language modeling performance and enhances the naturalness of generated speech compared to baseline approaches.

Decision: reject
Reviewer scores: 5, 8, 3, 6

Reviews and discussion:
<thread>
creation date:
1729282310672

thread_type:
official_review

<thread_object>
creator: reviewer_kqq1

<content>
summary:
This paper proposes a variational approach to speech-language modelling in contrast to traditional auto-regressive models. The aim is to capture information other than semantics.

strengths:
1. Using the variational method for speech-language modelling is novel and useful. The exploration is very interesting and I like the idea.
2. The paper is well-written with adequate derivations for key loss functions.

weaknesses:
1. My main concern is how this method is useful for more practical downstream tasks such as ASR, emotion or speaker recognition, to reflect that capturing the additional (mainly paralinguistic) information is useful. I strongly encourage the authors to conduct at least 2 of the above practical tasks using the variational speech LM and compare it to token-based speech LM to see if there are any potential benefits of using variational methods.
2. Experimental is conducted using LibriSpeech and Libri-light, which are datasets with quite small variabilities other than semantic information. I believe the variability remains in the speaker representation space, which is not explicitly reflected in the experimental design. 

Together with the question below, I am not convinced that this paper has acceptance quality at the moment. However, given the idea is interesting, I would like to see how the authors would improve the paper during the rebuttal period, and promise to raise my score if my concerns are addressed.

questions:
1. Why do we need to separate z^c and z^d? Are they really independent? I believe knowing z^c will tell you a lot about z^d, right? The authors are encouraged to explain this design choice further.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730176096330

thread_type:
official_review

<thread_object>
creator: reviewer_vvxw

<content>
summary:
This paper proposes learning para-linguistic information within a VAE framework by incorporating semantic information from a speech language model. However, there are issues with both the learning objective formulation and the experimental results.

1. The core idea is to learn an autoregressive prior through VAE training. This approach differs from VQ-VAE, where the prior is learned separately after training. However, the authors use the intermediate features directly to train \(\phi\) and then expect \(\phi\) to generate \(z_t^{c}\) for computing divergence. This does not achieve the traditional effect of KL divergence in regularizing the latent space. Furthermore, it diverges from standard approaches where the prior is parameterized independently from the latent features. Effectively, this approach is equivalent to learning the prior after training, which undermines the formulation's purpose.

2. Conditioning on semantic information to learn para-linguistic features is not new and has already been investigated in works like [1]. Additionally, the audio samples presented by the authors are audibly poor and notably worse than those produced by speech synthesis methods that directly utilize pitch features, as demonstrated in [2]. Furthermore, unlike most recent studies, the authors did not provide a web interface for easy access to samples, which makes evaluation cumbersome. If MOS evaluations were already conducted, it would be beneficial for reviewers to have a similar interface for sample evaluation.

References:  
[1] Zhang, Xin, et al. "Speechtokenizer: Unified speech tokenizer for speech large language models." arXiv preprint arXiv:2308.16692 (2023).  
[2] Polyak, Adam, et al. "Speech resynthesis from discrete disentangled self-supervised representations." arXiv preprint arXiv:2104.00355 (2021).

strengths:
Nothing particularly strong.

weaknesses:
Theoretical problem and poor result. Please refer to point 1 and 2 in my summary.

questions:
Why no demo page? Have you try train the AR prior post training?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730474482593

thread_type:
official_review

<thread_object>
creator: reviewer_xbpk

<content>
summary:
This paper describes an approach to improve speech LM modeling in a speech sequence completion task.  The central idea is to augment the speech tokens with an additional autoregressive variational input.  This results in improved naturalness and meaningfulness of the responses compared to a baseline and a speech-token + pitch representation.

strengths:
The proposal is well motivated, and builds on prior work on speech representation and variational modeling speech generation.

The results are generally quite strong.  The subjective evals show clear improvements to both meaningfulness and naturalness.  The only regressions are to sWUGGY and sBLIMP objective measures.

While the approach adds overall complexity to the inference call, the number of parameters added appear to be quite low -- only 2M additional params.

weaknesses:
It would be useful to have some confidence measure for the sWUGGY, sBLIMP and Perplexity values.  It is unclear how much the 61.75 -> 60.48 sWUGGY regression means.  Is this statistical noise, or an issue that should be addressed.

While the variational augmentation adds fewer than 1% of parameters, it would be useful to know if this meaningfully adds to inference latency.

questions:
See above -- it would help understand the approach more completely to have an understanding of any latency implications to this model adjustment.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527775342

thread_type:
official_review

<thread_object>
creator: reviewer_h7h5

<content>
summary:
This paper introduces a variational approach that automatically learns to encode these continuous speech attributes to enhance the speech tokens, and eliminates the need for manual paralinguistic feature selection and extraction, such as pitch features.

strengths:
1. Employing an encoder to derive continuous features instead of manually crafted paralinguistic features endows the features with greater flexibility and potency.
2. The symbols and formulas within the paper are clearly defined, and comprehensive details, encompassing mathematical derivations and experimental setups, are thoroughly documented in the Appendix.
3. Various advanced technologies are used to enhance the model, such as time-wise normalizing flow, and diffusion decoder (However, the ablation studies are not reported in the paper).

weaknesses:
1. In section 3.2, it is mentioned that "By using these tokens, the model no longer needs to encode as much phonetic information in Z^c, allowing Z^c to focus on other continuous speech attributes. ". To strengthen this argument, it would be more convincing to include some analytical experiments. For instance, demonstrating that Z^c excels in speaker verification or emotion recognition, but performs less effectively in speech recognition, would provide a more nuanced understanding of its capabilities.
2. The descriptions of the evaluation are unclear. The authors should provide a clear explanation of whether the AR model is utilized for each metric, possibly by referring to section 3.1 of the GSLM paper.
3. In the main results, the conclusion, "Speech generated from our proposed approach does not sacrifice meaningfulness compared to speech generated from the baselines.", is not strongly supported by the experiments, particularly when considering the observed declines in sWUGGY and sBLIMP. Despite the authors' speculations, the empirical data does not robustly corroborate this claim.
4. it is very weird that the sWUGGY of (Token + continues features) is worse than that of both Token-LM or continues features-LM. (Proposed vs. Token-LM, Proposed vs. Proposed - tokens, in Table 3).

questions:
1. What is the CER of the ground-truth, which serves as the upper bound for the ASR model?
2. When utilizing discrete tokens enriched with acoustic information, such as Encodec, can the proposed method yield enhancements?
3. Why does the Token-LM trained on the LibriSpeech dataset exhibit significantly better CER compared to the Token-LM trained on the Libri-light dataset (5.40 vs. 10.19), yet this improvement is not obtained in the proposed method (5.06 vs. 4.35)?
4. why the numbers of γ=0.5 in Table 4 can not be found in Table 3?
5. why the β is set to 0.04 instead of 0.03 in Table 4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732129931859

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response: Additional Experiments

comment:
We sincerely appreciate the reviewers' valuable comments and feedback. In response to their suggestions, we conducted two additional experiments—emotion recognition and speaker identification—to better demonstrate the information encapsulated by our learned features. We will summarize and include these results in the paper.

 | Method                          | Emotion Recognition (ACC, %) |
|---------------------------------|------------------------------|
| *Tokens*                        | 57.46 ± 1.59                |
| *Variational Features*          | 91.57 ± 0.35                |
| *Tokens + Variational Features* | **92.74** ± 0.37            |



| Method                          | Speaker Identification (ACC, %) |
|---------------------------------|----------------------------------|
| *Tokens*                        | 7.08 ± 0.40                    |
| *Variational Features*          | 63.41 ± 0.43                   |
| *Tokens + Variational Features* | 63.13 ± 0.45                   |
| *Utterance Embedding*           | **94.06** ± 0.32               |


    
- **Speech Emotion Recognition.** We evaluate speech emotion recognition on the EmoV-DB [1] dataset with a single speaker (Sam). We follow a 9:1 split on training and testing for the dataset. The dataset contains 5 emotion categories: amused, angry, neutral, disgust, and sleepiness. We train a classifier with the same structure to predict emotion categories based on different features. The experiments are repeated 20 times to report the mean and 95\% confidence interval. From the above table, we can observe that the variational features alone obtain significantly better performance compared to tokens, showcasing its capability of capturing paralinguistic information. Combining both tokens and variational features gives a slight improvement over using variational features alone.
 
- **Speaker Identification.** For speaker identification, we evaluate the performance on the VCTK [2] dataset, which consists of read English sentences, with 400 sentences each from 110 speakers. We again follow a 9:1 train-test split and repeat each run 20 times to report the mean and 95\% confidence interval. We additionally evaluate our utterance embedding, which is designed to capture static utterance-level information (see Section 3.5). From the above table, we can see that using tokens only results in poor speaker identification accuracy. With variational features, the classifier obtains improved accuracy. We attribute this improvement to the fact that speaking styles can be captured in the variational features to classify speakers. On the other hand, the utterance embedding outperforms the other features in this task. These results support our claim that the utterance encoder encodes global speaker information while variational features capture local paralinguistic attributes.


[1] A. Adigwe, N. Tits, K.E. Haddad, S. Ostadabbas, and T. Dutoit, The Emotional Voices Database: Towards Controlling the Emotion Dimension in Voice Generation Systems. ArXiv, abs/1806.09514, 2018.

[2] J. Yamagishi, C. Veaux, and K. MacDonald, CSTR VCTK Corpus: English multi-speaker corpus for CSTR voice cloning toolkit (version 0.92), 2019.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732820272395

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Revisions and Reviewers Feedback Responses

comment:
We sincerely appreciate the reviewers' valuable comments and feedback. As the revision deadline has passed, we would like to remind the reviewers that the paper has been revised to address the raised concerns, with the **updated sections marked in blue** for easy reference.

In summary, the revisions mainly address the following key points:

- **Additional Experiments**: We conducted supplementary experiments to gain a deeper understanding of the variational features, as suggested by Reviewers h7h5 and kqQ1. We included the experiment results in Appendix I.
- **Evaluation Metrics Clarification**: Tables 1 and 2 have been restructured, and we added a paragraph explaining the differences between evaluating speech reconstruction and speech continuation to improve clarity, addressing Reviewer h7h5’s concern.
- **sWUGGY and sBLIMP Clarification**: To address Reviewer h7h5's concern regarding the objective metrics, we clarified that sWUGGY and sBLIMP evaluate syntactic and grammatical correctness, while our claim about meaningfulness is supported by human evaluation scores, which offer the most direct measure. We have revised Section 5.1 to reflect this clarification. Additionally, we hypothesized potential causes for the decline in sWUGGY and sBLIMP scores in Section 5.1 and elaborated further in our author response.
- **Theoretical Background**: To address Reviewer VVXW's concern about the validity of the *VAE with autoregressive prior*, we added references to previous works that adopted parameterized autoregressive prior for VAE in Section 3.1. In our response, we also highlighted our proof that the objective still maximizes ELBO and explained that the regularization effect arises from the autoregressive model's limited capacity.
- We also included a discussion on the conditional independence of $z_t^c$ and $z_t^d$​ in Appendix J and outlined future work on foundation models in Section 8, based on our discussion with Reviewer kqQ1.

We believe these revisions and our responses address the reviewers’ concerns up to this point. However, should any issues remain or further clarifications be needed, **we kindly invite the reviewers to continue the discussion**.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733266435652

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Period Summary

comment:
We appreciate the positive feedback from the reviewers and their thoughtful evaluation of our work.
 - Reviewer `h7h5` noted that the symbols and formulas are clearly defined and that comprehensive details are thoroughly documented in the Appendix.
 - Reviewer `kqQ1` mentioned that the paper is well-written and provides adequate derivations.
 - Reviewer `XBpK` highlighted that our method is well-motivated and that the results are quite strong.

## Addressing Reviewers’ Concerns:

### Reviewer `kqQ1`:

 - **Additional Experiments**: We conducted supplementary experiments, as suggested, to further explore the variational features. These results are presented in Appendix I, addressing the need for deeper insights and validating our approach. We greatly appreciate the reviewer considering our responses and adjusting the score accordingly.

### Reviewer `XBpK`:

- **Inference speed**: The concern was whether the 1% parameter size increase translates to similar inference latency. We provided a latency comparison, demonstrating only a 1.3% increase, validating the efficiency of our method.

### Reviewer `h7h5`:

- **Clarification for the Evaluation Results**: Reviewer `h7h5` raised concerns about the clarity of presentation. We addressed concerns about presentation clarity by distinguishing between speech reconstruction and speech continuation, restructuring Tables 1 and 2, and adding detailed explanations.
- **sWUGGY and sBLIMP scores**: Reviewer `h7h5` raised concerns about objective metrics. We explained that these metrics evaluate syntactic and grammatical correctness, while our primary claim on meaningfulness relies on human evaluation. We also hypothesized reasons for the trend of the objective scores.

### Reviewer `VVXW`:

 - **Misunderstanding of the Performance Issue**: Reviewer `VVXW` noted that our provided syntheses appear notably worse than those in a prior work. However, as we clarified in our response, the mentioned prior work addresses a re-synthesis task, whereas our focus is on the more challenging problem: speech continuation. We believe this distinction indicates that the comparison is not directly fair and it does not indicate that our method has poor performance.
 - **Questions regarding Theoretical Validity**: Reviewer `VVXW` questioned the validity of the "VAE with auto-regressive prior", specifically about how the latent variables can be regularized with the parameterized prior. In our response, we explained that this approach has been established in prior works and that we derived the corresponding ELBO in Appendix A.1 to provide a solid theoretical foundation. Additionally, we highlighted that the KL divergence term in our formulation continues to offer a regularization effect on the latent space.
 - **Given that we have addressed all main concerns for the initial rejection decision of Reviewer `VVXW`, we believe our paper now merits a higher evaluation.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a variational approach to speech-language modelling compared to traditional auto-regressive models. The central idea is to augment the speech tokens with an additional autoregressive variational input. This results in improved naturalness of the responses compared to a baseline and a speech-token + pitch representation. 

The concerns raised by reviewers were mostly addressed during the rebuttal. However, the AC still considers that the paper does not provide sufficient results to demonstrate the usefulness of the proposed approach. Hybrid decoding methods combining semantic and acoustic units are widely used today to incorporate the paralinguistic information of spoken LLMs. However, the paper does not compare its approach with any of these methods. I encourage the authors to benchmark their model against more current approaches. 

Please refer to "Additional Comments on Reviewer Discussion" for more details.