Disentangling Textual and Acoustic Features of Neural Speech Representations

Authors: Hosein Mohebbi, Grzegorz Chrupała, Willem Zuidema, Afra Alishahi, Ivan Titov
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Disentangling Representations, Spoken language Processing, Speech Emotion Recognition, Interpretability

Abstract:
Neural speech models build entangled internal representations, which capture a variety of features (e.g., pitch, loudness, syntax, or semantics of an utterance) in a distributed encoding. This complexity makes it difficult to track how such representations rely on textual and acoustic information when used in downstream applications, limiting their interpretability. In this paper, we build upon the Information Bottleneck principle to propose a disentanglement framework that separates speech representations learned by pre-trained neural speech models into two distinct components: one encoding content (i.e., what can be transcribed as text) and the other encoding acoustic features relevant to a downstream task. We apply and evaluate our framework to emotion recognition and speaker identification target tasks, quantifying the contribution of textual and acoustic features at each model layer. We also apply our disentanglement framework as an attribution method to identify the most salient speech frame representations from both the textual and acoustic perspectives.

Decision: reject
Reviewer scores: 3, 3, 6, 3

Reviews and discussion:
<thread>
creation date:
1730066779530

thread_type:
official_review

<thread_object>
creator: reviewer_icvr

<content>
summary:
This paper proposes a new framework for disentangling speech representations from neural speech models (like Wav2Vec2 and HuBERT) into two distinct components: textual content (what can be transcribed as text) and acoustic features (like emotion or speaker identity). This separation is important because neural speech models typically create deeply entangled internal representations that combine various features, making it difficult to isolate specific information or suppress potentially sensitive acoustic features (such as gender or speaker identity) in real-world applications.
The authors present a two-stage training framework based on the Variational Information Bottleneck technique. In the first stage, a decoder is trained to map speech representations to text while minimizing irrelevant information from input, ensuring only features necessary for transcription are preserved. In the second stage, another decoder is trained that has access to the textual representation from previous stage and is trained to predict target labels for downstream task while minimizing information encoding. They evaluated their framework on emotion recognition and speaker identification tasks, demonstrating that the resulting representations were effectively disentangled - the textual representations could predict transcriptions but performed randomly when predicting acoustic features, while acoustic representations showed the opposite pattern.
The authors also analyzed how different layers of pre-trained and fine-tuned Wav2Vec2 models contribute to emotion recognition. They found that in models fine-tuned for automatic speech recognition (ASR), the acoustic contribution to emotion recognition decreases in higher layers while the textual contribution increases. Additionally, they showed that their framework can serve as a feature attribution method to identify the most significant frame representations for a given task, distinguishing between textual and acoustic contributions.

strengths:
The main strengths of the paper are as follows:
1. The authors provide a clear motivation and explanation for the problem under consideration.
2. The method is clearly explained, creating no confusion in grasping the idea. 
3. The experiment section is well-written with relevant experiments
4. The authors answer some key questions related to the work such as extent of disentanglement and its benefits
5. The last section of the paper talks about prior works which are in the same domain to provide readers an idea about the novelty in this work. 
6. The authors have further cited rsome extremely elevant works.

weaknesses:
Here are the main weaknesses:
1. I struggle to understand the new idea in this work because the VIB technique has existed for a while.
2. The concept of employing neural networks to learn or estimate bounds on Mutual information has existed for a long time (see 
    a. MINE: Mutual Information Neural Estimation by Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua 
        Bengio, Aaron Courville, R Devon Hjelm. 
    b. DEEP VARIATIONAL INFORMATION BOTTLENECK by Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, Kevin Murphy
    c.  Representation Learning with Contrastive Predictive Coding by Aaron van den Oord, Yazhe Li, Oriol Vinyals
3. The authors do not provide explanation in Table 1 regarding why WER increase for Fine-tuned models after disentanglement training. 
4. In Figure 2, there seems to be some strange behavior as far as prosody prediction is concerned. Pitch, intensity, rhythm, voice quality, etc have been identified as key contributors to the perception of emotion from speech. It makes little sense as to why the disentangled acoustic representation would remove that information.  
5. It has been shown before that different layers of Self-supervised models (HuBERT and W2V2) learn different types of representation from speech signal (acoustic, prosody and semantic). Therefore, section 6 reaffirms those prior studies while providing no new information for the infromed readers.

questions:
NA
</content>
</thread_object>

</thread>
<thread>
creation date:
1730340255768

thread_type:
official_review

<thread_object>
creator: reviewer_s8ol

<content>
summary:
The paper uses the Variational Information Bottleneck framework to separate textual and acoustic features of representations from SSL speech models, such as HuBERT and wav2vec2. This approach involves two stages: first, it isolates textual information by training models to transcribe content with minimized other unrelated information. The second stage targets acoustic features for tasks like emotion and speaker recognition.
They validate the proposed method through experiments on ASR, emotion recognition and speaker identification, showing its effectiveness in distinguishing between acoustic and textual attributes. This approach also has potential applications in privacy preservation, where disentangling speaker identity from transcription could help secure ASR systems.

strengths:
1. The paper is well-written and is easy to follow.
2. The proposed approach is easy to use.
3. Experiments in Section 6 align in part with the findings of previous work on layer-wise speech SSL models[1], reflecting the effectiveness of the proposed method.


[1] A. Pasad, B. Shi and K. Livescu, "Comparative Layer-Wise Analysis of Self-Supervised Speech Models," ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),

weaknesses:
1. There is previous work using the Information Bottleneck for feature disentanglement, such as in [2] and [3]. It would be better to cite these studies and highlight the distinctions between this paper and prior work.
2. Experiments comparing the proposed method with existing approaches are lacking. As there are lots of works for speech representation disentanglement like AutoVC, SpeechSplit, or FAcodec[4] , it would strengthen the paper to report the performance of at least one existing methods.
3. In Table 1, VIB loses essential information for textual representation, resulting in a much higher WER compared to Probing for HuBERT-FT and Wav2Vec2-FT. Training on a different dataset with positive outcome might help alleviate this issue.


[2] Pan, Z., Niu, L., Zhang, J., & Zhang, L. (2021). “Disentangled Information Bottleneck.” *Proceedings of the AAAI Conference on Artificial Intelligence*

[3] Gege Gao, Huaibo Huang, Chaoyou Fu, Zhaoyang Li, Ran He; “Information Bottleneck Disentanglement for Identity Swapping." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 3404-3413

[4] Ju, Zeqian, et al. "Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models." arXiv preprint arXiv:2403.03100 (2024).

questions:
1. What is the reason for using the mixture of LibriSpeech and Common Voice?
2. The WER reported in Table 1. seems to be higher than expected. What is the possible reason for that? For LibriSpeech, what subset is used in the experiments? LibriSpeech-clean or LibriSpeech-other?
3. The Information Bottleneck (IB) method focuses on retaining only information that’s relevant for predicting the target variable, filtering out anything unnecessary. This makes it dataset-dependent. For instance, when I train the stage 2 framework on a dataset for emotion recognition, the disentangled features capture emotional information but lacks speaker-specific information. I wonder if it would be possible to handle both speaker recognition and emotion recognition in stage 2, so that we preserve both emotion-related and speaker-related information. Alternatively, we could consider adding a stage 3 focused on speaker identity, while stage 2 remains dedicated to emotion recognition.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730490302377

thread_type:
official_review

<thread_object>
creator: reviewer_8vua

<content>
summary:
This paper describes an application of information bottleneck training to isolate aspects of a speech representation.  The description of the approach is quite clear.  The paper includes a variety of analyses of the learned representations that show that disentangling is achieved.

strengths:
The described approach is sensible and its specifics are clearly described.

There are a number of interesting analyses based on probing experiments to attempt to identify what information is still available in different layers of the network and assessment of the information related to distinct tasks in different frames of the input audio.

weaknesses:
The motivations in the abstract and conclusion are not well connected to the modeling and analysis.  E.g. one motivating application is to minimize the privacy risk from encoder representations.  This hasn't been assessed in the model or paper.

The disentangling approach is based on supervised tasks.  The contributions necessary for emotion classification or speaker id.  It is unclear how these learned representations would transfer to some new task. Would this approach need to be extended to a "stage 3 training process?

Multiple training stages incur additional complexity.  It would be interesting to see if these multiple objectives would be included into a single stage training.

The impact on performance in Table 1 does not deliver a consistent message.  The Transcription show substantial regressions in both of the FT representations.  The improvements to Emotion and Speaker Id are stronger but more consistent on the large sized models while on the Base sizes, there are regressions on the wave2vec variants.  This sensitivity to SSL objective and model size suggests that this approach may not be robust to new tasks or architectures.

questions:
Why subsample Librispeech and Common Voice so heavily for the transcription task?  Librispeech contains 960h of transcribed audio, but this approach uses less than 20.

How important is the ordering of the tasks?  Would the performance be identical if Emotion or Speaker Id were stage 1 and Transcription was stage 2?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730665959290

thread_type:
official_review

<thread_object>
creator: reviewer_hyny

<content>
summary:
Many standard speech representations are learned in a self-supervised way (HuBERT, w2v2, etc) and hence are, essentially, entangled blackboxes that have acoustic and textual features mixed in them in an arbitrary way. One can imagine scenarios where this is undesired, and it would be better to have a control over what/how features are encoded. This paper proposes a method building such disentangled representations, using the IB principle. As a running example, the paper opposes information that encodes textual content and acoustic information, encoding emotion or speaker identity. The paper shows that their method successfully disentangles the inputs features (variants of HuBERT, W2V2). The authors conclude with several interpretability studies of the models that use those features.

strengths:
* I find the "disentanglement evaluation" part pretty convincing.

weaknesses:
* The proposed method assumes that we have labeled tasks for all potential downstream tasks. So one starts with general-purpose self-supervised representations such as HuBERT -- which are entangled -- and ends up with representations that are (a) disentangled, but (b) are likely only useful for the tasks where we have labels. In this scenario, I am not entirely convinced that one has to use VIB (see the questions below).

* I am not entirely convinced by the motivations of the paper. If one needs to be confident that models do not use non-textual information while taking decisions, they can train models to make those decisions using pure transcripts. This is a simple baseline solution the paper should be having in mind.

* There are some concerns wrt the experimental setup -- see Questions 2, 3, 4.

questions:
1. L30 mentions that Whisper has highly entangled representations, in the same list as HuBERT or Wav2Vec2. It is never mentioned/evaluated later; is there any evidence that it is likely to have as entangled representations as SSL models from this list? It is trained purely in a supervised way for text transcription/translation, iirc, hence I would assume it learns purely text-focused features.

2. Do I understand it correctly that non-standard splits of LibriSpeech are used for the purpose of "ensuring an equal representation of gender and speaker ID" (S3.3) Is there a strong reason for that in the text transcription tasks? For the sake of comparability with all the existing literature, I would advise using standard some dev/test-{clean,other} splits.

3. Having a single linear probing classifier gets WER of ~50 for W2V2 and HuBERT. Only the pre-finetuned models get reasonable error rates. Is this a good evaluation setup to draw conclusions from?

4. What dataset is used to calculate WER in Table 1? Is this a mix of LibriSpeech and CommonVoice? Those are very different datasets, it would make sense to report them separately.

5. At least a part of motivation of the work is that by using disentangled representations one can be confident that model is using the features it is allowed to. For instance, the model doesn't rely on leaking gender or voice information when making text-based decisions. I generally get the idea, but the transcription vs gender/emotion classification task split is not a particularly convincing combination. If we are worried that the model uses something beyond the text content when making some downstream decisions, we can replace it with an (ASR + text classifier) model. Can we think of a more convincing scenario?

6. Do we actually need VIB? How different it would be if we used the labels to train a combination of ASR, Speaker and Emotion classifiers and used their outputs?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732539450322

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response to all reviewers

comment:
We sincerely appreciate the detailed and thoughtful feedback from the reviewers, which has been instrumental in improving our paper. 

Several reviews raised questions regarding the motivation for our setting and its relation to previous approaches for introducing disentanglement in speech modeling. Although we stand by our results and the relevance and novelty of our technical contribution, the reviews helped us realize that the framing of the paper and the discussion of and comparison to related work could be much improved. We have therefore made significant revisions to the paper (__marked in blue__), and hope all reviewers are willing to take the improvements and clarifications into account for their final assessments.

Specifically, we have:
- __Clarified the Targeted Setting / Desiderata:__ We have refined the description of the settings we are addressing, explicitly outlining the key desiderata of our proposed method.
- __Strengthened the Experimental Evaluation:__ We incorporated comparison to previous methods (FACodec and SpeechSplit) in Section F in the appendix (Figures F.1 and F.2). We show that, unlike the latent representations produced by our framework, the acoustic components of the previous work perform poorly in linear decoding of acoustic features, and their content vectors exhibit random performance in decoding audio transcriptions.
- __Enhanced the Related Work Discussion:__ The related work section has been thoroughly revised to provide a clearer explanation of how our approach aligns with, diverges from, and contributes to the existing body of research on disentanglement in speech processing.

We also provide responses to each individual reviewer.  We would appreciate it if you could let us know whether our responses properly address your concerns.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes to use VIB (Variational Information Bottleneck) as a framework for disentangling speech representations from neural speech models (like Wav2Vec2 and HuBERT) into two distinct components: textual content (what can be transcribed as text) and acoustic features (typically related to emotion or speaker identity). This separation could allow suppression of sensitive (speaker-related) information in future real-world applications. 
Authors present a two-stage training framework. In the first stage, a decoder is trained to map speech representations to text while minimizing irrelevant information from input, ensuring only features necessary for transcription are preserved. In the second stage, another decoder is trained that has access to the textual representation from previous stage and is trained to predict target labels for downstream task while minimizing information encoding. Authors evaluated their framework on emotion recognition and speaker identification tasks, demonstrating that the resulting representations were effectively disentangled - the textual representations could predict transcriptions but performed randomly when predicting acoustic features, while acoustic representations showed the opposite pattern. Authors also analyzed how different layers of pre-trained and fine-tuned Wav2Vec2 models contribute to emotion recognition. They found that in models fine-tuned for automatic speech recognition (ASR), the acoustic contribution to emotion recognition decreases in higher layers while the textual contribution increases. Additionally, they showed that their framework can serve as a feature attribution method to identify the most significant frame representations for a given task, distinguishing between textual and acoustic contributions.

Strengths
- The proposed approach is well motivated (after revisions) and clearly described.
- There are a number of interesting analyses based on probing experiments to attempt to identify what information is still available in different layers of the network and assessment of the information related to distinct tasks in different frames of the input audio.

Weaknesses
- The multi-stage training procedure is cumbersome, since ordering matters
- Results seem somewhat inconsistent and require lots of post-hoc explanations. Additional validations would help. 
- There is previous work using the Information Bottleneck for feature disentanglement

Authors were able to address many concerns in the rebuttal and reviewers appreciated these efforts. The main remaining weakness is that authors were not able to address all concerns in an updated revision. Given the wealth of results and comparisons, and the fact that authors want to make a claim about a very specific setting (with limited technical novelty due to VIB being a known technique), the AC does not recommend acceptance at this time, although reviews would not preclude it.