Disentangling Textual and Acoustic Features of Neural Speech Representations

## Abstract

Abstract Neural speech models build deeply entangled internal representations, which capture a variety of features (e.g., fundamental frequency, loudness, syntactic category, or semantic content of a word) in a distributed encoding.
This complexity makes it difficult to track the extent to which such representations rely on textual and acoustic information, or to suppress the encoding of acoustic features that may pose privacy risks (e.g., gender or speaker identity) in critical, real-world applications.
In this paper, we build upon the Information Bottleneck principle to propose a disentanglement framework that separates complex speech representations into two distinct components: one encoding content (i.e., what can be transcribed as text) and the other encoding acoustic features relevant to a given downstream task. We apply and evaluate our framework to emotion recognition and speaker identification downstream tasks, quantifying the contribution of textual and acoustic features at each model layer.
Additionally, we explore the application of our disentanglement framework as an attribution method to identify the most salient speech frame representations from both the textual and acoustic perspectives.

## 1 Introduction

The internal activation vectors of most modern deep learning systems, including Neural Speech Models (NSM) such as Wav2Vec2 (Baevski et al., ), HuBERT (Hsu et al., ), and Whisper (Radford et al., ),
are highly *entangled*.
This means that distinct input characteristics – such as fundamental frequency, loudness, syntactic category, or semantic features of a spoken word—are not separated into individual dimensions within the model’s latent space – but are instead intertwined within the same ones.
Entanglement is a major obstacle for our ability to interpret and to intervene; *disentanglement*, to the extent that it is possible and even if imperfect, is therefore often highly desirable.
For instance, when state-of-the-art NSMs are used in critical situations, we may want to be able to guarantee that information about the speaker’s identity, gender, or health characteristics are not used in downstream applications. However, the entangled nature of the NSM’s internal representation makes it difficult to surgically suppress such acoustic information.

In this paper, we investigate the extent to which we can learn to disentangle neural representations.
Although our proposed disentanglement framework is not modality-specific, in this paper we focus on two speech-related downstream tasks.
As our first case study, we choose emotion recognition as the target task: while the content of an utterance is a strong cue for detecting the emotion of the speaker, the additional acoustic signals offer significant advantages over text-based models.
For example, consider the utterances “I’m so happy” and “I’m fine,” where the former explicitly conveys happiness through its content, whereas the latter can represent a wide range of emotions depending on prosody and tone.
Our second case study closely follows the setup of the first, but we choose speaker identification as the target task. Here we hypothesize that text offers only limited information, and the acoustic information will play an even greater role in revealing the identity of the speaker than in the case of emotion recognition.

In Section 2, we propose a two-stage disentanglement framework (sketched in Figure1) based on the Information Bottleneck principle (Tishby et al., ; Alemi et al., ) to disentangle textual and acoustic features encoded in NSMs. In stage 1 of our framework, we train a decoder with two objectives: to map the internal representation of an existing speech model to text, but also minimize the presence of irrelevant information in these representations. The goal is to ensure that the latent representation preserves only the speech features necessary for accurate transcription, while filtering out any extraneous characteristics.
In stage 2, we train a second decoder on the same speech representations.
This decoder also has access to the latent ‘textual’ representation learned in stage 1, and is again trained with 2 objectives: to predict our target labels (emotion or speaker IDs), and to minimize the amount of information encoded in the vector.
This objective should ensure that the representation learned in stage 2 avoids encoding textual information – since the decoder already has access to it and the information minimization term discourages redundancy. Instead, it is expected to capture additional acoustic characteristics that are beneficial for the specific task.

In Section 3 we describe the models, dataset and training regime we use. In Sections 4 and 5 we evaluate our framework. We obtain highly compressed latent representations, that yield strong performance on both the standard transcription and our two target tasks. Moreover, our probing (Alain & Bengio, ) experiments show that the representations we obtain are almost perfectly disentangled from each other: while textual latent representations can predict transcriptions as effectively as the original speech representations, they exhibit random performance when predicting acoustic features (e.g., pitch or speaker identity). In contrast, acoustic representations excel at predicting these features but show random performance for transcriptions, validating the effectiveness of our disentanglement approach. The textual latent representations produced in the first stage are independent of the target task and can be easily applied to new downstream tasks.(^1^11Code and the disentangled models are publicly released at [https://github.com/hmohebbi/disentangling_representations](https://github.com/hmohebbi/disentangling_representations).)

Finally, we analyze the emergence of the two types of representations in the original speech model.
In Section 6, we trace back the contributions of the different layers of pre-trained and fine-tuned versions of the Wav2Vec2 (Baevski et al., ) to the representation of emotion. We find that as we progress through the layers, the acoustic contribution to emotion recognition significantly decreases in models fine-tuned on ASR, while the textual contribution increases, as they benefit from more accurate transcription and understanding of word polarity.
Additionally, in Section 7, we qualitatively demonstrate that our disentanglement framework can serve as a feature attribution method to highlight the most significant frame representations for a given target task. Unlike existing gradient-based methods (Sundararajan et al., ), our approach allows us to determine whether a frame’s contribution is textual or acoustic. Such disentangled attribution techniques can have many applications, e.g.,
in psychiatric research where the mismatch between textual and acoustic emotion expression was shown predictive of mood and disorders (Niu et al., ) or in bias control within speech agents.

Figure: Figure 1: An overview of our disentanglement framework: In stage 1, hidden states derived from a frozen speech model are compressed using VIB to decode transcription. In stage 2, the hidden states are compressed to decode emotion, conditioned on the frozen textual latent representations learned in stage 1. This procedure results in disentangled latent representations for textual and acoustic features relevant to the target task within each speech frame representation.
Refer to caption: x1.png

## 2 Disentanglement Framework

This section explains how we build upon the Information Bottleneck principle to disentangle textual and acoustic information within speech representations that contribute to a targeted downstream task.

### 2.1 Variational Information Bottleneck

The core idea of an Information Bottleneck (IB, Tishby et al., ) is to learn a stochastic encoding $Z$ that maximizes the prediction of a target variable $Y$, while minimizing the information retained about the input $H$. Accordingly, the loss function to be optimized based on this principle can be defined as follows:

$$ $\mathcal{L}_{\texttt{IB}}=I(Z,Y)-\beta I(H,Z)$ (1) $$

where $I(.,.)$ represents mutual information that measures the dependence between two variables. The coefficient $\beta\geq 0$ controls the trade-off between retaining information about either H or Y in Z.

The exact computation of IB loss, however, is intractable. To address this, a Variational Information Bottleneck (VIB, Alemi et al., ) has been proposed, it establishes a lower bound on the IB loss in Equation 1. The VIB loss is defined as(^2^22See Alemi et al. () for derivation.)

$$ $\mathcal{L}_{\texttt{VIB}}=\underbrace{\frac{1}{N}\sum^{N}_{n=1}\underset{z \sim p_{\theta}(z|h_{n})}{\mathds{E}}[\text{log}p_{\phi}(y_{n}|z)]}_{\texttt{ Task loss}}\;-\;\beta\underbrace{\frac{1}{N}\sum^{N}_{n=1}\text{KL}[p_{\theta} (z|h_{n}),r(z)]}_{\texttt{Information loss}}$ (2) $$

where $N$ denotes the sample size,
$p_{\phi}(y|z)$ acts as the decoder, a neural network that predicts the label $y$ from the latent representation $z$,(^3^33Formally, this represents a variational approximation of the true $p(y_{n}|z)$, which is intractable as VIB assumes that the joint distribution factorizes as $p(y_{n},z,h_{n})=p(h_{n})p(z|h_{n})p(y_{n}|h_{n})$ (Alemi et al., ).) while $p_{\theta}(z|h)$ serves as a stochastic encoder, mapping the input $h$ to the representation $z$; $r(z)$ approximates the marginal $p(z)$. The first term in the loss function encourages the encoder to preserve information relevant to the label, while the second term, the KL divergence, pushes it to discard as much information as possible. The approximate marginal $r(z)$ is typically assumed to be a spherical Gaussian (Alemi et al., ). The encoder $p(z|h)$ is parameterized using an MLP to predict the mean $\mu$ and the diagonal covariance matrix $\Sigma$: $p_{\theta}(z|h)=\mathcal{N}(z|\mu_{\theta}(h),\Sigma_{\theta}(h))$. The optimization is performed using the reparameterization trick (Kingma & Welling, ).

### 2.2 Our Proposed Method

Consider a sequence of speech representations ($\bm{h}_{1},...,\bm{h}_{T}$), produced by a pre-trained speech model, where $\bm{h}_{t}$ represents the speech frame representation at timestamp $t$, and $T$ denotes the total number of frames in a given utterance.
Given a target task, our goal is to decompose each frame representation into two distinct latent representations: $\bm{z}^{\texttt{textual}}_{t}$ and $\bm{z}^{\texttt{acoustic}}_{t}$.
Our approach to disentangling these two latent representations involves two stages, sketched in Figure 1. Both stages use frozen speech representations obtained from the same speech model as input.

#### 2.2.1 Stage 1

In the first stage, we aim to distill the textual content from speech frame representations, while discarding other non-textual features.
The textual capability of a speech model is typically evaluated using automatic speech recognition (ASR), and measured by the word error rate (WER) metric, which assesses how accurately the model can transcribe spoken utterances based on their representations.

We extract all speech frame representations for a given audio from a pre-trained speech model and compute a weighted average across model layers, with the weights learned during training. These frame representations are then given as input to a bottleneck encoder, which compresses the information into low-dimensional latent frame representations.
To decode transcriptions from the latent frame representations, we employ the Connectionist Temporal Classification (CTC, Graves et al., ) loss as the task loss term in Equation 2, thus minimizing the following loss function:

$$ $\mathcal{L}_{\texttt{CTC}}-\beta\mathcal{L}_{\texttt{Information}}$ (3) $$

In this way, we force the bottleneck encoder $p(\bm{z}^{\texttt{textual}}|\bm{h})$ to retain only the information necessary for transcription (as encouraged by the CTC loss), while discarding irrelevant features in the original representation ($\bm{h}$) (constrained by the information loss(^4^44In an ablation experiment, we also trained the encoders without the information loss. Although this led to a slight improvement in performance, it failed to disentangle the textual and acoustic information as intended (shown in Section 5), which confirms the central role of the information loss term in our framework.)). We refer to the latent frame representation for frame $t$ learned at this stage as $\bm{z}^{\texttt{textual}}_{t}$. Intuitively, these latent representations capture the minimum statistics of speech representations needed for decoding transcriptions.

#### 2.2.2 Stage 2

Our goal in the second stage is to capture acoustic features that complement the textual features learned in stage 1 and contribute to the downstream task. To achieve this, we replace the task loss in Equation 2 with the Cross-Entropy loss (CE) over the target class labels, thus, minimizing the following loss function:

$$ $\mathcal{L}_{\texttt{CE}}-\beta\mathcal{L}_{\texttt{Information}}$ (4) $$

Using labeled data for our target task in this stage, we extract speech frame representations from the same speech model and again learn a weighted average over layers. These representations are then fed into a bottleneck encoder $p(\bm{z}^{\texttt{acoustic}}|\bm{h})$ — with the same architecture as the encoder in stage 1 — to form the complimentary latent representations ($\bm{z}^{\texttt{acoustic}}_{t}$).
We apply the information loss to the frame-wise latent representations at the output of the encoder. Subsequently, we pass these latent representations through an attention layer(^5^55We choose an attention layer over a simple average pooling to later recognize the key frame representations for the sake of interpretability.) to have latent representations at the utterance level, since labels for our target tasks are assigned to the whole utterance.
Finally, the pooled latent representation $\bm{z}^{\texttt{acoustic}}$ is concatenated with the frozen textual latent representations $\bm{z}^{\texttt{textual}}$ (previously trained in Stage 1) to decode the target task. This conditional setup encourages the trainable latent representations to retain only non-textual features, particularly those absent in the textual latent representations. During the training process in this stage, no gradient updates are directed back to the $\bm{z}^{\texttt{textual}}_{t}$ learned in stage 1.

## 3 Experimental Setup

### 3.1 Training Details

For training VIB, we follow Alemi et al. () and model $r(\bm{z})$ and $p(\bm{z}|\bm{h})$ as multivariate Gaussian distributions: $r(\bm{z})=\mathcal{N}(\bm{z}|\bm{\mu}\!=\!\bm{0},\bm{\Sigma}\!=\!\bm{1})$ and $p(\bm{z}|\bm{h})=\mathcal{N}(\bm{z}|\bm{\mu}(\bm{h}),\bm{\Sigma}(\bm{h}))$.
The bottleneck encoders to estimate $\bm{\mu}$ and $\bm{\Sigma}$ consist of two shared linear layers with the same dimensionality as the original hidden representations ($\bm{h}_{t}$), followed by independent $d$-dimensional layers for each, with GELU (Hendrycks & Gimpel, ) activation functions in between.
We experiment with different bottleneck dimensions $d=\{16,32,64,128,256\}$ as the output size of the bottleneck encoders.
To estimate the gradients, we employ the reparameterization trick (Kingma & Welling, ): $\bm{z}_{t}=\bm{\mu}(\bm{h}_{t})+\bm{\Sigma}(\bm{h}_{t})\odot\bm{\epsilon}$, where $\bm{\epsilon}$ is sampled from the normal distribution $\mathcal{N}(\bm{0},\bm{1})$.
During inference, we use $\bm{z}_{t}=\bm{\mu}(\bm{h}_{t})$.
In our implementation, we gradually increase the $\beta$ coefficient linearly from $0.1$ to $1$ during training.
For decoding, we utilize a randomly initialized linear projection into $C$ classes, where $C\!=\!32$ for transcription (corresponding to the number of target characters), $C\!=\!4$ for emotion recognition, and $C\!=\!24$ for speaker identification. The training hyperparameters are detailed in Appendix A.

### 3.2 Target Models

To obtain speech representations, we use two prominent self-supervised speech models: Wav2Vec2 (Baevski et al., ) and HuBERT (Hsu et al., ) in their different sizes: Base (12-Transformer layers, 768-hidden size) and Large (24, 1024), obtained from
the HuggingFace (Wolf et al., ) library. Our experiments include both pre-trained (on raw speech) and fine-tuned(^6^66The HuBERT model lacks a released fine-tuned checkpoint in the Base size.) (on transcribed speech) versions of these models.
Both models employ the Transformer (Vaswani et al., ) architecture and learn speech representations through masked prediction in a self-supervised manner. Wav2Vec2 employs a contrastive loss to identify the masked speech frame among distractors, while HuBERT uses k-means clustering to create prediction targets. The models are further fine-tuned with additional labeled speech data by optimizing a linear classifier using CTC loss to decode transcription.

### 3.3 Data

##### Transcription.

For training in stage 1, we use subsets of two widely used read speech corpora: LibriSpeech (Panayotov et al., ) and Mozilla’s Common Voice 17.0 (Ardila et al., ). The former is derived from audiobooks, while the latter is recorded by contributors reading sentences displayed on a screen.
We randomly select 5,000 examples from each corpus, ensuring an equal representation of gender and speaker ID, with each sample having a maximum duration of 14 seconds. Following (Baevski et al., ), we remove non-spoken special characters (e.g., commas and periods) from transcriptions, as these are not included in our target vocabulary. We reserve 1000 examples from each corpus as test data, leaving the remaining for training, which results in 17.4, and 3.2 hours of transcribed speech data, in total, for training and test sets, respectively.

##### Target tasks.

For emotion data in stage 2, we utilize IEMOCAP (Busso et al., ) database, which consists of five dyadic sessions involving ten speakers (5 male, 5 female).
Following prior research (Li et al., ; ), we exclude utterances without transcripts and combine *Happy* and *Excited* labels to form a 4-way classification task. We then undersample the dataset to balance emotion classes, resulting in 4,064 utterances ($\sim\!5$ hours of audio, with an average duration of 4.4 seconds per segment). Each utterance is assigned one emotion from the label set: {*Angry*, *Happy*, *Neutral*, *Sad*}.
For Speaker Identity as our target task, we utilize a subset of Mozilla’s Common Voice 17.0 dataset (Ardila et al., ), consisting of 4,000 training and 1,000 test samples, stratified by two genders (male and female) and 24 speaker identities.
All audio files in this study are resampled to 16 kHz to match the sampling rate used for the pre-training data of the target models.

## 4 Task performance after VIB training

In this section, we test the effectiveness of the disentangled representations against the original entangled representations on the downstream tasks. We will verify the disentanglement in the next section.
For comparison, we also train identically structured decoders which rely on the original hidden states ($h_{t}$); the difference with VIB training is that the information loss is excluded during the training process. The performance of these classifiers (which can be viewed as a repurposed form of probing (Alain & Bengio, ; Tenney et al., )) serves as a strong baseline, representing the performance achievable relying on the hidden states, without compressing them into latent representations.

**Table 1: VIB ($d\!=\!128$) and probing performance. Lower WER and higher Accuracy are better. Random baselines: $\text{WER}\!=\!100$ for transcription, $\text{Accuracy}\!=\!25$ for emotion, and $\text{Accuracy}\!=\!4.1$ for speaker identification. Scores are averaged over three runs with different random seeds.**
|  |  | Stage 1 | Stage 2 | Stage 2 |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
|  |  | Transcription (WER $\downarrow$) | Emotion (Acc. $\uparrow$) | Speaker Id. (Acc. $\uparrow$) |  |  |  |
| Model | Size | Probing | VIB | Probing | VIB | Probing | VIB |
| HuBERT | Base | 49.7 | 46.0 | 61.8 | 62.7 | 97.8 | 99.1 |
| Large | 38.3 | 39.1 | 57.3 | 66.1 | 92.3 | 98.4 |  |
| Wav2Vec2 | Base | 53.7 | 49.5 | 61.4 | 58.9 | 99.8 | 97.9 |
| Large | 49.9 | 49.5 | 62.2 | 65.9 | 97.9 | 99.8 |  |
| HuBERT-FT | Large | 6.9 | 25.6 | 54.7 | 64.8 | 90.5 | 98.2 |
| Wav2Vec2-FT | Base | 10.3 | 16.7 | 63.5 | 56.6 | 99.6 | 98.2 |
| Large | 7.7 | 11.5 | 62.1 | 63.4 | 98.5 | 99.6 |  |

Table 1 reports both VIB and probing performances, averaged over three runs with different random seeds, for transcription, emotion recognition, and speaker identification tasks.
For the transcription task at stage 1, VIB demonstrates a similar or sometimes even lower word error rate (WER) compared to probing classifiers, implying the success of VIB training in compressing essential information for audio transcription ($\text{WER}\!=\!100$ serving as the performance of the random baseline).(^7^77We also evaluated the transcription performance using character error rate (CER) as an additional metric; results are in Appendix C.1.)
As expected, representations derived from fine-tuned models show better transcription performance compared to those from pre-trained models (for both VIB and probing) as they are specifically tuned for transcription.
The same success is evident in decoding our target tasks in stage 2, where we report the accuracy of both VIB and probing classifiers for emotion recognition and speaker identification tasks (the random baseline accuracy is 25 and 4.1, respectively).
Overall, the classifier benefits more from the specialized, compressed representations than from the original hidden states, as VIB encourages retaining only task-relevant information, resulting in more robust representations.

We also found VIB performance consistent across various bottleneck dimensions; see the results in Appendix B.
Both VIB and probing learn weights for layer averaging (see Figure 1). These weights are necessary because the information is not uniformly distributed across the NSM layers. As shown in Appendix D, both approaches use these weights; the weights provide insights into the contribution of individual layers to the acoustic and textual features.

## 5 Evaluation of disentanglement

### 5.1 Sanity Check Probing

The product of our training procedure (described in Sec. 2) are disentangled latent representations $\bm{z}^{\texttt{textual}}_{t}$ and $\bm{z}^{\texttt{acoustic}}_{t}$ for each speech frame representation $h_{t}$. In this section, we investigate these latent representations to validate if they are truly disentangled by probing them for various types of textual and acoustic information.
Given an audio input, we expect that $\bm{z}^{\texttt{textual}}_{t}$, which is specialized for text, should not encode any aspects of the acoustic characteristics of the audio. In contrast, $\bm{z}^{\texttt{acoustic}}_{t}$, which is specialized for audio features, should not contain any information about the audio transcription.

#### 5.1.1 Setup

To assess textual capability, we train a linear probing classifier on frozen latent representations to predict their transcriptions.
To estimate acoustic capability, we train a set of probing classifiers followed by an attention layer on latent representations to predict various acoustic features at the utterance level, including Mean Intensity, Mean Pitch, Gender, and Speaker Identity.
For comparison. we also probe the original hidden states for the same objectives with the identically-structured classifiers.
We utilize a subset of Mozilla’s Common Voice 17.0 dataset (Ardila et al., ), consisting of 4,000 training and 1,000 test samples, stratified by two genders (male and female)(^8^88The exclusion of other groups is due to the binary labeling in the dataset, rather than a choice by the authors.) and 24 speakers. We extract labels for acoustic features directly from the raw audio waveforms using the Parselmouth toolkit (Jadoul et al., ). We then discretize them into four equally sized buckets based on quantiles to cast the task as a four-way classification problem.

Figure: Figure 2: Probing performances of latent representations ($d\!=\!128$) learned at stages 1 and 2, along with hidden states derived from Large models for transcription and a set of audio features. For the WER metric, the lower score is better, while for other metrics, the higher is better. The dashed line in each plot represents the random baseline.
Refer to caption: x2.png

#### 5.1.2 Results

Figure 2 illustrates the classification results for Large models. Dashed lines represent the random baseline performance. As we can see, acoustic latent representations (for both target tasks; $\bm{z}^{\texttt{acoustic}}_{\texttt{emotion}}$ and $\bm{z}^{\texttt{acoustic}}_{\texttt{speaker}}$) exhibit no awareness of the textual content of the audio as their performance for CTC matches the random baseline ($\text{WER}\!=\!1$). Conversely, textual latent representations are as effective as the original hidden states and – for pre-trained models – even outperform them at decoding transcription.

Looking into predicting acoustic features, textual latent representations consistently show random performance, suggesting no acoustic features are encoded within them. Acoustic latent representations, however, show substantial probing performance despite not having any explicit acoustic objective in their training at stage 2. Interestingly, acoustic latent representations for the task of speaker identification are better at encoding acoustic features than those of emotion recognition. It is likely due to acoustic information playing a greater role in revealing the speaker identity.

Additionally, in contrast to $\bm{z}^{\texttt{acoustic}}_{\texttt{speaker}}$, acoustic latent representations for emotion ($\bm{z}^{\texttt{acoustic}}_{\texttt{emotion}}$) do not match the performance of the original hidden states. For example, for the Wav2Vec2 model, the probing performance for Speaker ID based on hidden states is 0.98, while it is 0.26 for acoustic latent representations (the random baseline is $1/24\approx 0.04$). This disparity suggests that not all those acoustic features encoded in the hidden representations are crucial for recognizing emotion, thus, not all of those features were retained in stage 2. These findings could be important in real-world scenarios where, for privacy protection, encoding acoustic features without precisely identifying the speaker can be essential.
The pattern of the probing results is also similar for Base models and for various bottleneck dimensions, as reported in Appendix E.

### 5.2 Qualitative Evaluation

Figure: Figure 3: t-SNE plots of the textual and acoustic latent representations for Wav2Vec2-Large, marked and colored according to their transcription and emotion labels, respectively.
Refer to caption: x3.png

Next, we visualize the latent representations to gain insight into how they have been encoded in the representation space for emotion recognition. We make use of RAVDESS (Livingstone & Russo, ) dataset, which contains only two identical statements spoken by 24 actors (12 male, 12 female) with 4 different emotions. This makes it ideal for this analysis, as the linguistic content remains constant across utterances with different emotions.

We select examples with matching emotion labels from the IEMOCAP dataset, resulting in 384 utterances (each averaging 3.7 seconds).
We obtain the speech representations from the large Wav2Vec2 model and generate their corresponding latent representations using the bottleneck encoders trained in stages 1 and 2 by doing only inference without any further training. Then, we compute the average of frame representations over all frames in each utterance and apply t-SNE.

Figure 3 shows a 2D projection of these latent representations with data points marked and colored according to their transcription and emotion labels, respectively.
The textual latent representations (learned in stage 1) are perfectly clustered according to their transcriptions. In contrast, the acoustic latent representations (learned in stage 2) are not separated by transcription, suggesting that no textual information is retained there. Instead, these acoustic representations are roughly clustered by emotion, which nicely aligns with our desired goal.
The pattern is also the same for the Base model, reported in Appendix F.1.

## 6 Layerwise emotion contribution

With our disentangled framework established and validated, we can now quantify, separately, the extent to which each layer in a speech model contributes textually and acoustically to the target task of emotion recognition.
We first train the latent representations using the same setup as we described in Section 2, but instead of using a weighted layer average in the input, we use the frame representations from a specific layer of the model. The layerwise results for stages 1 and 2 are shown in Figure 4, with the black horizontal dashed line representing random performances.

For HuBERT, the transcription ability improves as we move through the layers. In contrast, Wav2Vec2 shows a U-shaped pattern, with the best performance in the middle layers and the final layers being unable to decode transcription. For Wav2Vec2-FT, however, transcription performance improves sharply in the last layers, which is expected given that the model is fine-tuned for ASR, indicating substantial changes in the last layers during fine-tuning. In stage 2 (middle panel), models perform best at emotion recognition in the middle layers. Compared to pre-trained, the fine-tuned model (Wav2Vec2-FT) shows a significant decline in emotion recognition performance in the final layers, where these layers lose acoustic information in favor of encoding text transcription.

Figure: Figure 4: Layerwise performance of VIB ($d\!=\!128$) for transcription at stage 1 (left) and emotion classification at stage 2 (middle), compared to layerwise performance using original hidden state activations. Right: layerwise textual and acoustic contribution to emotion classification, compared to layerwise performance using hidden states of Base models. Lower WER and higher accuracy are better. The black horizontal dashed line represents the random baseline.
Refer to caption: x4.png

### 6.1 Layerwise emotion probing

We next train a probing classifier (followed by an attention layer) on top of frozen $\bm{z}^{\texttt{textual}}_{t}$ and $\bm{z}^{\texttt{acoustic}}_{t}$ latent representations to decode emotion. Each latent representation is specialized to preserve either textual or acoustic features from the original hidden states of each model layer. Therefore, the probing performance reveals the extent to which these textual and acoustic features contribute to emotion recognition. For comparison, we also train the same probing classifier on the original hidden states across each layer.

Figure 4 (right) illustrates emotion probing performances.
Comparing pre-trained and fine-tuned models, the latent acoustic representations learned from the final layers of Wav2Vec-FT contribute significantly less to emotion recognition. This is likely due to the model losing some acoustic information during fine-tuning in favor of transcription capabilities. We can see, however, that these layers benefit more from textual features in predicting emotion, as their representations offer more accurate transcriptions.
Also, Compared to Wav2Vec2, HuBERT shows a greater contribution to emotion recognition; acoustically in the middle layers, and textually in the last layer.

## 7 Localizing salient textual and acoustic frames

In this section, we use our disentanglement framework to localize salient input features for the target tasks. The attention layer in stage 2 of our framework (see Figure 1) can be used to identify those frames in the original audio input whose latent representations contribute most to our target tasks.
Crucially, the disentanglement mechanism allows us to clearly separate the contributions of acoustic features from those of textual features, providing insight into their individual roles. This disentangled attribution could be particularly useful in fields like psychiatry, where differences between textual and acoustic emotional expressions can aid in the diagnosis of disorders (Niu et al., ), or in detecting bias in speech agents’ responses to user requests. While a comprehensive evaluation of disentangled attribution is beyond the scope of this work, we do conduct a preliminary investigation.

Finding salient input features for model decisions is often done by computing the gradient of the model’s output with respect to the inputs (Sundararajan et al., ; Ancona et al., ; Yuan et al., ; Samek et al., ).
To compare our attention scores with gradient-based attribution scores, we train another classifier for the target task on the original hidden states of the Wav2Vec2 model and use Integrated Gradients (IG) to compute attribution to individual frames (Sundararajan et al., ). We normalize the IG scores to sum to $1$, ensuring a fair comparison with attention scores.

We then obtain frame-by-frame distributions of both acoustic and sentiment features to compare them directly with attribution results. For acoustic features, we focus on intensity and pitch, identifying and representing peaks and valleys in their temporal patterns. To analyze sentiment, we use the spaCy toolkit (Honnibal & Montani, ) to annotate word-level polarity (positivity, negativity, or neutrality, with 0 0 representing neutrality) within the utterances. Using Montreal Forced Aligner (McAuliffe et al., ), we can map these word-level labels to frames (see details in Appendix G). As a result, each feature vector for a speech frame includes the following dimensions: (1) the presence of a peak or valley in intensity, (2) the same for pitch, and (3) the presence of a ‘sentiment-laden’ word, all normalized to the range $[0,1]$.

We then compute the dot product between the frame-wise attribution scores and the corresponding feature vectors.
Figure 5 presents these results, averaged across all examples in the IEMOCAP test set. The figure demonstrates that acoustic attention effectively captures peaks and valleys in acoustic features (intensity and pitch), while textual attention focuses on word polarity. Both have higher agreement with features than Integrated Gradient scores (see Appendix H for qualitative examples).
Note that textual attention exhibits fairly high similarity with acoustic features as polar words are often pronounced with emotional emphasis.

Figure: Figure 5: Dot product of attribution scores and different acoustic and textual cues.
Refer to caption: x7.png

## 8 Related Work

Learning disentangled representations of complex data, such that distinct factors are separable and controllable has been the focus of many studies. Bengio et al. () motivate this approach within the context of deep learning, while Wang et al. () provide a general up-to-date overview.

For speech, disentanglement has been used for controllable style transfer in voice conversion, where the goal is to modify para-linguistic information while preserving linguistic content. Speaker conversion is a canonical example of this line of work: for example van den Oord et al. () evaluate their modality-agnostic discrete representation learning framework on this task, while Polyak et al. () apply discrete representation learning specifically for speech re-synthesis.
In an alternative approach, Qian et al. () propose AutoVC, as a style transfer autoencoder framework, that disentangles speaker identity information from content by training an autoencoder conditioned on speaker identity representations (encoded by another model).
Nagrani et al. () exploit videos containing speaker faces as a self-supervision signal for learning disentangled speech embeddings.
SpeechSplit (Qian et al., ) extends AutoVC by constraining latent representation dimensions and adding noise, splitting speech representations into four specific attributes (content, pitch, timbre, and rhythm).

With the assumption that most voice synthesis tasks can be defined by controlling pitch, amplitude, linguistic content, and timbre, Choi et al. (; ) develop a framework for manipulating and synthesizing voice signals by autoencoding these four properties and reconstructing the waveform using a combination of information perturbation and bottleneck techniques.
In a different approach Wang et al. () trained an adversarial network to minimize correlations between the representations of content, pitch, timbre and rhythm.
Wang et al. () propose a multi-modal information bottleneck approach aimed at disentangling relevant visual and textual features to enhance the interpretability of vision-language models.

Qu et al. () build upon the quantization module of HuBERT (Hsu et al., ) and propose Prosody2Vec, which consists of four modules: a unit encoder that employs a deep network to map the discrete speech representations produced by the HuBERT’s quantization module to continuous latent representations; a pre-trained deep network to encode speaker; a prosody encoder; and a decoder that reconstructs mel-spectrograms from the outputs of the three encoders. The trainable parameters are first pre-trained on spontaneous speech and then fine-tuned for emotion recognition.
In the current study, our goal is to propose a general method which accommodates many possible factors of variations and makes it possible to configure them. Unlike Prosody2Vec, our approach requires fewer trainable parameters, is directly trained on the target task, and is applicable to any speech model.

## 9 Conclusions

Our work presents a disentanglement framework based on the Information Bottleneck principle, which effectively separates entangled representations of neural speech models into distinct textual and acoustic components, while retaining only features relevant to the target tasks.
In our experiments with the emotion recognition and speaker identification tasks, we have demonstrated that the framework can effectively isolate key features, improving interpretability, whilst maintaining the performance of the original model.
We believe the framework holds potential for a range of applications where disentanglement is key. One important class of such applications are speech recognition systems where privacy-preservation is a key concern for responsible deployment (and sometimes even a legal requirement); separating sensitive speaker-specific information from task-relevant information is essential in such systems.
The framework is also proving useful, as we showed, in providing a route to perform disentangled feature attribution, revealing the most significant speech frame representations from both textual and acoustic perspectives.
We hope our findings thus help advance neural speech technology, by helping to improve transparency and control.

A unique aspect of our framework is its modality-independent architecture: in principle, any type of entangled multi-modal representation can be disentangled as long as the right pair of tasks are selected for training in stage 1 and 2. This opens up the application of the framework beyond just neural speech recognition, for instance to grounded language models, where we might want to disentangle linguistic and visual information. We leave further exploration of these possibilities for future work.

## Acknowledgments

Support from the Netherlands Organization for Scientific Research (NWO), through the National Research Agenda project “InDeep: Interpreting Deep Learning Models for Text and Sound” (NWA-ORC NWA.1292.19.399) and
the VICI project “Language-Driven Modularity in Neural Networks” (VI.C.212.053) is gratefully acknowledged.
We also thank SURF ([www.surf.nl](www.surf.nl)) for the support in using the National Supercomputer Snellius. The work was partially done while HM was visiting the University of Edinburgh.

## Appendix A Training Hyperparameters

In our experiments, all models were trained for $50$ epochs using the AdamW optimizer with gradient norm clipping. The learning rate was set to $1\mathrm{e}{-3}$ and $1\mathrm{e}{-4}$ for Base and Large models, respectively, with a warmup ratio of $0.1$ and a cosine decay scheduler, together with weight decay.
Specifically for training transcription prediction using CTC loss, we use a batch size of $1$; the effective batch size here is the number of frame representations since the CTC loss is computed across all frames. For other training objectives, we use a batch size of $8$.

## Appendix B Ablation on the bottleneck dimension

Figure: Figure B.1: VIB performance across various bottleneck dimensions for Base models.
Refer to caption: x8.png

Figure: Figure B.2: VIB performance across various bottleneck dimensions for Large models.
Refer to caption: x11.png

## Appendix C Transcription evaluation with other metrics

Figure: Figure C.1: VIB performance at stage 1, evaluated by Character Error Rate (CER) metric, across various bottleneck dimensions for Base (left) and Large (right) models.
Refer to caption: x14.png

## Appendix D Layer Weight Averaging

In the input of both VIB and probing training setups in Section 4, we train weights for layer averaging (see Figure 1).
Let us have a look at these trained layer weights to see the contribution of each layer at each stage.
Figure D.1 shows these layer weights for both Base and Large models.
Overall, layer weights learned in the VIB setup closely follow the weights trained with original hidden states in the probing setup.

In stage 1 we observe that, in pre-trained models, the upper-middle layers are more capable of decoding transcription. In fine-tuned models, however, this information shifts to and becomes concentrated at the final layer. This pattern is consistent across both Base and Large model sizes as well as for different bottleneck dimensions.

For emotion recognition and speaker identity tasks in stage 2, however, there is a notable difference between Base and Large models. In Base models, the earlier layers contribute more in both pre-trained and fine-tuned setups. In contrast, layers in Large models contribute uniformly, suggesting that the acoustic information useful for these target tasks is distributed across layers in Large models.

Figure: Figure D.1: Layer weights of Base (left) and Large (right) models, trained at each VIB stage, compared to the probing on hidden states.
Refer to caption: x16.png

## Appendix E Replication of sanity check probing experiment for other model sizes and bottleneck dimensions

Figure: Figure E.1: Probing performances of latent representations ($d\!=\!128$) learned at stages 1 and 2, along with hidden states derived from Base models for transcription and a set of audio features.
Refer to caption: x18.png

Figure: Figure E.2: Probing performances of latent representations ($d\!=\!32$) learned at stages 1 and 2, along with hidden states derived from Large models for transcription and a set of audio features
Refer to caption: x19.png

## Appendix F Replication of t-SNE visualization

Figure: Figure F.1: t-SNE plots of the textual and acoustic latent representations for the Wav2Vec2-Base model, marked and colored according to their transcription and emotion labels, respectively.
Refer to caption: x20.png

## Appendix G Aligning Frame Representations with Audio Frames and Transcription

To map the frame representations to their original input frames and their corresponding word transcriptions, we use Montreal Forced Aligner (McAuliffe et al., , MFA). Using this tool, we extract the start time ($t_{s}$) and the end time ($t_{e}$) of each word in an utterance, and map them to boundary frames $f_{s}$ and $f_{e}$:

$$ ${f=\lceil\frac{t}{\mathcal{T}}\times T}\rceil$ (5) $$

where $\mathcal{T}$ and $T$ denote the total time of a given audio and the total number of frames in the frame representation, respectively.

## Appendix H Qualitative examples for feature attribution

Figure H.1 showcases the textual and acoustic attention scores derived from our disentanglement framework for Wav2Vec2, together with the gradient scores and framewise acoustic features (pitch and intensity), for two utterances from the IEMOCAP (Busso et al., ) test dataset labeled ‘*Happy*’.
Both figures are vertically segmented according to the word’s time stamps in the utterances.
In the left panel, the textual attention highlights part of the frames corresponding to the positive words “THANKS” and “NICE”, while, the acoustic attention peaks at a frame where there is a drastic change for both pitch and intensity, at the beginning of the pronunciation of the word “OH,”.
In the right panel for the second utterance, we can see a uniform pattern for textual attention, which makes sense as there is no textual cue in the utterance for emotion recognition.
However, the acoustic attention again shows significant peaks where pitch and intensity change drastically.
In contrast, Integrated Gradient tends to highlight the silent parts of the audio.

Figure: Figure H.1: Textual and acoustic attentions, and Integrated Gradient scores, together with two framewise acoustic features (Pitch and Intensity) for two utterances labeled *Happy*.
Refer to caption: x21.png

## Peer Reviews

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. L30 mentions that Whisper has highly entangled representations, in the same list as HuBERT or Wav2Vec2. It is never mentioned/evaluated later; is there any evidence that it is likely to have as entangled representations as SSL models from this list? It is trained purely in a supervised way for text transcription/translation, iirc, hence I would assume it learns purely text-focused features.

2. Do I understand it correctly that non-standard splits of LibriSpeech are used for the purpose of "ensuring an equal representation of gender and speaker ID" (S3.3) Is there a strong reason for that in the text transcription tasks? For the sake of comparability with all the existing literature, I would advise using standard some dev/test-{clean,other} splits.

3. Having a single linear probing classifier gets WER of ~50 for W2V2 and HuBERT. Only the pre-finetuned models get reasonable error rates. Is this a good evaluation setup to draw conclusions from?

4. What dataset is used to calculate WER in Table 1? Is this a mix of LibriSpeech and CommonVoice? Those are very different datasets, it would make sense to report them separately.

5. At least a part of motivation of the work is that by using disentangled representations one can be confident that model is using the features it is allowed to. For instance, the model doesn't rely on leaking gender or voice information when making text-based decisions. I generally get the idea, but the transcription vs gender/emotion classification task split is not a particularly convincing combination. If we are worried that the model uses something beyond the text content when making some downstream decisions, we can replace it with an (ASR + text classifier) model. Can we think of a more convincing scenario?

6. Do we actually need VIB? How different it would be if we used the labels to train a combination of ASR, Speaker and Emotion classifiers and used their outputs?
soundness: 2
strengths: * I find the "disentanglement evaluation" part pretty convincing.
summary: Many standard speech representations are learned in a self-supervised way (HuBERT, w2v2, etc) and hence are, essentially, entangled blackboxes that have acoustic and textual features mixed in them in an arbitrary way. One can imagine scenarios where this is undesired, and it would be better to have a control over what/how features are encoded. This paper proposes a method building such disentangled representations, using the IB principle. As a running example, the paper opposes information that encodes textual content and acoustic information, encoding emotion or speaker identity. The paper shows that their method successfully disentangles the inputs features (variants of HuBERT, W2V2). The authors conclude with several interpretability studies of the models that use those features.
weaknesses: * The proposed method assumes that we have labeled tasks for all potential downstream tasks. So one starts with general-purpose self-supervised representations such as HuBERT -- which are entangled -- and ends up with representations that are (a) disentangled, but (b) are likely only useful for the tasks where we have labels. In this scenario, I am not entirely convinced that one has to use VIB (see the questions below).

* I am not entirely convinced by the motivations of the paper. If one needs to be confident that models do not use non-textual information while taking decisions, they can train models to make those decisions using pure transcripts. This is a simple baseline solution the paper should be having in mind.

* There are some concerns wrt the experimental setup -- see Questions 2, 3, 4.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: Why subsample Librispeech and Common Voice so heavily for the transcription task?  Librispeech contains 960h of transcribed audio, but this approach uses less than 20.

How important is the ordering of the tasks?  Would the performance be identical if Emotion or Speaker Id were stage 1 and Transcription was stage 2?
soundness: 3
strengths: The described approach is sensible and its specifics are clearly described.

There are a number of interesting analyses based on probing experiments to attempt to identify what information is still available in different layers of the network and assessment of the information related to distinct tasks in different frames of the input audio.
summary: This paper describes an application of information bottleneck training to isolate aspects of a speech representation.  The description of the approach is quite clear.  The paper includes a variety of analyses of the learned representations that show that disentangling is achieved.
weaknesses: The motivations in the abstract and conclusion are not well connected to the modeling and analysis.  E.g. one motivating application is to minimize the privacy risk from encoder representations.  This hasn't been assessed in the model or paper.

The disentangling approach is based on supervised tasks.  The contributions necessary for emotion classification or speaker id.  It is unclear how these learned representations would transfer to some new task. Would this approach need to be extended to a "stage 3 training process?

Multiple training stages incur additional complexity.  It would be interesting to see if these multiple objectives would be included into a single stage training.

The impact on performance in Table 1 does not deliver a consistent message.  The Transcription show substantial regressions in both of the FT representations.  The improvements to Emotion and Speaker Id are stronger but more consistent on the large sized models while on the Base sizes, there are regressions on the wave2vec variants.  This sensitivity to SSL objective and model size suggests that this approach may not be robust to new tasks or architectures.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. What is the reason for using the mixture of LibriSpeech and Common Voice?
2. The WER reported in Table 1. seems to be higher than expected. What is the possible reason for that? For LibriSpeech, what subset is used in the experiments? LibriSpeech-clean or LibriSpeech-other?
3. The Information Bottleneck (IB) method focuses on retaining only information that’s relevant for predicting the target variable, filtering out anything unnecessary. This makes it dataset-dependent. For instance, when I train the stage 2 framework on a dataset for emotion recognition, the disentangled features capture emotional information but lacks speaker-specific information. I wonder if it would be possible to handle both speaker recognition and emotion recognition in stage 2, so that we preserve both emotion-related and speaker-related information. Alternatively, we could consider adding a stage 3 focused on speaker identity, while stage 2 remains dedicated to emotion recognition.
soundness: 2
strengths: 1. The paper is well-written and is easy to follow.
2. The proposed approach is easy to use.
3. Experiments in Section 6 align in part with the findings of previous work on layer-wise speech SSL models[1], reflecting the effectiveness of the proposed method.


[1] A. Pasad, B. Shi and K. Livescu, "Comparative Layer-Wise Analysis of Self-Supervised Speech Models," ICASSP 2023 - 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
summary: The paper uses the Variational Information Bottleneck framework to separate textual and acoustic features of representations from SSL speech models, such as HuBERT and wav2vec2. This approach involves two stages: first, it isolates textual information by training models to transcribe content with minimized other unrelated information. The second stage targets acoustic features for tasks like emotion and speaker recognition.
They validate the proposed method through experiments on ASR, emotion recognition and speaker identification, showing its effectiveness in distinguishing between acoustic and textual attributes. This approach also has potential applications in privacy preservation, where disentangling speaker identity from transcription could help secure ASR systems.
weaknesses: 1. There is previous work using the Information Bottleneck for feature disentanglement, such as in [2] and [3]. It would be better to cite these studies and highlight the distinctions between this paper and prior work.
2. Experiments comparing the proposed method with existing approaches are lacking. As there are lots of works for speech representation disentanglement like AutoVC, SpeechSplit, or FAcodec[4] , it would strengthen the paper to report the performance of at least one existing methods.
3. In Table 1, VIB loses essential information for textual representation, resulting in a much higher WER compared to Probing for HuBERT-FT and Wav2Vec2-FT. Training on a different dataset with positive outcome might help alleviate this issue.


[2] Pan, Z., Niu, L., Zhang, J., & Zhang, L. (2021). “Disentangled Information Bottleneck.” *Proceedings of the AAAI Conference on Artificial Intelligence*

[3] Gege Gao, Huaibo Huang, Chaoyou Fu, Zhaoyang Li, Ran He; “Information Bottleneck Disentanglement for Identity Swapping." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 3404-3413

[4] Ju, Zeqian, et al. "Naturalspeech 3: Zero-shot speech synthesis with factorized codec and diffusion models." arXiv preprint arXiv:2403.03100 (2024).

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: NA
soundness: 3
strengths: The main strengths of the paper are as follows:
1. The authors provide a clear motivation and explanation for the problem under consideration.
2. The method is clearly explained, creating no confusion in grasping the idea. 
3. The experiment section is well-written with relevant experiments
4. The authors answer some key questions related to the work such as extent of disentanglement and its benefits
5. The last section of the paper talks about prior works which are in the same domain to provide readers an idea about the novelty in this work. 
6. The authors have further cited rsome extremely elevant works.
summary: This paper proposes a new framework for disentangling speech representations from neural speech models (like Wav2Vec2 and HuBERT) into two distinct components: textual content (what can be transcribed as text) and acoustic features (like emotion or speaker identity). This separation is important because neural speech models typically create deeply entangled internal representations that combine various features, making it difficult to isolate specific information or suppress potentially sensitive acoustic features (such as gender or speaker identity) in real-world applications.
The authors present a two-stage training framework based on the Variational Information Bottleneck technique. In the first stage, a decoder is trained to map speech representations to text while minimizing irrelevant information from input, ensuring only features necessary for transcription are preserved. In the second stage, another decoder is trained that has access to the textual representation from previous stage and is trained to predict target labels for downstream task while minimizing information encoding. They evaluated their framework on emotion recognition and speaker identification tasks, demonstrating that the resulting representations were effectively disentangled - the textual representations could predict transcriptions but performed randomly when predicting acoustic features, while acoustic representations showed the opposite pattern.
The authors also analyzed how different layers of pre-trained and fine-tuned Wav2Vec2 models contribute to emotion recognition. They found that in models fine-tuned for automatic speech recognition (ASR), the acoustic contribution to emotion recognition decreases in higher layers while the textual contribution increases. Additionally, they showed that their framework can serve as a feature attribution method to identify the most significant frame representations for a given task, distinguishing between textual and acoustic contributions.
weaknesses: Here are the main weaknesses:
1. I struggle to understand the new idea in this work because the VIB technique has existed for a while.
2. The concept of employing neural networks to learn or estimate bounds on Mutual information has existed for a long time (see 
    a. MINE: Mutual Information Neural Estimation by Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua 
        Bengio, Aaron Courville, R Devon Hjelm. 
    b. DEEP VARIATIONAL INFORMATION BOTTLENECK by Alexander A. Alemi, Ian Fischer, Joshua V. Dillon, Kevin Murphy
    c.  Representation Learning with Contrastive Predictive Coding by Aaron van den Oord, Yazhe Li, Oriol Vinyals
3. The authors do not provide explanation in Table 1 regarding why WER increase for Fine-tuned models after disentanglement training. 
4. In Figure 2, there seems to be some strange behavior as far as prosody prediction is concerned. Pitch, intensity, rhythm, voice quality, etc have been identified as key contributors to the perception of emotion from speech. It makes little sense as to why the disentangled acoustic representation would remove that information.  
5. It has been shown before that different layers of Self-supervised models (HuBERT and W2V2) learn different types of representation from speech signal (acoustic, prosody and semantic). Therefore, section 6 reaffirms those prior studies while providing no new information for the infromed readers.

Meta-review: This paper proposes to use VIB (Variational Information Bottleneck) as a framework for disentangling speech representations from neural speech models (like Wav2Vec2 and HuBERT) into two distinct components: textual content (what can be transcribed as text) and acoustic features (typically related to emotion or speaker identity). This separation could allow suppression of sensitive (speaker-related) information in future real-world applications. 
Authors present a two-stage training framework. In the first stage, a decoder is trained to map speech representations to text while minimizing irrelevant information from input, ensuring only features necessary for transcription are preserved. In the second stage, another decoder is trained that has access to the textual representation from previous stage and is trained to predict target labels for downstream task while minimizing information encoding. Authors evaluated their framework on emotion recognition and speaker identification tasks, demonstrating that the resulting representations were effectively disentangled - the textual representations could predict transcriptions but performed randomly when predicting acoustic features, while acoustic representations showed the opposite pattern. Authors also analyzed how different layers of pre-trained and fine-tuned Wav2Vec2 models contribute to emotion recognition. They found that in models fine-tuned for automatic speech recognition (ASR), the acoustic contribution to emotion recognition decreases in higher layers while the textual contribution increases. Additionally, they showed that their framework can serve as a feature attribution method to identify the most significant frame representations for a given task, distinguishing between textual and acoustic contributions.

Strengths
- The proposed approach is well motivated (after revisions) and clearly described.
- There are a number of interesting analyses based on probing experiments to attempt to identify what information is still available in different layers of the network and assessment of the information related to distinct tasks in different frames of the input audio.

Weaknesses
- The multi-stage training procedure is cumbersome, since ordering matters
- Results seem somewhat inconsistent and require lots of post-hoc explanations. Additional validations would help. 
- There is previous work using the Information Bottleneck for feature disentanglement

Authors were able to address many concerns in the rebuttal and reviewers appreciated these efforts. The main remaining weakness is that authors were not able to address all concerns in an updated revision. Given the wealth of results and comparisons, and the fact that authors want to make a claim about a very specific setting (with limited technical novelty due to VIB being a known technique), the AC does not recommend acceptance at this time, although reviews would not preclude it.

Decision: Reject