JOOCI: A FRAMEWORK FOR LEARNING COMPREHENSIVE SPEECH REPRESENTATIONS

## Abstract

Abstract Information in speech can be divided into two categories: “what is being said” (content) and “how it is expressed” (other). Current state-of-the-art (SOTA) techniques model speech at fixed segments, usually 10-25 ms, using a single embedding. Given the orthogonal nature of other and content information, attempting to optimize both within a single embedding results in suboptimal solutions. This approach divides the model’s capacity, limiting its ability to build complex hierarchical features effectively. In this work, we present an end-to-end speech representation learning framework designed to jointly optimize the “other” and “content” information (JOOCI) in speech. By using separate learnable parameters, JOOCI addresses this optimization challenge by modeling other and content information independently. Our results show that JOOCI consistently outperforms other SOTA models of similar size (100 million parameters) and pre-training data used (960 hours) by a significant margin when evaluated on a range of speech downstream tasks in the SUPERB benchmark, as shown in Table 1 . Code and models are available at TBA .

## 1 Introduction

Self-supervised learning (SSL) has played a significant role in learning high-level representations of text (Brown et al., ), vision (Alexey, ), and audio (Baevski et al., ; Mohamed et al., ; Défossez et al., ) data. In this work, we focus on learning high-level representations from raw speech. These learned representations are used as input features for various downstream tasks that require effective modeling of the “content” (what is being said) and “other” information (closely related to how it is expressed) present in speech (Yang et al., ; Borsos et al., ; Wang et al., ). The SUPERB (Yang et al., ) benchmark is designed to evaluate the generalization capabilities of different methods by evaluating the learned representations on a range of downstream tasks such as automatic speech recognition (ASR), phoneme recognition (PR), speaker identification (SID), emotion recognition (ER), and voice conversion (VC). ASR and PR performance depends on effective modeling of content information, while SID and ER rely on effective modeling of other information. The VC task requires the model to not only learn both information but also disentangle them.

Figure: Figure 1: Our proposed JOOCI Method. A raw audio is passed through a shared encoder. The output is passed to the two separate encoders to model the other and content information. Other encoder can attend to the content encoder embeddings but the gradients do not flow from other encoder to content encoder. Better viewed in color.

Current state-of-the-art SSL methods, such as WavLM (Chen et al., ), on the SUPERB benchmark share two key similarities. First, these methods first downsample and embed raw speech into fixed segments, typically 25 ms, using a CNN encoder followed by a powerful transformer encoder (^1^11Transformer encoder and model are used interchangeably.) to learn contextual embeddings for each segment. Second, masked prediction loss (MPL) (Hsu et al., ; Chen et al., ) is used during pre-training.

However, two key limitations exist in current SOTA methods:

- •
Firstly, using a single embedding to learn other and content information results in sub-optimal representations for both, due to the orthogonal nature of content and other information, which makes it difficult for the optimization algorithm to jointly learn both within a single embedding (^2^22Perfect features for an ASR system should only encode content information i.e., what is being said and forget about everything other i.e., how it is expressed.).
- •
The second limitation is the MPL loss. (Yadav et al., ) showed that it maximizes content information learned during pre-training, which inadvertently minimizes the other information, particularly speaker-related information. This indicates that MPL alone is not enough for learning other information. Additionally, (Feng et al., ) demonstrated that models pre-trained with MPL encode speaker (other) information in the silent parts of audio, further corroborating this issue.

For example, WavLM uses data augmentation for learning other (speaker) information. The authors observed that the model divides its total modelling capacity/layers in two, with later layers learning content and earlier layers learning other information. This ultimately prevents the model from fully leveraging all layers to build the complex, hierarchical representations characteristic of deep learning.

Furthermore, when a method maximizes content information to the extreme to enhance the performance of ASR and PR tasks, such as MS-HuBERT (Yadav et al., ), the strategy of dividing layers to encode content and other information, as used in WavLM, becomes ineffective. Therefore, there is a need for a better framework capable of jointly learning other and content information in speech by utilizing all layers to build complex, hierarchical representations. In this paper, we present a framework to jointly optimize the “other” and “content” information (JOOCI). JOOCI models other and content information using separate learnable parameters and distinct loss functions, differentiating it from the previous SOTA method, WavLM (Chen et al., ), for learning comprehensive speech representations.

Our contributions can be summarized as follows:

- 1.
We present JOOCI, a framework designed to jointly learn other and content information using separate learnable parameters and distinct loss functions. We achieve SOTA performance on the SUPERB benchmark, as shown in Table 1.
- 2.
Adapters are one way to adapt the behavior of the pre-trained model (Chen et al., ; ) for new tasks. In Section 3.2, we discuss in detail how JOOCI differentiates itself from adapters and why it is a more efficient choice.

## 2 JOOCI

In this section, we explain the proposed JOOCI framework in detail, covering its various components and the training criteria used. The complete JOOCI framework is illustrated in Figure 1.

### 2.1 Components

Shared Encoder (E^S) : The shared encoder serves a dual purpose i.e., it downsamples the raw speech and provides an embedding layer for both the content and other encoders. Inspired by previous work (Baevski et al., ; Hsu et al., ), we use seven blocks of temporal convolution layers which results in a down-sampling factor of 320x (^3^33each embedding represents 20ms of audio). For more details of shared encoder please refer to the original work (Hsu et al., ; Yadav et al., ) that we have followed.

Content Encoder (E^C) : The content encoder models “what is being said”, i.e., the content present in speech. Following prior studies (Hsu et al., ; Yadav et al., ), the output from the shared encoder is randomly masked and passed through a series of $n$ self-attention based transformer encoder layers to model the content information. For more details related to the content encoder please refer to the original work (Yadav et al., ) that we follow.

Other Encoder (E^O) : The other encoder models non-linguistic information present in speech. First, the shared encoder output is further downsampled using 1D average pooling layer with a kernel size and stride set to DF, where DF is the downsampling factor. The updated downsampled output is then passed through $n$ other encoder blocks, each of which consists of the following transformations:

- •
1D Res2Net block (Gao et al., ) with a kernel size, stride and dilation all equal to 1. To increase the overall non-linearity of the module.
- •
A split and append layer.
- •
A depthwise 1D CNN layer with a kernel size, stride and dilation equal to DF+1, DF+1, and 1 respectively.
- •
Another 1D Res2Net block with a kernel size, stride and dilation equal to 3, 1, and 4 respectively.
- •
A residual connection, followed by a 1D batch normalization (BN) layer

The split and append layer enables information from the content encoder to flow into the other encoder during the forward pass. No gradients flow from other encoder to content encoder during the backward pass. Therefore the other encoder can extract useful information, if any, from the powerful transformer content encoder. Specifically, content encoder embeddings are split into $m$ groups of size DF, and one embedding from the other encoder is appended at the end of each group sequentially. Finally, all groups are concatenated, increasing the total length, as shown in Figure 1. To reduce the input back to its original length, a depthwise CNN layer is applied with a kernel size and stride equal to DF+1.

Post Network (PN) : The post network performs three main functions crucial for training the other encoder using the student-teacher framework. 512D embeddings are extracted from the RDINO (Chen et al., ) model acting as a teacher, for each utterance.

- •
An attentive statistical pooling (ASP) layer, similar to (Okabe et al., ), which aggregates variable-length inputs.
- •
A 1D BN layer.
- •
A fully connected (FC) layer with an output dimension of 512. Used only during the pre-training stage.

Content Decoder : Similar to (Ao et al., ), we train a single layer decoder using pseudo targets as ground truth. A gradient reversal layer (GRL) is applied before passing the gradients to the other encoder during back-propagation. The role of content decoder is to discourage the other encoder from learning features necessary for solving tasks that require content information. The content decoder and the GRL is used during the pre-training stage only.

### 2.2 Training

During pre-training, an audio utterance $\bm{X}=x_{1},x_{2},\ldots,x_{t}$ is transformed using the shared encoder to produce embeddings ($\bm{E}$). These embeddings serve as inputs to both the content and other encoders, which transforms them even further separately. (i) Before passing the embeddings to the content encoder, approximately 50% of the embeddings are masked and replaced with the masked embedding $[M]$. (ii) And before passing the embeddings to the other encoder, they are transformed using a 1D average pooling layer with kernel size and stride equal to the DF, reducing the embeddings again by a factor of DF (^4^44Each embedding now represents DF times 20ms of audio for e.g., if DF is set to 10 then the other encoder is processing 200ms of audio.).

Content Loss (CL): The Multicluster Masked Prediction Loss (MMPL), inspired by (Yadav et al., ), calculates the masked prediction loss (MPL) across multiple layers of the content encoder using six sets of different pseudo labels. These layers are selected at regular intervals between the last and an intermediate layer. MMPL is the sum of MPL over a set (^5^55The set is of size 6, similar to the MS-HuBERT (Yadav et al., ).) = ( layer number, number of pseudo labels ) as shown in Equation 1.

$$ $L_{CL}=\sum_{d}(MPL)$ (1) $$

where $d$ is a dictionary indicating which pseudo label set corresponds to which transformer layer. And MPL is computed only at the masked indices, as shown in Equation 2.

$$ $L_{MPL}=\frac{1}{\mathcal{M}}\sum_{i=1}^{\mathcal{M}}\frac{exp\ (\text{sim}(Ah _{i},\mathbf{e}_{c})/\tau)}{\sum_{c^{\prime}=0}^{C-1}exp\ (\text{sim}(Ah_{i}, \mathbf{e}_{c^{\prime}})/\tau)}$ (2) $$

Here, $\mathcal{M}$ is the masked indices, $A$ is the projection matrix, $h_{i}$ is the content encoder embedding, $e_{c}$ and $e_{c^{\prime}}$ represent the correct and incorrect embeddings for the pseudo labels, $sim(\textperiodcentered,\textperiodcentered)$ computes cosine similarity between two vectors, and $\tau$ is a scaling factor for the logits. For further details, refer to (Yadav et al., ; Hsu et al., ).

Other Loss (OL) : The other encoder is trained using a student-teacher framework, where RDINO (Chen et al., ), a self-supervised model trained to learn speaker-discriminative embeddings, serves as the teacher. RDINO has 22.74 million parameters, significantly more than the 3.52 million parameters in the other encoder. We maximize the cosine similarity between the output of the PN (Student) module and RDINO (Teacher) embeddings for a given utterance. Similar to the BYOL approach (Grill et al., ) we only use the positive pairs for loss calculation. The full objective is shown in Equation 3.

$$ $\mathcal{L}_{\text{OL}}=1-\text{sim}({Student^{PN},Teacher^{RDINO}})$ (3) $$

Regularization Loss (RL) : Following (Ao et al., ), we train a transformer decoder to predict pseudo labels, using cross-entropy loss as defined in Equation 4. The GRL scale the gradients during backpropagation by a factor of 1/10, preventing interference with the other loss.

$$ $\mathcal{L}_{\text{(RL)}}=-\sum_{t=1}^{T}\sum_{i=1}^{V}y_{t,i}\log\hat{y}_{t,i}$ (4) $$

Where $T$ is the total number of time steps , $V$ is the size of the vocabulary, $y_{t,i}$ and $\hat{y}_{t,i}$ are the true one-hot encoded pseudo labels and predicted probability for the $t^{\text{th}}$ time step and $i^{\text{th}}$ word in the vocabulary respectively.

Total Loss : The overall training objective is the sum of all previous losses, as defined in Equation 5. Since JOOCI uses separate learnable parameters, the losses are summed directly without requiring additional hyperparameter tuning.

$$ $L_{Total}=L_{CL}+L_{OL}+L_{RL}$ (5) $$

## 3 Experiments

### 3.1 Experimental setup

Pre-training Dataset : We use the LibriSpeech 960-hour dataset (Panayotov et al., ), excluding transcriptions, for pre-training JOOCI consistent with other studies focusing on speech representation learning (Schneider et al., ; Chung et al., ; Hsu et al., ).

Data Augmentation: Given that (i) LibriSpeech is a read speech dataset recorded in clean environment free from real world noises and (ii) the amount of dataset is under 1000 hours. We augment the data very lightly, so not to interfere with the content encoder a lot, which would result in the model dividing its capacity simliar to WavLM (Chen et al., ). We augment only 12.5% of the audio samples using the MUSAN corpus (Snyder et al., ) with a high signal-to-noise ratio (SNR) in the ranges of [5, 15] for noise, [13, 20] for speech, and [5, 15] for music. For comparison, WavLM augments 50% of the audio with a SNR value in between [-5, 5]. Lastly, we also add Room Impulse Response (RIR) for reverberation to the selected audios similar to RDINO (Chen et al., ).

Pre-training : We use 12 layers for the other and content encoder. For the decoder, we use one transformer decoder layer with 8 heads and 768 dimension. It is trained with a vocabulary size of 1005. Other encoder is trained using the student-teacher framework. Content encoder is trained exactly to MS-HuBERT (Yadav et al., ). A detailed description is provided in the Appendix.

Shared Encoder and Content Encoder Initialization : The shared encoder and content encoder of JOOCI are initialized with the pre-trained MS-HuBERT model weights (Yadav et al., ). This choice is based on MS-HuBERT’s superior performance in modeling content information using the Multicluster Masked Prediction Loss (MMPL), which has been shown to achieve SOTA performance on content-based tasks.

SUPERB : To comprehensively analyze both the content and other types of information learned by JOOCI, we evaluated several tasks from the SUPERB (Yang et al., ) benchmark and compared our method with other SOTA methods of similar model size. The SUPERB benchmark is designed to compare models on their ability to learn comprehensive audio characteristics across a variety of downstream tasks including speaker recognition, content analysis, semantics, para-linguistics, and generation. For more details, see Yang et al. (). In short, SUPERB freezes the encoder and learns a weighted sum of all the layers to produce features for different downstream tasks.

Fine-tuning : No fine-tuning was performed. After pre-training, we discarded the content decoder and the fully connected (FC) layer from the post-network (PN) module and froze the parameters of JOOCI for all evaluation purposes in this work.

**Table 1: UNIVERSAL SPEECH REPRESENTATION EVALUATION ON SUPERB BENCHMARK. The performance on the content and semantic tasks can be further increased as shown in Table 5. To provide readers with a broader understanding of how performance scales with data size, we also include results for WavLM+, which has been trained on 100 times more data. However, WavLM+ should not be directly compared to JOOCI, as the pre-training data disparity makes such comparisons inappropriate.**
| Method | #Params | Corpus | Speaker | Content | Semantics | ParaL |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SID | ASV | PR | ASR | IC | SF | ER |  |  |  |  |
| Acc $\uparrow$ | EER $\downarrow$ | PER $\downarrow$ | WER $\downarrow$ | Acc $\uparrow$ | F1 $\uparrow$ | CER $\downarrow$ | Acc $\uparrow$ |  |  |  |
| FBANK | 0 | - | 8.5e-4 | 82.01 | 9.56 | 23.18 | 9.10 | 69.64 | 52.94 | 35.39 |
| modified CPC | 1.84M | LL 60k hr | 39.63 | 12.86 | 42.54 | 20.18 | 64.09 | 71.19 | 49.91 | 60.96 |
| HuBERT | 94.68M | LS 960 hr | 81.42 | 5.11 | 5.41 | 6.42 | 98.34 | 88.53 | 25.20 | 64.92 |
| WavLM | 94.70M | LS 960 hr | 84.51 | 4.69 | 4.84 | 6.21 | 98.63 | 89.38 | 22.86 | 65.94 |
| JOOCI (Ours) | 109M | LS 960 hr | 90.79 | 4.15 | 4.25 | 5.35 | 98.42 | 88.77 | 23.74 | 65.24 |
| WavLM + | 94.70M | Mix 94k hr | 89.42 | 4.07 | 3.92 | 5.59 | 99.0 | 90.58 | 21.20 | 68.65 |

### 3.2 Main Results

Evaluating JOOCI on the SUPERB Benchmark : Table 1 summarizes the performance of JOOCI on a range of downstream tasks from SUPERB benchmark. The results clearly indicate that JOOCI outperforms the current state-of-the-art (SOTA) models on the majority of tasks, except few where the margin is very less (^6^66WavLM used very big batch sizes and this could be the reason for the little margin gains compared to HuBERT and JOOCI.). In particular, JOOCI maximizes the performance on speaker-related tasks such as speaker identification (SID) and speaker verification (ASV), and exhibits the lowest error rates in content-related tasks like phoneme recognition (PR) and automatic speech recognition (ASR). Overall, JOOCI’s performance on the SUPERB benchmark reinforces our initial claim of it being a comprehensive framework for learning speech representations.

Comparison with Adapters : Instead of fine-tuning pre-trained models for every new downstream task, adapters (Chen et al., ; ) offer a lightweight solution to adapt these models for new tasks. Adapters introduce a small number of additional parameters tailored for each specific task. Table 2 compares JOOCI with HuBERT fine-tuned using various adapters. A key limitation of adapters, however, is that for every new task, it requires a new forward pass through the pre-trained model. JOOCI stands out by overcoming this limitation, as both the other and content embeddings are extracted in parallel in a single forward pass. With (i) comparable performance to adapters and (ii) the advantage of requiring only one forward pass, JOOCI stands as a more efficient framework for learning comprehensive speech representations. Lastly, Adapters can be applied to JOOCI also.

| Method | Params | SID $\uparrow$ | ASV $\downarrow$ | ASR $\downarrow$ | PR $\downarrow$ |
| --- | --- | --- | --- | --- | --- |
| HuBERT |  |  |  |  |  |
| FT | 94.68M | 64.56 | 5.15 | 6.35 | 2.45 |
| Weighted-sum | 13 | 81.42 | 5.11 | 6.42 | 5.41 |
| HuBERT + Adapter (Yang et al., ) |  |  |  |  |  |
| Houlsby | 0.60M | 87.71 | 5.29 | 5.88 | 3.00 |
| CHAPTER | 4.67M | 91.56 | 4.95 | 6.22 | 2.95 |
| JOOCI (Ours) |  |  |  |  |  |
| Weighted-sum | 13 | 90.79 | 4.15 | 5.35 | 4.25 |

Table: Table 2: Comparison of JOOCI with HuBERT using adapters. FT stands for finetuning. Here 13 is not a mistake and the weighted-sum only uses 13 parameters for each of the 13 layers.

Figure: Figure 2: Studying the effect of data augmentation on the content encoder using CCA word label similarity. Higher the CCA similarity for more number of layers better the method is. Better viewed in color.

Effect of Data Augmentation on the Content Encoder : We studied the extent to which data augmentation alters the learned representations in the content encoder. Following the approach of (Pasad et al., ; ), we plotted Canonical Correlation Analysis (CCA) versus word label similarity, as shown in Figure 2. Interestingly, the similarity in the later layers increased, suggesting that even a small amount of data augmentation aids in learning more robust features. We hypothesize that this is due to the small size of the pre-training dataset. Based on these insights and the findings from Section 4.3, we selected layers 6 to 11 (six layers) that exhibited very high CCA word label similarity and re-ran the experiments on ASR, PR, SF, and IC tasks. This selection resulted in a performance boost on the content tasks, either outperforming or matching WavLM on semantic tasks, as shown in Table 5. Therefore, taking a weighted sum of layers that encode similar information is a more robust choice than using all layers.

## 4 Ablation Study

**Table 3: Ablation results. Data augmentation helps on the ER task as a lot. Surprisingly, for the ER task content encoder embeddings with data augmentation performs better than the other encoder embeddings. JOOCI-C means using the content encoder embeddings, JOOCI-O means using other encoder embeddings and JOOCI-O-DGRL means using other embeddings trained with the decoder and GRL layer.**
| Method | Corpus | SID | PR | ASR | ER |
| --- | --- | --- | --- | --- | --- |
| Acc $\uparrow$ | PER $\downarrow$ | WER $\downarrow$ | Acc $\uparrow$ |  |  |
| Teacher model (22.74 million) |  |  |  |  |  |
| RDINO | VC2 2,442hr | 96.68 | - | - | 53.21 |
| Content Encoder is Frozen + NO Data Augmentation |  |  |  |  |  |
| JOOCI-C | LS 960hr | 75.40 | 4.17 | 5.32 | 62.05 |
| JOOCI-O | LS 960hr | 88.94 | - | - | 61.86 |
| JOOCI-O-DGRL | LS 960hr | 88.50 | - | - | 62.72 |
| Content Encoder is Fine-tuned + Data Augmentation |  |  |  |  |  |
| JOOCI-C | LS 960hr | - | 4.25 | 5.35 | 65.27 |
| JOOCI-O-DGRL | LS 960hr | 90.79 | 99.02 | - | 64.38 |

**Table 4: Ablation results. Performance of the SID task for different configurations. Better viewed in color.**
| Layers | Acc $\uparrow$ | Importance of other encoder layers |
| --- | --- | --- |
| Weighted Sum |  |  |
| 13 | 86.34 |  |
| 6 | 86.30 |  |
| 3 | 83.24 |  |
| Single Layer |  |  |
| LastLayer | 78.00 | Not Applicable |
| + ASP | 89.49 |  |
| + BN | 90.79 |  |
| + FC | 88.80 |  |

### 4.1 Frozen vs Fine-tuned Content Encoder with Data Augmentation.

JOOCI consists of three major components: (i) content encoder (C), (ii) other encoder (O), and (iii) content decoder with GRL (DGRL). We explore different combinations of these components across four downstream tasks, which tests effective modeling of both content and other information. Table 3 presents our ablation results. First, we pre-train JOOCI by training only the other encoder (O) and keeping the shared and content encoder frozen, either with or without the DGRL module. Incorporating the DGRL module significantly boosts ER performance, although it slightly degrades SID performance. Overall, the inclusion of DGRL increases robustness, and thus we employ the DGRL in all subsequent JOOCI experiments.

Next, we study the effect of data augmentation during pre-training, when all parameters of JOOCI are trained. Data augmentation results in substantial performance gains on the SID and ER tasks. These improvements are likely due to the characteristics of the LibriSpeech pre-training dataset, consisting of read speech with minimal background noise. For the PR and ASR task, data augmentation leads to very minimal degradation, which is negligible when considering the significant gains on other tasks. Our hypothesis is that with larger and more diverse datasets, the dependence on data augmentation will diminish.

Figure: Figure 3: Ablation results. ASP layers uses attention weights, between 0 and 1, to create one embedding for a variable sequence length. Sorting the weights, we see that majority of weights were given to few indices. Each histogram plots the attention weight values for different percentage of indices used. As we can see 50% of the total attention weight is given to only the top 20% indices. Better viewed in color.
Refer to caption: extracted/5930475/images/histogram-sid-attention.png

**Table 5: Ablation results. Performance of JOOCI when only using layers with high CCA similarity score.**
| Method | PR | ASR | SF | IC |  |
| --- | --- | --- | --- | --- | --- |
| PER | WER | F1 | CER | Acc |  |
| WavLM | 4.84 | 6.21 | 89.38 | 22.86 | 98.63 |
| JOOCI | 4.25 | 5.35 | 88.77 | 23.74 | 98.42 |
| JOOCI (6 - 11) | 4.19 | 5.20 | 89.61 | 23.32 | 98.61 |

### 4.2 Information Encoded at Different Layers in the Other Encoder for the SID Task

The expectation is that JOOCI’s other encoder should build complex hierarchical features, with later layers learning complex higher-level representations. Table 4 shows the accuracy for the SID task across different configurations. Surprisingly, when using a weighted sum of the other encoder’s all the layers, the initial layers were assigned higher weights. To investigate this further, we first used only the last six layers, which did not result in a performance drop, yet the same trend persisted. Next, we tried using only the last three layers, which led to a  3% performance decline, but again showed the same pattern. Finally, using only the last layer results in significant performance drop.

However, applying an ASP layer to the output of the last layer boosts the performance by about 3% compared to using all the layers.
We hypothesize that using ASP during pre-training could be the cause for the degradation. Because when studying the attention weights of the ASP layer, we found that the top  20% of the indices of a sequence constituted the most attention weight values as shown in Figure 3.
Lastly, adding a Batch Normalization (BN) layer further boosts performance by around  1%. On the other hand, using the last fully connected (FC) layer resulted in a performance drop of approximately  2%. This performance decline when using the last layer has been observed in vision model trained using SSL, where it is often recommended to drop the last layer after pre-training (Chen et al., ).

Given that the last layer yields the best performance, this suggests that the model can effectively construct complex high-level features by utilizing all layers. The failure of the weighted-sum approach to capture this may be attributed to the use of ASP layer in the post-network for pre-training. We leave this exploration for future work.

### 4.3 Disentanglement property of JOOCI

Disentanglement is the ability of the model to represent multiple factors of variation in data as separate and independent latent variables or embeddings. These embeddings encode different aspects of the data without influencing each other.

Solving the voice conversion from any to any (VC-a2a) speaker requires effective disentanglement of the current speaker from the content information in speech. Our findings indicate that either using one layer or the layers with high CCA similarity results in improved performance compared to using all the layers, as demonstrated in Table 6. This shows that JOOCI is able to disentangle content from the other information in the layers with high CCA scores. Furthermore, Table 3 shows that the other encoder also is not encoding content information as the performance on the PR task is  100%. Both these observations prove that JOOCI is able to disentangle content and other information successfully.

**Table 6: Ablation results. Using layers with high CCA scores (layer 6 to 11) is able to disentangle content information from the other information. We also use JOOCI other ASP output as speaker embeddings and find lower MCD, which shows more natural audio but lower speaker metric. In contrast, d-vector is a supervised trained system and (Wan et al., ) has 1.48 million parameters. Using JOOCI’s other encoder as speaker embedding results in better MCD. But lower ASV could be because of small pre-training data used.**
| Method | Speaker-Embedding | VC-a2a |  |  |
| --- | --- | --- | --- | --- |
| MCD $\downarrow$ | WER $\downarrow$ | ASV $\uparrow$ |  |  |
| All the layers |  |  |  |  |
| HuBERT | d-vector | 9.19 | 3.4 | 23.25 |
| JOOCI | 9.34 | 3 | 22 |  |
| one layer |  |  |  |  |
| HuBERT | d-vector | 8.49 | 3.3 | 66 |
| JOOCI | 8.31 | 2.9 | 86.5 |  |
| Using later layers |  |  |  |  |
| HuBERT (8 - 9) | d-vector | 8.52 | 3.5 | 65.5 |
| JOOCI (6 - 11) | 8.40 | 3.1 | 91.5 |  |
| JOOCI (6 - 11) | JOOCI | 8.12 | 3.1 | 83.00 |

**Table 7: Limitations. Performance on the SD task. We can not compare it with WavLM because other encoder’s resolution is 10 times more. We have reduced the content encoder’s resolution by a factor of 10 using mean pooling.**
| Encoder | SD (DER $\downarrow$) |
| --- | --- |
| JOOCI-C | 5.27 |
| JOOCI-O | 5.96 |

## 5 Conclusion

This paper presents JOOCI, a novel framework for learning comprehensive speech representations by jointly optimizing other and content information. JOOCI addresses the limitations of existing methods that struggle to encode both types of information effectively. By employing separate encoders and distinct training criteria for other and content information, JOOCI achieves state-of-the-art performance on the SUPERB benchmark, outperforming existing models across a range of downstream speech tasks, including speaker identification, speaker verification, phoneme recognition, and automatic speech recognition. The efficiency of JOOCI is also emphasized, as it extracts both other and content embeddings in a single forward pass, in contrast to adapter-based methods that require multiple forward passes. The ablation study highlights the significance of the DGRL module in enhancing the robustness and performance of JOOCI, particularly for the emotion recognition task. Additionally, data augmentation during pre-training proves beneficial for speaker identification and emotion recognition tasks, likely due to the size of the LibriSpeech dataset.

## 6 Limitations and Future work

Speaker Diarization (SD) Task : The SD task involves determining who spoke when in an audio recording. It requires the system to segment the audio and assign speaker labels to each segment, effectively separating multiple speakers within a conversation or audio stream. Unexpectedly, despite the other encoder’s strong performance on tasks like SID and ASV, it falters in the SD task compared to the content encoder. Data augmentation proves to be highly effective for learning fine-grained, discriminative features. However, using the ASP layer, during pre-training, for the entire utterance might be the reason for this degradation. The results are shown in Table 7.

Better Pre-training Methodology : (i) Employing a different pre-training methodology to learn fine-grained, discriminative features for the other encoder, such as using neural codec latent variables as training targets. (ii) To improve content modelling, utilizing the content decoder to also train the content encoder.

Multi-Talker ASR : JOOCI is well-suited for multi-talker ASR without requiring any external modules. It can be easily adapted by (i) using other embeddings to classify similar speakers, and (ii) applying the resulting classification as a mask to segment content embeddings, which can then be used for the ASR task.

#### Acknowledgments

Hemant Yadav is supported by Microsoft Research India PhD Fellowship program. Rajiv Ratn Shah is partly supported by the Infosys Center for AI, the Center of Design and New Media, and the Center of Excellence in Healthcare at IIIT Delhi.

## Appendix A Appendix

### A.1 Pre-training

Parameter count comparison : Table 8 shows the parameter count of different components of JOOCI during pre-training and inference.

**Table 8: Parameter count for different components of the JOOCI framework compared to HuBERT millions.**
|  | Training | Inference |
| --- | --- | --- |
| HuBERT | 94.70M | 94.70M |
| Shared encoder & Content encoder | 96.18M | 94.70M |
| Other encoder & PN | 3.52M | 3.12M |
| Content decoder | 9.3M | - |
| JOOCI (Ours) | 109M | 97.82 M |

### A.2 SUPERB batch size comparison

Table 9 shows the comparison of batch sizes used by WavLM and JOOCI. And what is used in the SUPERB benchmark.

**Table 9: Batchsize used by WavLM and JOOCI and the SUPERB benchmark default.**
| Task | WavLM | JOOCI | SUPERB |
| --- | --- | --- | --- |
| SID | 512 | 32 | 32 |
| ASR | 128 | 32 | 32 |
| IC | 128 | 32 | 32 |
| SF | 128 | 32 | 32 |
| ER | 32 | 32 | 32 |

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 1
questions: See Weaknesses section.
soundness: 2
strengths: The proposed method is sound. The experimental results on a subset of SUPERB benchmark are strong.
summary: This paper proposes a method for speech representation learning, particularly, for disentangle content information from non-content information. The paper reported strong experimental results on SUPERB benchmark.
weaknesses: - The novelty is limited. The proposed method is very close to a number of existing works, e.g.:
  - Chan et al., Content-Context Factorized Representations for Automated Speech Recognition, InterSpeech 2022.
  - Zhao et al., CCSRD: Content-Centric Speech Representation Disentanglement Learning for End-to-End Speech Translation, EMNLP 2023.
- The main claim is flawed. The paper claims SOTA on SUPERB. However, it only reports experimental results on a subset of the tasks from SUPERB (7 out of 10).
- The writing needs improvements:
  - Importantly, the name "other encoder" is a poor choice, which causes a lot of confusion for reading. Some simple choices such as "non-content encoder" would do a much better job.
   - Secondly, many small claims are questionable throughout the paper. A few examples:
     - Abstract: content and non-content information are orthogonal -- in the words from the paper, “how it is expressed” depends on “what is being said”
     - Sec 2.2: "Since JOOCI uses separate learnable parameters, the losses are summed directly without requiring
additional hyperparameter tuning." -- The previous paragraph said the opposite: " The GRL scale the gradients during backpropagation by a factor of 1/10, preventing interference with the other loss."
  - Lacks details of the model. While references to prior works is great, for completeness of the paper, you should describe the details of you model clearly, so that the readers understand your approach without having to jumping to many other papers.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: - The paper states that WavLM the "previous SOTA method".  By what measure is WavLM SOTA?  On what task(s)?

- I don't follow the sentences "As a result, the model cannot fully leverage all layers ... within a single embedding." nor the following sentence "The strategy of dividing the layers..."  Can you clarify what is meant there?

- The description of the split and append layer is a bit hard to follow.

- In Eq. 1, the index d is never used in the summand.  Also, should "MPL" be "L_MPL"?

- In Eq. 3, what exactly are Student^PN and Teacher^RDINO?

- In Table 1, where are the results for FBANK and other competitor methods obtained from?  Citations should be provided.  I also suggest including MS-HuBERT since JOOCI is based on it, and ideally also data2vec which has good results on many SUPERB tasks (but please let me know if you think these would not be relevant for some reason).

- I don't quite follow the sentence "We augment the data very lightly, so not to interfere with the content encoder a lot and divide its capacity."

- The description of the main results in Section 3.2 seems a bit misleading.  The paper states that the "results clearly indicate that JOOCI outperforms the current state-of-the-art (SOTA) models on the majority of tasks, except few ...".  However, in Table 1 JOOCI appears to outperform other models on exactly half the tasks, and it is never explained in what sense those models are SOTA (though they are clearly commonly used models).

- I do not understand the purpose of the comparison in Table 2, since JOOCI is not an alternative to adapters.  Also, "Houlsby" and "CHAPTER" need to be defined.  

- How does Figure 2 show the effect of data augmentation?  Is there a pair of curves that differs only in the use of data augmentation?

- For Figure 2, more information is needed about the y-axis.  How is CCA similarity defined?  How are the word labels encoded and how many words are there?  There has been prior work using CCA similarity for layer-wise analyses, e.g. Pasad et al., "Comparative layer-wise analysis of self-supervised speech models," ICASSP 2023.  Figure 2 seems similar to some of this prior work, and so it would also be helpful to state how your CCA-based analysis is the same or different, and whether your HuBERT results are similar to Pasad et al.'s.

- The ablation study in Sec. 4.1 is a bit confusing to me.  It claims to separately show the effect of DGRL and data augmentation, but as far as I can tell these two variables are changed simultaneously in the experiments.

- In Table 3, why are the "-" results not included?  If those could be included, they could help to show to what extent JOOCI-C and JOOCI-O specialize for linguistic vs. non-linguistic information.

- In Section 4.2, I have trouble following the first paragraph.  What kind of information is considered "higher-level" in the "other" branch, and what is the "same trend" that is referred to here?

- Section 4.3 claims to "prove that JOOCI is able to disentangle content and other information", but I don't follow how the results show this.  (Also, the word "prove" is too strong here, as in most descriptions of empirical findings.)

- In Table 5, what is the difference between the experiments in the last two lines (both labeled "JOOCI (6-11)"?
soundness: 2
strengths: + Addresses an important need to account for both linguistic and non-linguistic content in speech representation learning.

+ Obtains impressive results on several tasks, including speech recognition and speaker identification.
summary: The paper proposes a self-supervised speech representation model that combines two encoders, one intended to encode linguistic content and the other intended for "other" content like speaker and emotion information, trained with different losses.  The idea is that, by training a single encoder with a single loss, previous approaches have trouble encoding these two types of information equally well.  The various elements of the model are largely borrowed from previous work, but combined in a new way.  The model is compared in terms of performance on 8 common tasks (from the SUPERB benchmark) to other commonly used models (HuBERT, WavLM), finding improved performance on 4 of the tasks.  The paper also includes some ablation studies and analyses of several model components.
weaknesses: - Presentation of many details is unclear.  For example, the definition of "content" and "other" is never clearly stated.  Also, the model description is very brief, leaving many details to cited papers or the imagination (for example, is prosody ever/always/sometimes considered "content"?).  Either the writing should be much more precise or the paper should include equations specifying all of the model components.  See some other specific questions below.

- The key claimed contribution is that the model encodes both linguistic and non-linguistic information and that these are disentangled into the two encoders' representations.  However, the results don't quite show this, since the results on tasks are mixed and the analyses don't really demonstrate disentanglement (again see questions below).  Overall, I don't see the community starting to use this model as a replacement for other currently popular models.

- Some of the experiments do not, as far as I can tell, show the claimed findings (see details in "Questions" below).  

- The writing is in general hard to follow at times, in part due to many grammatical errors.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: * Can you offer insights into the relationship between SUPERB downstream task performance and model architecture designs?
* How do your results compare to recent speech representation work that has also been evaluated on SUPERB?
soundness: 2
strengths: * The research community is highly interested in the topic of speech representation learning.
* The proposed method's evaluation on certain SUPERB tasks yielded better results compared to the cited systems.
* The discussions and comparisons presented are technically sound.
summary: This submission introduces a framework for distinct representation learning of "content" and "other" properties in speech. The authors report improved performance on certain SUPERB tasks compared to other systems. Additionally, the submission includes comparisons with adapters, ablation studies on encoders, data augmentation, and learned representations.
weaknesses: Major issues:

* The model's effectiveness is unconvincing. The baselines cited are outdated and not state-of-the-art, and the model's performances on the semantic tasks are not better.
* The paper's discussion of different model architectures is shallow, limiting its contribution and making it difficult to draw general conclusions. 

Minor:

* Figure 1 could be simplified by removing the hyperparameters.
* The discussion of "Data augmentation" in Line 52 seems out of place, as the initial focus was on model architecture for speech representation learning.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1) How does the JOCCI compare to the contentvec, SPIN, and Data2vec models?

2) What is the impact of initialization on the model performance e.g. random vs Ms-HuBERT initialization 

3) Can the model be trained with just the GRL loss and without the RDINO teacher?
soundness: 3
strengths: 1) The paper is well-written and addresses an important challenge of disentangling content and other information in the speech representation.

2) The model performs well on the SUPERB benchmark and outperforms HuBERT and wavLM.
summary: The paper proposes to disentangle the content "what is being said" and other “how it is expressed” information present in the speech data. The paper proposes the JOCCI framework, which uses two submodules, focused on maximizing the content and the other information. The content module is trained with a self-supervised objective whereas, the other module is optimized with a teacher-student objective. A regularization loss is added to minimize the information overlap in the two submodules.
weaknesses: 1) The baseline comparisons are limited. There have been other attempts to remove other information such as speaker information from the self-supervised representations such as contentvec[1] and SPIN[2]. Even the MS-HuBERT model used for initializing JOCCI is missing from Table 1.

2) JOCCI relies on a pretrained method RDINO for training the other encoder whereas baseline methods such as HuBERT do not. 

[1] “Contentvec: An improved self-supervised speech representation by disentangling speakers

[2] Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clustering

Meta-review: The paper proposes an approach to disentangle linguistic information from non-linguistic ones by using an additional speaker model for supervision.

I recommend a rejection because of  the unsupported claims in the paper.

All reviewers raised concerns about unsupported claims. Whether the representations are properly disentangled is not well supported, not to mention disentanglement by itself is not a very well defined concept. Whether it is theoretically possible to disentangle linguistic and non-linguistic information is another inherent claim that is not supported.

The biggest argument is around the term "state of the art." Most reviewers are not happy with the claims involving the state of the art. I personally don't mind using the term to refer to the current "state." However, the paper is using the term to refer to the "best." It becomes tricky to claim the best because there isn't a single metric in the evaluation. (It's also generally not useful to talk about the best in this context anyways.) It's much more convincing to claim victory over a strong baseline, rather than being the best. Besides, since the paper uses a speaker model, it might not even be fair to claim victory over those that don't.

Overall, a lot of care is needed in the experiments and in the writing.

Decision: Reject