## 3 The Proposed Approach: Disentanglement of Pre-Trained Speech Embeddings
Figure 2: Voice conversion scheme: the input signal is encoded with a self-supervised learning (SSL) pre-trained model (i.e., WavLM); the conversion model generates a converted representation using this content representation along with a different speaker embedding; and, finally, the vocoder synthesizes the converted waveform. Let us denote the input waveform as \(x\). After applying a set of random signal-based transformations \(T_{s}\), we obtain a modified waveform with the same spoken content and prosody but different speaker characteristics as follows:  
\[\hat{x}=T_{s}(x). \tag{1}\]  
Specifically, \(T_{s}(\cdot)\) randomly alters the formant frequencies and pitch, as discussed in [65], as follows:  
* the pitch is altered by a random shift in range \([1/1.4-1.4]\) and by a random scale in the range \([1/1.5-1.5]\);
* similarly, the formants are shift by a random value in the range \([1/1.4-1.4]\). As in the original paper, we used the ParselMouth Python library ([[https://parselmouth.readthedocs.io/en/stable/](https://parselmouth.readthedocs.io/en/stable/)]([https://parselmouth.readthedocs.io/en/stable/](https://parselmouth.readthedocs.io/en/stable/)) (accessed on 1 March 2024)). Given the encoder part \(Enc(\cdot)\) of the pre-trained WavLM model, we denote the latent representations of the original and modified waveform as \(X_{e}=Enc(x)\) and \(\hat{X}_{e}=Enc(\hat{x})\), respectively. These two representations share the same content but exhibit two different speaker identities. Therefore, the model is fine-tuned using a contrastive loss that penalizes the dissimilarity between \(X_{e}\) and \(\hat{X}_{e}\) as follows:  
\[\mathcal{L}_{\text{cosine}}(X_{e},\hat{X}_{e})=\frac{X_{e}\hat{X}_{e}}{\|X_{e} \|\|\hat{X}_{e}\|}. \tag{2}\]  
The cosine loss above is then combined in the fine-tuning process with the traditional CTC loss, which is computed on both versions on the input waveform as follows:  
\[\mathcal{L}=\alpha\mathcal{L}_{\text{cosine}}(X_{e},\hat{X}_{e})+(1-\alpha)[ \mathcal{L}_{\text{CTC}}(y,\hat{y})+\mathcal{L}_{\text{CTC}}(y,\hat{y}_{e})], \tag{3}\]  
where \(y\) are the ground-truth tokens, while \(\hat{y}\) and \(\hat{y}_{e}\) are the predicted tokens obtained from the original and modified waveforms, respectively. Once the disentanglement model is trained, the resulting speech representation is used as an upstream model in the S3 PRL-VC toolkit [66]. The S3 PRL-VC toolkit was proposed to help compare the SSL model performance against the SUPERB-SG [67; 68] standard framework. Specifically, it allows us to evaluate the generative capabilities of pre-trained models, as well as the generalizability of the resulting conversion model. Figure 3: Scheme for the disentanglement mechanism: the original signal and a modified version (with pitch and formant shifts) are used as pairs for training the downstream task (e.g., Automatic Speech Recognition). Aside from the original Connectionist Temporal Classification (CTC) loss, an additional loss based on a cosine similarity on the signal pair induces an additional constraint in the resulting representations, which forces the two representations to be similar and hence progressively reduce the information associated with the speaker characteristics. The resulting anonymization task was mainly derived from the setup proposed in [37] for voice conversion. The speech embeddings were computed using the introduced disentanglement mechanism on the WavLM features in the present work. The anonymization model was then trained after the preliminary fine tuning of a pre-trained WavLM model to a recognition task (i.e., with the usual CTC loss), and it was augmented with an illustrated contrastive loss that penalizes the dissimilarities between the signal pairs due to speaker characteristics (see Figure 3).