Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units
Authors: Gallil Maimon
Yossi Adi
Venue: EMNLP 2023 Findings (2023)
Keywords: Voice Conversion
Audio Processing
Self-Supervised Learning
TL;DR: We introduce a novel method for prosodic aware voice conversion in an entirely textless manner.

Abstract: We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of a recording to a target speaker in a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses a pretrained, self-supervised model for encoding speech to discrete units, which makes it simple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quantitative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at https://pages.cs.huji.ac.il/adiyoss-lab/dissc/.

Speaking Style Conversion in the Waveform Domain Using Discrete Self-Supervised Units
Gallil Maimon, Yossi Adi
School of Computer Science and Engineering The Hebrew University of Jerusalem gallil mta.iilmmimon@@mii.... hujii..huui.....
Abstract
Speaker
Speaker
We introduce DISSC, a novel, lightweight method that converts the rhythm, pitch contour and timbre of recording a textless manner. Unlike DISSC, most voice conversion (VC) methods focus primarily on timbre, and ignore people's unique speaking style (prosody). The proposed approach uses pretrained, self-supervised model for encoding speech to discrete units, which makes it sim- ple, effective, and fast to train. All conversion modules are only trained on reconstruction like tasks, thus suitable for any-to-many VC with no paired data. We introduce a suite of quan- titative and qualitative evaluation metrics for this setup, and empirically demonstrate that DISSC significantly outperforms the evaluated baselines. Code and samples are available at mttp::/pages.cs.huj huji.ac. ac. www.ii//sssssaabaaaeeo
target speaker in
Please
Stella
Please
Stella
0.3
15
0.5
13
Speaking Style Conversion
Figure I: Comparing traditional VC speaking style conversion. White lines on the spectrograms show the pitch contour. While VC methods change the spec- tral features the new utterance sounds like the speaker, they do not change the rhythm and pitch. Con- versely, SSC matches the faster speaking style, and the target speaker's tendency to finish sentences with pitch increase. This a real sample converted by DISSC. Due variance within speaking style do not expect the converted speech to exactly match the target.
1 Introduction
Imagine hearing famous catchphrase of your favourite television character spoken in their voice, but uncharacteristically fast, while atypically em- phasising the end. This would immediately raise suspicion that something is "wrong". As humans we learn to recognise familiar people and voices, not only by their voice texture (timbre), but also by their typical speaking style (Williams, 1965). Therefore, true VC method should convert both voice texture and speaking style (rhythm, FO, etc.). Figure describes this visually.
Another line of work uses text transcription supervision and shows the potential benefit VC (Kim et al., 2022: Liu al., 2021). How- ever, this limits the application to high resource languages and requires large scale data labelling.
Traditional VC methods mainly focused on changing the timbre of a given speaker while leav- ing the speaking style unchanged (Stylianou et al., 1998; Kain and Macon, 1998; Nakashika et al., 2013; Chou et al., 2019; Huang et al., 2021). Re- cent methods propose to additionally convert speak- ing style (Qian al., 2020; Chen and Duan, 2022: Qian et al., 2021; Kuhlmann al., 2022). How- ever, these mainly use only single target utterance which does not fully capture speaker prosody.
Some recent prosody aware VC methods use spectrogram representations as input and out- put (Qu et al., 2022; Chen et al., 2022), rather than operating in the waveform domain. Thus, they involve another phase of converting from the spec- tral domain to the time domain using a vocoder. Moreover, using discrete self-supervised speech representations and generating waveforms from these was demonstrated to provide superior per- formance on plenty downstream tasks such as

---

speech and audio language modelling (Lakhotia
HuBERT (Hsu et al., 2021) tokens, pitch repre- sentation based on YAAPT (Kasi and Zahorian, 2002) and a learned speaker representation to re- synthesise the audio waveform. They showed how replacing the speaker representation at infer- ence time performs VC. Inspired by this model- ing paradigm, our approach additionally introduces speaking style modeling and conversion (i.e., pitch contour and rhythm) thus performing SSC.
et al., 2021; Borsos et al., 2022; Qian al., 2022),
multi-stream processing (Kharitonov et al., 2022b),
speech emotion conversion (Kreuk et al., 2021), spoken dialogue (Nguyen et al., 2022), speech-to- speech translation (Lee ct al., 2022a,b; Popuri et al., 2022), and audio generation (Kreuk et al., 2022a.b). Inspired by this line of work, we propose sim- ple and effective method for Speaking Style Con- version (SSC) based on discrete sulf-super and partially disentangled speech representations. Cu- rating samples of different speakers pronouncing the same utterances is challenging, thus we follow the setup of unpaired data, similarly to Lee et al. (2021); Yuan et al. (2021); Kameoka et al. (2018); Kaneko and Kameoka (2017). We give exact de- tails of our approach in Section 3. We formalise the setup of SSC and introduce an extensive evaluation suite in Section 4. Results suggest the proposed approach is greatly superior to the evaluated base- lines (see Section 5), while utilising a simple and effective method which can be trained on a single GPU within couple of hours.
Speaking Style Conversion. Various existing methods aim to change the prosody of given utterance. However, many convert it based on a single target utterance (Qian et al., 2020; Chen and Duan, 2022) which does not truly capture the general speaking style of the speaker, and often creates artifacts when the contents do not match. Other methods only linearly alter the speaking rate (Kuhlmann et al., 2022) thus ignoring the change of rhythm for different content.
Even leading SSC methods are still limited. Some focus only on rhythm, paying less atten- tion to pitch changes (Qian et al., 2021; Lee et al., 2022c). Other approaches require text transcrip- tions for training (Lee al., 2022c; Liu al., 2021). In contrast, our approach uses HuBERT units trained in SSL fashion without using textual annotations, thus can support lower resource lan- guages (see Fig. 6 in Appendix A.5). In addition, two concurrent methods for SSC (Qu et al., 2022: Chen et al., 2022) were proposed recently. Al- though showing impressive results, they based spectrogram representations, hence require additional vocoding step.
Our contributions: (i) We introduce a novel, sim- ple, and effective method which uses discrete self- supervised speech representations to perform text- less speaking style conversion in the waveform domain; (ii) We formalise the task of speaking style conversion and propose diverse evaluation methods to analyze different speech characteris- ties; (iii) We empirically demonstrate the efficacy of the proposed approach using extensive empirical evaluation on several benchmarks considering both objective and subjective metrics.
Our Approach DISSC
2 Related Work
We operate under the following setup: we perform any-to-many SSC which means we take any record- ing and convert it to any of the training set speakers. Our training set contains only unpaired utterances e.g. no training utterance is said by more than one speaker. The learning process is completely textless with no form of text supervision.
Unpaired Voice Conversion. Many existing methods perform VC in the setting of unpaired utterances (Kameoka et al., 2018; Kaneko and Kameoka, 2017). Most of them contain vocoder which generates the audio (or an intermediate repre- sentation) from several representations, one meant to capture speaker information and others to cap- ture the remaining information (Lin et al., 2021b,a). They encourage disentanglement through infor- mation bottlenecks by neural network architec- ture (Chen et al., 2021; Qian et al., 2019), mutual information and adversarial losses (Lee et al., 2021; Yuan al., 2021), pretrained models (Huang et al., 2022; Polyak et al., 2021) or combinations of these. Notably, Polyak et al. (2021) used discrete
Our approach uses Discrete units for Speaking Style Conversion, and is denoted DISSC. We de- compose speech to few representations to syn- thesise speech in the target speaking style. We consider three components in the decomposi- tion: phonetic-content, prosodic features (i.e., FO and duration) and speaker identity, denoted by Zey (Zutur ZF.) ) Zypk respectively. We propose a cascaded pipeline: (i) extract z, from the raw wave-

---

form using SSL model; (ii) predict the prosodic features of the target speaker from Ze and sspii (ii) synthesise the waveform speech from the content, predicted prosody and target speaker identity. See Figure for a visual description of DISSC. This modelling approach means moving from a contin- uous space to a discrete space, hence resulting in easier optimisation and better quality generations.
Inout
Output
Eg
Zaplee
G
3.1 Speech Input Representation
Edur
Epo
Phonetic-content representation. To represent speech phonetic-contee resentation of the audio signal using a pre-trained SSL. model, namely HuBERT (Hsu et al., 2021). We use a SSL representation for phonetic-content, and not text or text based methods, to maintain non- textual cues like laughter and allow support for di- verse languages. We discretise this representation to use as rhythm proxy (by number of repetitions) and for ease of modelling. We chose HuBERT for the phonetic-content units Polyak al. (2021) showed it better disentangles between speech con- tent and both speaker and prosody compared to other SSL-based models.
extract discrete rep-
Zapk
Figure overview DISSC. We use an SSL pre- trained content encoder E, to extract discrete units from the waveform. We independently train a rhythm pre- dictor Edur and pitch predictor Er, reconstruct the original unit repetitions and pitch contour, respectively. input all representations into pretrained vocoder G.
ing the prosody prediction and waveform synthesis phases. To learn Zepk we optimise the parameters of a fixed size look-up-table. Although such mod- eling limits our ability to generate voices of new and unseen speakers, it produces higher quality generations (Polyak et al., 2021).
Denote the domain of audio samples by X' C IR. Audio waveforms are therefore represented by se- quence of samples x (x'.....x²)) where each xf E X for all 1 S t < T. The content encoder E. is HuBERT model pre-trained on the LibriSpeech corpus (Panayotov al., 2015). The input to the content encoder E. an audio waveform x, and the output is a latent representation with lower temporal resolution z' = (=')) 22² where I. = T/n. Since HuBERT outputs continuous representations,
Prosody representation. As explained in the content representation section, the number of repe- titions of units indicates the rhythm of the speaker in the context. More repetitions means a sound was produced for a longer time. Therefore when converting rhythm the sequence "deduped", and when only converting the pitch the full z, used. Following Polyak et al. (2021) we encode the pitch contour of sample using YAAPT (Kasi and Zahorian, 2002). We mark this pitch contour Zrg (2), ..... 24,,, where 2/2, Rzn- We mark the per-speaker normalised version as T
needs an additional k-means step in order to quantise these representations into discrete unit sequence. This sequence is denoted by ze (z...... , =L() where z E {l...., K} and K is the vocabulary size. For the rest of the paper, we refer to these discrete representations as "units". We extracted representations from the final layer of HuBERT and set K = 100. When we wish to predict the rhythm, we must decompose it from the sequence, therefore repeated units are omitted (e.g.. 0,0, 0, 2, >> 0, 2) and the number of repeti- tions correlates to the rhythm. Such sequences are denoted as "deduped""
3.2 Speaking Style Conversion
Using the above representations we propose to syn- thesise the speech signal in the target speaking style. We aim to predict the prosodic features (duration and FO) based the phonetic-contert units, while conditioned on the target speaker representation. We inflate the sequence according the predicted durations, and use it, the predicted pitch contour, and the speaker vector as conditioning for the waveform synthesis phase.
Speaker representation. Our goal is convert the speaking style while keeping the content fixed. To that end, we construct a speaker representation Zopk. and include as additional conditioning dur-

---

Fr 1pr(2r 20)30.3
During conversion we replace the speaker vec- tor with that of the target and wish for the model to convert the rhythm. To this end our predictor model must strongly depend on Zspk- In order to encourage this we introduce two minor adjustments. HuBERT units hold speaker information and are not fully disentangled, however, the shorter the sequence the less information exists (Kharitonov et al., 2022a). We therefore receptive field. In addition, we introduce train time masking of units in the sequence, this means that Edrr has access to even less units at train time, hence they contain less speaker information. Exact details can be found in Appendix A.3.
F.o
Pr Pr + 0)
Epo
CNN Pitch
CNN Voicing
Regressor
Classifier
Shared CNN Block
CNN with small
Concatenate
Embedding Layer
Positional Encoding
Eapk
Z Zapk
Ze
Pitch prediction. We train the pitch predictor to predict the per-speaker normalised pitch contour F0, from the HuBERT umits Ze. The pitch values are normalised to zero mean and standard devia- tion of one per-speaker the vocalised sections. We use these values labels predictor condi- tioned on content units zes positional encoding and a speaker vector sspk The positional encoding is added to Zepk and denotes the absolute position the unit compared to the start and finish of the se- quence, though preliminary results showed that any standard positional encoding works. Intuitively, positional encoding will help learning localised speaker pitch patterns such ending sentences in pitch drop.
Figure 3: An overview of the En, architecture. The positional encoding is linear positional embedding relative to the start and the end of the sequence, added to the repeated speaker vector. The model jointly predicts the presence of vocalised units (binary classification) together with continuous FO values for the voiced units.
Rhythm prediction. Each HuBERT unit corre- lates with the phoneme in the given 20 ms. This means that longer repetitions of the same unit indi- cate a longer voicing of the sound - i.c. rhythm. In order to learn to predict, and convert the rhythm we build a model which learns to predict unit durations, denoted as Edur- During training of Edurr we input the deduped phonetic-conten units Ze and speaker representation conditioning Zopk: with the original unit durations (before dedup) as supervision.
Our pitch predictor is denoted E An overview of it is described in Figure 3. It is a CNN which has two predicted outputs: binary speaking detection, and regression head to predict the exact pitch. The binary speaking classification is trained with BCE loss the binarised pitch labels i.e. 2F0 0. The regression head is trained with MSE loss on the non-zero, vocalised pitch labels. Like the rhythm predictor this model trained with a small unit receptive field and unit masking.
This simple reconstruction task formulated as a regression task with Mean Squared Error (MSE) loss. As the units do not perfectly correlate to phonemes, neighbouring units often represent the same phoneme even after "dedup". This makes it harder to predict each unit's length separately, Therefore, we encourage the model's nearby errors to cancel each other out by adding an MSE loss on the total error of 4 neighbouring units. This helped with errors all being on the negative end and creating shortening bias in our preliminary test. In addition, we round the regression outputs to be used integer repetitions of the unit, how- ever this can also cause a bias in which predictions like 1.51, 1.51, ... 1.51 and 2.49, 2.49. ... 2.49 would result in the same total length. To mitigate this we carryover the remainder after rounding and sum those differences.
3.3 Speech Synthesis
We follow Polyak et al. (2021); Kreuk et al. (2021) using a variation of the HiFi-GAN vocoder (Kong et al., 2020). Like previous work, we adapt the generator to take as input sequence of phonetic-content units, pitch contour and a speaker- embedding. During training, these are only features from real samples the vocoder trained indepen- dently for reconstruction. At inference time we use the unit sequence inflated with predicted durations, the predicted pitch contour and the target speaker-

---

embedding. The above features are concatenated along the temporal axis and fed into a sequence of convolutional layers that output a I-dimensional audio signal. We match the sample rates of unit sequence and FO by means of linear interpolation, while replicating the speaker embedding.
clear pitch patterns. This means that the speech is quite monotonous and that variation across record- ings was greater than across speakers. To ef- fectively evaluate pitch conversion we created a synthetic dataset based on VCTK - denoted as Syn_ WVC_C In this dataset we took 6 speakers from VCTK, encoded them based on the pipeline by Polyak et al. (2021). We then altered the pitch contour by introducing linear trend of either up, down or flat for each of the speakers. We then generated the new synthetic recordings in the same approach. We encourage readers to listen to these samples as well as the other datasets in the accom- panying page to grasp the different speaking styles.
To sum up, after each of the components de- scribed in previous sections are trained, the full pipeline of DISSC is composed of: first extract- ing content representation, Ze. then, predicting the prosodic features of the target speaker based on be and zspk: and lastly, synthesising the waveform using these new representations.
4 SSC Evaluation Suite
4.2 Metries
In this section we describe our setup which uses several datasets and fine-grained offline metries for each part of the conversion timbre, pitch, rhythm. As no standardised framework for evaluating SSC currently exists, we hope the proposed metrics, will help further advance research in this field. A de- tailed description of training configurations and hyper-parameters can be found in the Appendix.
4.2.1
Objective Metries
We introduce new metrics which aim to capture the rhythm in a fine-grained manner, and objec- tively evaluate the pitch contour (even when the rhythm does not match). We hope these metrics will contribute to other advances in this field.
Timbre. For regular VC we use the common Equal Error Rate (EER) of speaker verification us- ing SpeechBrain's (Ravanelli et al., 2021) ECAPA- TDNN (Desplanques et al., 2020) which achieves 0.8% EER on Voxceleb. For each sample, we ran- domly select ni positive and n negative samples. Positive samples are random utterances of other speakers converted the current speaker, and neg- ative samples are random utterances of the current speaker converted random target speaker.
4.1 Datasets
Most previous work focus on a single dataset such as VCTK (Veaux et al., 2017), since few large, multi-speaker datasets with parallel test sets exist. However, VCTK has fairly monotonous speech with less distinct speaking style.
To address this, we use several datasets. First, we use VCTK as it has many speakers. We leave all paired utterances (speakers say the same content) the test set, to mimic a truly unpaired setup. In preliminary results, having paired data in training improved the results notably, but harms the wanted setup. We follow Qian et al. (2021) in selecting the two fastest (P231, P239) and two slowest (P245, P270) speakers by mean log duration.
Content. For phonetic content preservation we use the standard Word Error Rate (WER) and Char- acter Error Rate (CER), using Whisper (Radford et al., 2022) our Automatic Speech Recognition (ASR) model.
In addition, we wished add more expressive dataset, yet found that most expressive datasets are from the emotional speech domain. In order to adapt them to our use case we take the Emotional Speech Dataset (ESD) (Zhou et al., 2022), and se- lect one emotion for each of the 10 speakers. This means that all recordings of given speaker are in a certain emotion thus making it part of their speak- ing style. Here, we also take the two fastest and two slowest speakers for rhythm metric evaluation.
Rhythm. In order to evaluate the rhythm we wish to go beyond the length of the recording, which only indicates the average rhythm and not which sounds are uttered slowly and which quickly. We therefore use the text transcriptions of the sam- ples, and the Montreal Forced Aligner (MFA) (McAuliffe et al., 2017) to get start and end times for each word and phoneme in the recording. We then compare the lengths of the converted speech and the target speech, thus defining Phoneme Length Error (PLE), Word Length Error (WLE) and Total Length Error (TLE). We do not compare
After preliminary experiments we found that the speakers in the above datasets do not always have

---

CONTENT
&
RHYTHM WLEJ
SPEAKER
MODEL
WER,
CER
EMDI
TLE
PLEμ
EER
AutoVC (Qian et al., 2019)
71.3
47.1
17.68
1.214
0.072
0.028
7.5
AutoPST (Qian al., 2021)
40.6
26.7
21.9
1.379
0.123
0.037
24.1
VCTT
Seq2seq- VC (Liuu ct al., 2021)
2.9
1.2
20.95
1.214
0.072
0.028
10.9
SR (Polyak et al., 2021)
6.6
3.3
1.214
0.071
0.025
1.8
DISSC Rhythm (Ours)
10.9
5,4
10.58
0.832
0,056
0.023
1.7
DISSC_Bole (Ours)
13.0
6.9
10.53
0.832
0.056
0.023
1.7
AutoVC (Qian et al., 2019)
87.0
59.9
31.82
0.591
0.106
0.048
6.6
ESD
AutoPST (Qian et al., 2021)
50.3
31.8
37.2
0.549
0.097
0.043
15.7
SR (Polyak et al., 2021)
14.9
6.0
30.3
0.591
0.097
0.041
2.9
DISSC_Rhythen (Ours)
19.1
7.9
24.8
0.350
0.076
0.037
2.6
Table I: A comparison of the proposed method against VC and SSC baselines considering content (WER, CER),, Rhythm and FO (EMD, TLE, WLE, PLE), and speaker identification (EER). Only DISSC improves rhythm metrics compared to standard VC baseline Speech Resynthesis.
the length of silences as their occurrence can differ between recordings thus breaking the alignment. Formally, for each input recording and matching text, MFA returns list of tuples To ((5m1 Cwi Gwy) ) (Smm Cwm: Cwn,) )) for word level and another for phoneme level Tp = ((8p)> Sp1 9m) ... (9pm Cym (pm)). ((mpp) indicates the start time of the sound at index i in the recording (word or phoneme according the level), and ((w,p) indicates the end time. C(u,p), represents the content of the time segment, the word phoneme. We filter only non-silence time segments leaving Twpp) = +
content is harmed dramatically in the conversion, MFA might fail to align the text. We penalise such samples by assuming linear and even split of the recording phonemes and words.
Pitch. Popular evaluation methods for FO similar- ity include the Voicing Decision Error (VDE) and FO frame error (FFE). Intuitively, VDE measures the portion of frames with voicing decision error, while FFE measures the percentage of frames that contain deviation of more than 20% in pitch value or have voicing decision error. Formally.
ΣT- 1.2, 0z} VDE(U, 0)
s.t. c) (w.p)
(2)
((w(p)) e (w.p).: (w.p), silence). Thus formally, the errors per sample compare durations of a reference sample and a synthesised sample:
T
FFE p, u, VDE(U, 0)
ET-11 1
[lp
pr > 0.2
pc]
[ [b]]
PLE(TTeU Tp Tref)=
T
Ten(T,))
((epy"
- ) - (e /-
$p551 re()];
where p, D are the pitch frames from the target and generated signals, v, 0 the respective voicing decisions, and is the indicator function.
#=1 WLE pyym fires)
(en(TT))
Although these metrics are widely used in speech synthesis (Wang et al., 2021), these meth- ods are not well adapted to misaligned samples, i.e when they are spoken at different rates. These metrics require the samples to have the same time length, therefore many implementations linearly interpolate the generated sample's pitch contour to that of the target before computing them. How- non-linear (i.e certain
He
201)
(c
The TLE is just the duration difference between the entire recordings. All rhythm metrics in time units, in this case seconds. In addition, on rare occasions the number of phonemes for the same sound by different speakers different (due to accents etc.), we ignore these samples for more well-defined metric. addition, when the phonetic
ever, rhythm differences phonemes could be "stretched" more than others) thus not alleviating the misalignment problem. In- stead, we suggest using segments obtained from the

---

forced-aligner and calculate the FFE while inter- polating the segments match the segment sizes, This measures the match in shape and value of the pitch contour for each word and phoneme, even they are misaligned. We denoted these as P_FFE for phoneme segments and W FFE for words.
Neturalness
3-
2-
Reference
AutoPST
DISSC Both
DISSC Rhythm
Rhythm Pitch. We compute Earth Movers Distance (EMD) (Rubner et al., 1998) between the pitch contour of converted and target samples. Es- sentially we check how much "pitch mass" needs to be moved between them. This metric considers errors along the time domain (rhythmm) and pitch contour errors (such predicting the wrong trend). Our results show strong correlation human per- ception of rhythm and pitch with the above metrics.
Similartly
3-
2-
AutoPST
DISSC DISSC
Figure 4: We compare the human evaluated naturalness of real reference utterances, Speech Resynthesis (SR), AutoPST, two versions of DISSC which convert rhythm only or also FO. also compare the similarity of how likely the converted sample to come from the the speaker as the target recording.
This newly proposed evaluation suite is generic and can transfer to other languages. However, cer- tain metrics are based language-specife pre- trained models (e.g., ASR-based metries). In such cases, a multi-lingual ASR can be used. In our setup, we use Whisper model (Radford al., 2022) which supports 100 languages. Another option would be to use the MMS model (Pratap et al., 2023) which supports more than 1000 lan- guages for both ASR and forced-alignment.
detailed description of the subjective tests can be found in the Appendix.
Results
4.2.2 Human evaluation
We compare DISSC to several any-to-many (or many-to-many) baselines. Notably, we evaluate Speech Resynthesis (SR), and AutoVC (Qian et al., 2019) which are well known VC methods. In ad- dition, compare AutoPST and BNE-MoL- seq2seqVC (Liu et al., 2021) which prominent prosodic-aware VC (i.e.. SSC) methods. We the official Github implementations and train the models on our datasets using the authors recom- mended configuration. For a fair comparison, all baselines were evaluated under the any-to-many sctup. i.c., generating only seen speakers. Only seq2seq VC supports new speakers, because only that approach was published. However, evaluate them known speakers, with the entire training as reference recordings. We show three variants of our method: DISSC DISSC_Rhythm, DISSC_Pitch and DISSC Both which predict duration, pitch contour and both respectively.
To confirm that the proposed objective metrics cor- relate human perception, and evaluate to what extent the prosody impacts humans' speaker recog- nition we conduct a human evaluation study. We use the common mean opinion score (MOS) to eval- uate the quality and naturalness of generated and real samples. Specifically, each sample was scored from 1-5 naturalness and quality. We took the same 40 samples for each method and the original. Each sample was annotated by least three raters,
Next, to evaluate the effectiveness of the speak- ing style conversion, we present the users sample from the target speaker and sample converted by each method. Users rated each sample from 1-5 ("Very Different"""Van Similar") how likely it is be uttered by the same speaker and instructed to "pay attention speaking style such rhythm and pitch changes, and not only the voice texture". We evaluated 10 samples per-method, with at least three raters per-sample. We report the mean score and 05% confidence intervals for both tests. We modified the WebMUSHRA! tool for the tests. A
Objective results for VCTK and ESD appear in Table On both datasets DISSC improves the length errors across all scales: phonemes, words and utterances. The relative improvement com- pared to the next best baseline is 56.8% on ESD TLE. At word level we get 27 97.6% and 26.7% rela-
http:::////gtthuuucommm/uuue com/audi wlwp:////aaaaaa//aaaaa/a/aaaa webMUSHRA

---

CONTENT WER,
RHYTHM&FO
RHYTHM
FO
MODEL
CER
EMD1
TLE,
WLE1
PLE.
W _FFE
FFE.
SR (Polyak al., 2021)
12.4
6.3
24.23
1.101
0.064
0.024
46.1
43.4
DISSC_ CesSC_ (Ours)
18.8
9.5
20.78
0.775
0.053
0.023
46.1
43.7
DISSC Pitch (Ours)
12.5
6.3
13.55
1.101
0.065
0.024
24.5
20.5
DISSC Both (Ours)
19.6
10.2
10.47
0.775
0.053
0.023
25.3
21.4
Table 2: VC and SSC results on the Syn_VCTK datasct. This ablation measures the impact of rhythm and pitch conversion over SR considering content (WER, CER), Rhythm and (EMD, TLE, WLE, PLE, W FFE, FFE).
tive improvement on ESD and VCTK respectively. All other baselines show rhythm errors comparable worse than SR which does not convert rhythm.
Abnornal
find
We can minor decrease in content quality (by WER and CER) of DISSC compared to SR which does convert prosody, but still superior to AutoPST. addition, seq2seq also has better WER, at the expense harming the rhythm conversion. Potentially, the supervised phoneme recogniser is too fine, thus does not impose enough of rhythm bottleneck. The harm in quality of DISSC may be due to additional information bottlenecks by de de-duplicating the speech units, and predicting FO.
Original
Ands
DISSC
finds
AutopST
fints
We additionally provide subjective evaluation considering both naturalness of the converted sam- ples and their similarity to the target recording con- sidering speaking style. These results are shown in Figure 4. We found that the naturalness of both DISSC variants was comparable to SR, and not far from the reference recordings while far outper- forming AutoPST. In addition, we noticeable increase in speaking style similarity when convert- ing prosody with DISSC compared to SR. How- ever, also converting the pitch did noticeably improve the results. We believe that this has to do with the lack distinct speaking style which peo- ple can notice, especially from a single recording.
Figure 5: DISSC corrects abnormal rhythm patterns in specific words, compared AutoPST. Here take the utterance "people look, but no one ever finds it" and stretch the duration of the word "finds" times three.
Irregular rhythm. Lastly, to demonstrate that DISSC predicts rhythm depending content, and not only on average we create sample with ab- normal rhythm patterns. We take an utterance and artificially stretch one of the words using speech resynthesis. We then attempt to "convert" the sam- ple to the original speaker thus checking the ability to only change the rhythm of the abnormal part, We compare DISSC and AutoPST in Figure 5, and see that DISSC manages to mostly correct the ir- regular rhythm, while hardly changing the other parts. By contrast, AutoPST only slightly changes the rhythm. If E, performs optimally, we are guar- anteed perfect results, as o, would identical for irregular and original after dedup. However, Hu- BERT may introduce some errors specifically when considering signal variations (Gat et al., 2022).
Ablation study. In order to properly evaluate the ability of DISSC learn the pitch speaking style, when such exists, we use the synthetic VCTK data (as explained in Section 4.1) with known pitch trends. These results (Table 2) are an ablation study for each converter of DISSC. They highlight the control-ability of our approach when predicting the rhythm, only length errors improve, likewise for pitch. We that when a pitch pattern exists in the speaking style- DISSC learns it, provid- ing 88-1133 133% relative increase in the metrics. We also see that they can converted jointly without noticeable drop in pitch and length performance.
6 Conclusion & Future Work
In this work, we propose a simple and effective method for speaking style conversion based on dis- crete self supervised speech representations. We

---

provide a formal definition the speaking style conversion setup while introducing a set of eval- uation functions to analyse and evaluate different speech characteristios separately. Results suggest, the proposed approach is superior to the evaluated baselines considering both objective and subjective metrics. We hope these advance the study in the field, and lead to improved performance.
References
Zalán Borsos, Raphaël Marinier, Damien Vincent, Eu- gene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2022. Audiolm: language mod- eling approach to audio generation. arXiv preprint arXiv:2209.03/
Li-Wei Chen. Shinji Watanabe, and Alexander Rud- nicky. 2022. A unified one-shot prosody and speaker conversion system with self--uue dis- crete speech units. arXiv preprint arXiv:2211.06533
The current model suffers from slight content loss when using the deduped units. In addition, repeated utterances get slightly different units (in the abnormal rhythm experiment). Therefore, in future work we aim to improve the robustness and disentanglement of the speech representation con- sidering speaker and prosodic information.
Meiying Chen and Zhiyao Duan. 2022. Con- trolve: Zero-shot voice conversion with time-na controls on pitch and rhythm. arXiv preprint arXiv:229.111866.
Yen-Hao Chen, Da-Yi Wu, Tsung-Han Wu, and Hung- yi Lee. 2021. Again-ve: A one-shot voice conver- sion using activation guidance and adaptive instance normalization. In ICASSP 2021-2021 IEEE Interna- tional Conference Acoustics, Speech and Signal Processing (ICASSP), pages 5954-5958, IEEE
Limitations
Our approach can only perform conversion into a closed set of speakers seen at train time. This is due to the LUT used to learn speaker representations and to the use of per-speaker pitch statistics. This might be hard to overcome speaker style (rhythm and pitch changes) can be hard to judge from small number of samples. Additionally, we see that speaking style conversion and especially rhythm conversion harm the WER and CER slightly. The additional bottleneck of de-duplication which al- lows us to convert rhythm comes with the cost of slight content loss. We hope that future improve- ment of the hidden representations will help im- prove alleviate the tradeoff.
Ju-chieh Chou, Cheng-chieh Yeh, and Hung-yi Lee. 2019. One-shot voice conversion by separating speaker and content representations with instance normalization. arXiv preprint arXiv:1904.00 007.0254.
Brecht Desplanques, Jenthe Thienpondt, and Kris De- muynck. 2020. ECAPA-TTNN: emphasized chan- nel attention, propagation and aggregation TDNN based speaker verification. In Interspeech 2020. pages 3830-3834. ISCA.
Itai Gat, Felix Kreuk, Ann Lee, Jade Copet, Gabriel Synnaeve. Emmanuel Dupoux, and Yossi Adi. 2022. On the robustness of self-supervised representa- tions for spoken language modeling. arXiv preprint arXiv:2209.1546
Ethical Statement
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdel- rahman Mohamed. 2021. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3461
The broader impact of this study is, as in any gener- ative model, the development of high quality and natural speech synthesis model. This has special sensitivity and ethical concerns as such technology might be used to alter voices and speaking style in speech recordings, which can be considered as biometric data and biometric processing. To deal with that we limit the number of speakers that can be synthesized with the proposed approach using pre-defined voices using look-up-table, Another potential risk of the proposed method which is also one of its limitations is that the generated speech content is not always perfect, hence might lead to wrong pronunciations.
Wen-Chin Huang. Yi-Chiao Wu, and Tomoki Hayashi. 2021. Any-to-one soguence-to-sequene voice con- version using self- supervieee discrete speech repre- sentations. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pages 5944-5948. IEEE.
Wen-Chin Huang, Shu-Wen Yang, Tomoki Hayashi, Hung- Yi Lee, Shinji Watanabe, and Tomoki Toda. 2022. S3prl-ve: Open-source voice conversion framework with self-supervised speech representa- tions. In ICASSP 2022-2022 IEEE International Con- ference on Acoustics, Speech and Signal Processing (ICASSP), pages 6552-6556. IEEE.

---

Alexander Kain and Michael W Macon. 1998. Spec- tral voice conversion for text-to-speech synthesis. In Proceedings of the 1998 IEEE International Confer- ence on Acoustics, Speech and Signal Processing, ICASSP'98 (Cat. No. 98CH36181), volume 1, pages 285-288. IEEE
Felix Kreuk, Gabriel Synnaeve, Adam Polyak, Uriel Singer, Alexandre Défossez, Jade Copet, Devi Parikh, Yaniv Taigman, and Yossi Adi. 2022a. Audiogen: Textually guided audio generation. arXiv preprint wrXv:2209.15352.
Felix Kreuk, Yaniv Taigman, Adam Polyak, Jade Copet. Gabriel Synnaeve, Alexandre Défossez, and Yossi Adi. 20226. Audio language modeling using perceptually-guide discrete representations. arXiv preprint 2rX0v2221..01223
Hirokazu Kameoka, Takuhiro Kaneko, Kou Tanaka, and Nobukatsu Hojo. 2018. Stargan-ve: Non-parallel many voice conversion using star generative adversarial networks. In 2018 IEEE Spoken Lan- guage Technology Workshop (SLT), pages 266-273. LEEE.
Michael Kuhlmann, Fritz Seebauer, Janek Ebbers, Petra Wagner, and Reinhold Hacb-Umbach. 2022. Investi- gation into target speaking rate adaptation for voice conversion. arXiv preprint arXiv:2209.01978
Takuhiro Kaneko and Hirokazu Kameoka. 2017. Parallel-data-fie voice conversion using cycle- consistent adversarial networks. arXiv preprint
arXiv:1711.11293.
Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, et al. 2021. On generative spoken lan- guage modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336- 1354.
Kavita Kasi and Stephen A Zahorian. 2002. Yet another algorithm for pitch tracking. In 2002 ieee interna- tional conference on acoustics, speech, and signal processing, volume I, pages I-361. IEFE.
Eugene Kharitonov, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Paden Tomasello, Ann Lee, Ali Flkahky, Wei-Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. 2022a. textless- lib: a library for textless spoken language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies: System Demonstrations, pages 1-9, Hybrid: Seattle, Washington + Online. Association for Computational Linguistics.
Ann Lee, Peng-Jen Chen, Changhan Wang. Jiatao Gu. Sravya Popuri, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang. Juan Pino, and Wei-Ning Hsu. 2022a. Direct speech- to-eehht translation with dis- crete units. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pages $327-3339, Dublin, Ireland, Association for Computational Linguisties.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Yossi Adi, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. 2022b. Textless speech-to-speedt translation on real data. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 860-872, Seattle, United States. Association for Computational Lin- guistics.
Eugene Kharitonov, Ann Lee. Adam Polyak, Yassi Adi, Jade Copet, Kushal Lakhotia, Tu Anh Nguyen, Mor- gane Riviere, Ahdelrahman Mohamed, Emmanuel Dupoux, and Wei-Ning Hsu. 2022b. Text-free prosody- generative spoken language modeling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Pupers). pages 8666-8681, Dublin, Ireland. Association for Computational Linguistics.
Sang-Hoon Lee, Ji-Hoon Kim, Hyunseung Chung, and Scong-Whan
Kang-wook Kim, Seung-won Park, Junhyeok Lee, and Myun-chul Joe. 2022. Assem-ve: Realistic voice conversion by assembling modern speech synthesis techniques. In ICASSP 2022-2022 IEEE Interna- tional Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6907-7001. IEEE
Whan Lee. 2021. Voicemixer: Adversarial voice style mixup. In Advances in Neural Informa- tion Processing Systems, volume 34, pages 294-308. Curran Associates, Inc.
Sang-Hoon Lee, Hyeong-Rac Noh, Woo-Jeoung Nam, and Scong-Wine Whan Lee. 2022c. Duration controllable voice conversion via phoneme-based information bot- tleneck. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 30:1173-1183.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. 2020. Hifi-gan: Generative adversarial networks for effi- cient and high fidelity speech synthesis. Advances in Neural Information Processing Systems, 13:122 17033.
Jheng-hao Lin, Yist Y Lin, Chung Ming Chien, and Hung-yi Lee. 2021a. S2ve: a framework for any-to- any voice conversion with self-supervised pretrained representations. arXiv preprint (00Xi:::104.000 02901.
Felix Kreuk, Adam Polyak, Jade Copet, Eugene Kharitonov, Tu-Anh Nguyen, Morgane Rivière, Wei- Ning Hsu, Abdelrahman Mohamed, Emmanuel Dupoux, and Yossi Adi. 2021. Textless speech emo- tion conversion using decomposed and discrete rep- resentations. arXiv preprint arXiv: 11100222..
Yist Y Lin, Chung-Ming Chien, Jheng-Hao Lin, Hung- yi Lee, and Lin Lin-shan Lee. 2021b. Fragmentve: Any- to-any voice conversion by end-to-end extracting and

---

fusing fine-grained voice fragments with attention. In ICASSP 2021-2021 IEEE International Confer- ence on Acoustics, Speech and Signal Processing (ICASSP), pages 5939-5943, IEEE
Kaizhi Qian, Yang Zhang, Shiyu Chang, Xuesong Yang. and Mark Hasegawa- Johnson. 2019. Autove: Zero- shot voice style transfer with only autoencoder loss. In International Conference on Machine Learning, pages 5210-5219. PMLR.
Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, and Helen Meng. 2021. Any-to-many voice conversion with location rellttvvee sequence-to sequence modeling. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:1717- 1728.
Kaizhi Qian, Yang Zhang, Heting Gao, Junrui Ni, Cheng-I Lai, David Cox, Mark Hasegawa-Johnson, and Shiyu Chang. 2022. Contentvec: An improved self-supervised speech representation by disentan- gling speakers. In International Conference on Ma- chine Learning, pages 118003-88011 PMLR.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sonderegger. 2017. Montreal Forced Aligner: Trainable Text-Speech Aligument Using Kaldi. In Proc. Interspeech 2017,
Leyuan Qu, Taihao Li, Coreelis Weber. Theresa Pekarek- Fuji Ren, and Stefan Wermter. 2022. Disentangling prosody representations with unsu- pervised speech reconstruction. arXiv preprint arXiv:2212.0697..
pages 502-
Toru Nakashika, Ryoichi Takashima, Tetsuya Takiguchi, and Yasuo Ariki. 2013. Voice conversion in high- order eigen space using deep belief nets. In Inter- speech, pages 372.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brock- man, Christine McLeavey, and Ilya Sutskever. 2022. Robust speech recognition via large-scale weak pervision. arXiv preprint arXiv:2212.0433.
Tu Anh Nguyen, Eugene Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Elkahky, Paden Tomasello, Robin Algayres, Benoit Sagot, Abdelrahman Mo- hamed, et al. 2022. Generative spoken dialogue lan- guage modeling. arXiv preprint arXiv:2203. 16502.
Mirco Ravanelli, Titouan Parcollet, Peter Plantinga. Aku Rouhe, Samuele Cornell, Loren Lugosch, Cem Subakan, Nauman Dawalatahad, Abdelwahab Heba, Jianyuan Zhong. Ju-Chieh Chou, Sung-Lin Yeh, Szu-Wei Fu, Chien-Feng Liao, Elena Rastorgueva. François Grondin, William Aris, Hwidong Na, Yan Gao, Renato De Mori, and Yoshua Bengio. 2021. SpeechBrain: A general-purpose speech toolkit. ArXiv:2106.04624
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. 2015. Librispeech: an asr cor- pus based on public domain audio books. In 2015 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 5206-5210. IBEE.
Yossi Rubner, Carlo Tomasi, and Leonidas Guibas. 1998. A metric for distributions with applications to image databases. In Sixth international conference on computer vision (IEEE Cat. No. 98CH36271). pages 59-66. IEEE
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- delrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentan- gled self-supervised representations. arXiv preprint arXiv:2104.00355.
Yannis Stylianou, Olivier Cappé, and Eric Moulines. 1998. Continuous probabilistic transform for voice conversion. IEFF Transactions speech and audio processing, 612):131-
Sravya Popuri, Peng-Jen Chen, Changhan Wang. Juan Pino. Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. 2022. Enhanced direct speech-to-speech transla- tion using self-supervised pre-training and data aug- mentation. arXiv preprint 20Xiv:2204002960
Christophe Veaux, Junichi Yamagishi, Kirsten MacDon- ald, et al. 2017. Superseded-cstr vetk corpus: English multi-speaker corpus for estr voice cloning toolkit.
Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, and Juan Pino. 2021. fairseq s*2: A scalable and inte- grable speech synthesis toolkit. In Proceedings of the 2021 Conference on Empirical Methods in Nat- ural Language Processing: System Demonstrations, pages 143-152 Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazzl-Zarandi, et al. 2023. Scaling speech technology to 1,000+ languages. arXiv preprint arXiv:2305.1 13311666..336...
Kaizhi Qian, Yang Zhang. Shiyu Chang. Mark Hasegawa-Johnson, and David Cox. 2020. Unsu- pervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pages 7836-7846. PMLR.
Carl Earl Williams. 1965, The effects of factors on the aural identification of speakers. University of Illinois at Sirban-Chhappigg
Kaizhi Qian, Yang Zhang, Shiyu Chang, Jinjun Xiong, Chuang Gan, David Cox, and Mark Hasegawa- Johnson. 2021. Global prosody style transfer without text transcriptions. In International Conference on Machine Learning, pages 8650-8660. PMLR
Siyang Yuan, Pengyu Cheng. Ruiyi Zhang, Weituo Hao,
Zhe Gan, and Lawrence Carin. 2021. Improving zero-shot voice style transfer via disentangled repre- sentation learning. International Conference on Learning Representations.

---

Kun Zhou, Berrak Sisman, Rui Liu, and Haizhou Li. 2022. Emotional voice conversion: Theory, databases and esd. Speech Communication, 137: 1- 18.
A Appendix
Here we give fine-grained details of computational costs, hyper-parameee and other implementation details omitted from the main part for brevity and readability.
A.1 Dataset details
We use several datasets: (i) VCTK dataset (Veaux et al., 2017); (ii) ESD (Zhou et al., 2022); (iii) and the synthetic VCTK dataset (Syn_ VCTK), gen- erated by For VCTK the training data contains 41k samples, and the evaluation set contains 2.5k. For ESD the training data contains 3k samples, the validation contains 200, and the test 300. For Syn_VCTK VCTK the training data contains 2.4k samples and the validation has 142. For ESD we use the of- ficial train, val, test split. For both VCTK versions enforce unpaired data in the train sets, and use paired data only for evaluation.
A.2 Computational Price
We use a pretrained HuBERT Base model for input encoding. This model has 90 million parameters and takes less than 15 minutes to inference over all of our datasets. For training times see the original paper. For the HiFi-GAN vocoder which is trained once, independently of the speaking style conver- sion, we use model with 13.8 million parameters (for the generator). It takes days on to train on VCTK using two RTX 6000 GPUs, other datasets take less than that.
The rhythm predictor Edur has 330 thousand parameters, and takes about 30 minutes to train on a single RTX2080 GPU. Likewise, the pitch predictor Eyo has 527 thousand parameters, and also takes about 30 minutes to train on a single RTX2080 GPU.
A.3 Hyper-Parameters
We encode the HuBERT units and pitch contour with textlesslib. For the content endocer we use mubbert-base-ssee and use K-means with 100 clus- ters for the clustering step. For the pitch contour we use the default parameters from textless lib which match the content encoder we used.
For the vocoder use the official implementa- tion of speech resynthesis with the configuration suitable for VCTK, hubert100, using fixed size lookup-table. We adapt the model so that re- ceives as input the normalised pitch without quanti- sation. We train it with a batch size of 64 over two

---

GPUs for the minimum between 2000 epochs and 400,000 batches.
automated hyper-tuning was done.
A.4 Human Evaluation Details
The rhythm predictor Edur is built of 8 blocks of CD followed by Batch Normalisation, with 128 hidden units. The speaker embedding and to- ken embedding are Iearned with a lookup-table, both with size 32. The speaker vector is repeated along the time axis and concatenated with the unit embedding. Each CNN has a receptive field of 3 and padding of 1 so that the output shape is the same the input. For activation ReLU. We normalise the length for prediction with the mean and standard deviation of the training data. Train time masking masks each unit in the input independently with 20% probability. We train the model for 30 epochs, with a batch size of 32, a learning rate of 3e - 4 and Adam optimiser. We take the model with lowest validation MSE. This architecture and parameters were fairly standard values which worked out of the box, no manual or automated hyper-tuning was done.
We had over 20 different annotators in our sub- jective tests, who were all non-paid volunteers. They are all fluent, though not necessarily native, in English, and vary in academic background and connection to the field. For privacy reasons the questionnaires are anonymous and no details are collected about the participants.
The quality question is phrased as follows: "Evaluate the quality and naturalness of the fol- lowing audio segment", and the raters give a scale of 1-5 with the labels: "Poor", "Decent", "Good", "Very Good", "Excellent".
use Leaky
The similarity question is phrased as follows: "The first recording in this section is reference, mark all recordings in how similar the speaking style is to the original. speaking style refers to the speaking rhythm, intonation etc. Try to ignore the quality of the recording itself", and the raters give a scale 1-5 with the labels: "Very different", "Dif- ferent", "Slighlly similar", "Similar", "Very Simi- lar".
The pitch predictor Ero encodes the speaker and units in the same way as Eduor with fixed size lookup-table of size 32. However, here we add to the speaker embedding a linear positional encod- ing - this means that indices 0-15 of the speaker vector get added the unit index relative to the start and indices 16-31 get added the unit index rela- tive to the end. In preliminary experiments more typical sinusoidal positional encoding worked sim- ilarly but we chose the linear version for simplicity. The model comprised of 9 shared Conv ID lay- ers with kernel size 3, and followed by Conv ID with kernel sizes 3 and for each head (speaking classification and FO regression). We use Leaky ReL.U activation. The labels for training are re- ceived from the YAAPT algorithm, normalised per- speaker with the mean and standard deviation of the vocalised parts of the training set. We do not normalise the non- non-voculised parts. The BCE loss for binary classification is multiplied by 100 and added to the pitch MSE loss, to attempt to give both losses a similar scale. We train the model for 20 epochs with learning rate 3e - 4, batch size of 32 and Adam optimiser. We take the model with the lowest validation MSE. The results in Table 1, and Figure 4 are with similar model, however it does not use positional encoding. The positional encod- ing was added afterwards as an improvement. This architecture and parameters were fairly standard values which worked out of the box, no manual or
A.5 Low Resource Language Conversion
As mentioned, DISSC does not use text any intermediate representation in conversion (or text supervision for training). This means that it can work in low-resource languages which do not have large annotated datasets.
We demonstrate the ability of DISSC to con- vert a Hebrew utterance to a target VCTK speaker, while not being trained on any Hebrew data (see Fig. 6). While converting the accent in cross- lingual (Hebrew utterance to English speaker) re- mains challenging, this shows good potential. No- tably, this example is out of domain for all model training data. Training DISSC in a textless self- supervised way on Hebrew data will likely improve the performance dramatically.
A.6 WER detailed analysis
While DISSCis the only evaluated method to suc- cessfully convert prosody, it has a limitation of slightly harming content compared to some meth- ods which do not convert prosody (c.g. speech resythesis). We see this in the WER and CER met- rics and wish to further analyse the cause for this.
We first manually inspect the samples with high- est word error rate. We have found that the content loss is mainly due to minor phonological errors

---

Unseen Hebrew Speaker
Converted Fast Speaker
Target Speaker
Method
SR
DISSC Rhythm
p231
6.2
11.2
p239
7.0
11.4
p245
6.6
11.6
sen 500
'IUO 10100
p257
6.4
10.5
Converted to Slow Speaker
p270
7.2
10.9
p284
6.4
11.5
Figure 6: An example using DISSC to convert a non- English sample. Hebrew is lower resource language, hence there are little to no transcribed datasets. Thus. no expressive TTS methods exist, making an ASR-TTS VC pipeline impossible. Training DISSC untabelled Hebrew data, will likely improve results further. We en- courage readers to listen the accompanying samples.
p306
7.2
11.4
p341
7.1
11.8
Table 4: Word rate comparison VCTK, split by target speaker when converting from all other speakers as sources. For instance the first row denotes the WER when converting fixed set of samples from all source speakers in the list (except p231) to speaker p231. We compare DISSC DIssC_Rhythm and Speech resynthesis and that the results hardly depend on the target speaker.
Source Speaker
Method
SR
DISSC Rhythm
p231
6.4
11.1
p239
12.2
18.6
target speaker, this further supporting the fact that this is an issue with the SSL encoding method. An interesting line of future work would be in improv- ing SSL for such cases, or even applying error- correcting codes on the converted speech.
p245
4.5
7.9
p257
9.8
15.7
p270
3.0
7.7
p284
12.1
18.2
p306
4.5
7.0
p341
1.3
4.0
Table 3: Word error rate comparison VCTK, split by source speaker when converting to all other speakers targets. For instance the first row denotes the WER when converting fixed set of samples from speaker p231 to all other speakers in the list. We compare DISSC Rhythm and Speech resynthesis and see that the results vary greatly, and that the success of both methods are correlated.
which cause transcription change. For example, in cases with a high error rate, the original text is: "Please call Stella" and the transcription after conversion is: "Peace cool Stalla". However, when looking at the WER of both DISSC and speech resynthesis, observe that such phonological er- rors are observed in the same samples for both methods (even without conversion). Therefore, we believe that the root cause of such phenomenon is the quality of the SSL model and DISSC amplified this effect.
To further, evalute this hypothesis evaluate the WER per source speaker and per target speaker (results in Tables 3 and 4). This analysis clearly shows that WER per-speaker varies greatly based on the source speaker, but does not depend on the


Reviews:
[{"review_id": "6cC6Ncs6Ta", "reviewer": "Reviewer_rc2g", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": 1, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We believe there is a misunderstanding as __we do not violate the double-blind review policy__. While we do have a non-anonymous pre-print published online this is in line with the official EMNLP anonymity guidelines. As the official guidelines state _“If you have posted a non-anonymized version of your paper online before the start of the anonymity period, you may submit an anonymized version to the conference. The submitted version must not refer to the non-anonymized version, and you must inform the program chairs that a non-anonymized version exists”_. We uploaded the non-anonymous version much before the start of the anonymity period (uploaded in December 2022, while anonymity starts in May 2023). We would also like to highlight that we did not promote it online or make changes during the anonymity period. \n\nWe hope the reviewer reconsiders their review and will rate our paper based on its merit.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I will reject this paper due to a violation of blind review, as I discovered that it was publicly available at https://arxiv.org/pdf/2212.09730.pdf.", "reasons_to_reject": "I will reject this paper due to a violation of blind review, as I discovered that it was publicly available at https://arxiv.org/pdf/2212.09730.pdf.", "questions_to_authors": "", "paper_topic_and_main_contributions": "I will reject this paper due to a violation of blind review, as I discovered that it was publicly available at https://arxiv.org/pdf/2212.09730.pdf.", "excitement": 1, "reproducibility": null, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Rup2xN86bn", "reviewer": "Reviewer_RPWj", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for taking the time to carefully read our paper and provide valuable feedback. We are happy to hear the reviewer __found our paper relevant, novel and thoroughly evaluated__. We are also glad that they recognised the __importance of the evaluation suite proposed for future research__.\n\n\n__Regarding the lack of attention to other languages__ – we agree that developing speaking style conversion (SSC) models for more languages is an interesting line of research. Unfortunately, in that setting, there is a lack of large datasets with parallel, expressive speech. Moreover, to be consistent with the literature we follow prior work which focuses on English datasets [1, 2, 3]. However, unlike many existing methods, our approach is textless which means datasets without text annotations are also relevant for training. Hence, “lowering the bar” for SSC in other languages. \nWe demonstrate in the appendix the ability to perform SSC on a low-resource language without additional training in the target language (i.e., zero-shot language transfer). This demonstrates the potential for low-resource languages when performing textless self-supervised training, and hopefully opens the door to multilingual models in future research.\n\n__Regarding the ability of the evaluation methods to transfer to different languages__ – the newly proposed evaluation suite is generic and can transfer to other languages. However, certain metrics are based on language-specific pre-trained models (i.e., ASR-based metrics). In such cases, a multi-lingual ASR can be used. In our setup, we use the Whisper model [4] which does support ~100 languages. Another option would be to use the MMS model [5] which supports more than 1000 languages for both ASR and forced-alignment.\n\n&nbsp;\n\n\n[1] Kaizhi Qian, Yang Zhang, Shiyu Chang, Jinjun Xiong, Chuang Gan, David Cox, and Mark Hasegawa-Johnson. 2021. Global prosody style transfer without text transcriptions. In International Conference on Machine Learning, pages 8650–8660. PMLR.\n\n[2] Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab-delrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355\n\n[3] Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, and Helen Meng. 2021. Any-to-many voice conversion with location-relative sequence-to-sequence modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1717–1728.\n\n[4] Radford, Alec, et al. \"Robust speech recognition via large-scale weak supervision.\" International Conference on Machine Learning. PMLR, 2023.\n\n[5] Pratap, Vineel, et al. \"Scaling speech technology to 1,000+ languages.\" arXiv preprint arXiv:2305.13516 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "I think this paper is a good candidate to be accepted because of its 1) relevance, 2) novelty 3) complexity of evaluations performed. This paper will be interesting for a wide audience of EMNLP, both for academia and industry professionals. The topic of voice conversion is very popular especially in industry application where the style transfer of the voice is the most complicated task. The paper provides a novel approach for style transfer and demonstrates through a thorough evaluation that is superior against the baseline.  On top of it they provide an ablation study that gives extra insights into the model. As a final important reason to accept this paper I consider a proposal of a complex and diverse approach on how to evaluate the style transfer between voices, which a non trivial task with limited literature on it.", "reasons_to_reject": "There is not much attention in this paper on how this method on both voice style conversion and its evaluation transfers across languages and what are the main limitations expected.  ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is focusing on one of the hot topic in TTS- speaker conversion. It talks about the most difficult part of this task- conversion of the speaker style. While there are a lot of methods that allow to make voice identity conversion, speaker style is non trivial. The author propose simple and effective method for speaker style conversion, moreover they also propose a way how to measure the quality of style conversion. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ao7xZV6kgt", "reviewer": "Reviewer_GxPi", "summary": "", "questions": "1. Can you provide more details about the generalization capabilities of DISSC? Specifically, how does the model perform on unseen speakers, and what strategies have been employed to address the closed-set limitation?\n\n2. The paper mentions slight content loss during de-duplication. Can you elaborate on the impact of this content loss on the overall quality of the generated speech? Are there any strategies or techniques considered to mitigate this issue?\n\n3. Can you provide more insights into the failure cases of DISSC? An error analysis highlighting the limitations and potential weaknesses of the proposed method would be valuable for understanding its performance and future improvements.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate the reviewer’s feedback, and their comments and suggestions for future work are of great interest. We are delighted that __they found our approach novel and acknowledged the value of the evaluation setup__. We are pleased that they found the __evaluation and ablation studies to be comprehensive__.\n \n__Regarding the content de-duplication loss__ - As mentioned in the paper (lines 595-598, 609-614), the de-duplication and the ability of the HuBERT units to model content independently are not perfect. We analyze the impact of such phenomena through the ASR metric when de-duplicating (converting rhythm) compared to only converting pitch in Table 1. As we discussed in Section 6, there could be different methods to improve this including better SSL models (with higher correlation to phonemes), use of more expressive continuous representation for each de-duplicated time step, etc. We believe these will be addressed in future work. \nHowever, when considering the quality of the converted speech (i.e., MOS scores), DISSC still outperforms the SSC evaluated baselines and comparable to the baselines which do not alter the prosodic features (i.e., speech-resynthesis [2]). \n\nNevertheless, following the reviewer’s comment, we further analyze conversions with respect to the content loss, see the Tables below and paragraph below.\n \n__Regarding the lack of large scale experiments__ – We follow the standard practice in the literature [1,2,3], and even go beyond these by adding ESD and synthetic VCTK. We hope that formalizing the setup, including the evaluation requirements, will drive data annotation efforts which will create even larger real-world datasets for speaking style conversion.\n\n__Regarding generating only seen speakers__ - Speaking style tends to be more diverse than speaker timbre, we believe it is harder to capture true speaking style from only a few seconds. Hence, we focus on seen speakers only. Moreover, as stated in the paper (lines 208-211), prior work found that limiting the generator to seen speakers only, improves the generation quality [2]. Furthermore, while some works convert prosodic features based on a single utterance [6] this does not capture the general speaking style of the speaker. \n\n__Regarding error analysis__ - Thanks for the suggestion, following the reviewer’s request we conducted a fine-grained analysis to better understand the limitations of the proposal with respect to content loss. We analyze the WER as a function of the different source speakers and target speakers. See the Tables below. \n\n| source speaker \\ WER | Speech-Resynthesis | DISSC_Rhythm |\n|----------------------|----|--------------|\n| p231                 |  6.4  | 11.1         |\n| p239                 |  12.2  | 18.6         |\n| p245                 |  4.5  | 7.9          |\n| p257                 |  9.8  | 15.7         |\n| p270                 |  3.0  | 7.7          |\n| p284                 |  12.1  | 18.2         |\n| p306                 |  4.5 | 7.0          |\n| p341                 |  1.3  | 4.0          |\n\n\n| target speaker \\ WER | Speech-Resynthesis | DISSC_Rhythm |\n|----------------------|----|--------------|\n| p231                 |  6.2  | 11.2         |\n| p239                 |  7.0  | 11.4         |\n| p245                 |  6.6  | 11.6         |\n| p257                 |  6.4  | 10.5         |\n| p270                 |  7.2  | 10.9         |\n| p284                 |  6.4  | 11.5         |\n| p306                 |  7.2  | 11.4         |\n| p341                 |  7.1  | 11.8          |\n\nWe have found the content loss is mainly due to minor phonological errors which cause transcription change. For example, in cases with a high error rate, the original text is: “Please call Stella” and the transcription after conversion is: “Peace cool Stalla”. \nHowever, when looking at the WER of both DISSC and speech resynthesis, we observe the phonological errors are also observed in the same samples in both methods (even without conversion), hence, we believe the root cause of such phenomenon is in the quality of the SSL model and DISSC amplified such an effect. The fact that WER per-speaker varies greatly based on the source speaker, and not the target speaker further supports the fact that this is an issue with the SSL encoding method. An interesting line of future work would be in improving SSL for such cases, or even applying error-correcting codes on the converted speech.\n\n&nbsp;\n\n[1] Kaizhi Qian, Yang Zhang, Shiyu Chang, Jinjun Xiong, Chuang Gan, David Cox, and Mark Hasegawa-Johnson. 2021. Global prosody style transfer without text transcriptions. In International Conference on Machine Learning, pages 8650–8660. PMLR.\n\n[2] Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab-delrahman Mohamed, and Emmanuel Dupoux. 2021. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint arXiv:2104.00355\n\n[3] Songxiang Liu, Yuewen Cao, Disong Wang, Xixin Wu, Xunying Liu, and Helen Meng. 2021. Any-to-many voice conversion with location-relative sequence-to-sequence modeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 29:1717–1728.\n\n[6] Kaizhi Qian, Yang Zhang, Shiyu Chang, Mark Hasegawa-Johnson, and David Cox. 2020. Unsupervised speech decomposition via triple information bottleneck. In International Conference on Machine Learning, pages 7836-7846. PMLR.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Approach: The paper proposes a novel approach, DISSC, for speaking style conversion using discrete self-supervised speech representations. This novel method contributes to the advancement of speech synthesis techniques and provides an innovative solution to the speaking style conversion problem.\n\n2. Formal Definition and Evaluation Metrics: The paper provides a formal definition of the speaking style conversion setup, which adds clarity and structure to the research in this area. Moreover, the introduction of new fine-grained offline evaluation metrics, such as Phoneme Length Error, Word Length Error, Total Length Error, and Earth Movers Distance (EMD) for pitch contour comparison, helps in assessing the quality of speaking style conversion in a more detailed manner, which can be valuable for future research and benchmarking.\n\n3. Empirical Results: The paper presents comprehensive empirical results that demonstrate the superiority of DISSC over several baseline methods, including both objective and subjective evaluations. This empirical evidence and performance comparison provide confidence in the effectiveness of the proposed approach and its potential practical applicability.\n\n4. Ablation Study and Control-ability: The inclusion of an ablation study that analyzes the effect of individual components of DISSC shows the control-ability of the approach in handling rhythm and pitch conversion separately or jointly. This understanding of the model's behavior is valuable for researchers and developers seeking to tailor the approach to specific requirements.", "reasons_to_reject": "1. Content Loss in De-Duplication: The paper mentions that de-duplication of speech units may lead to slight content loss. This content loss might impact the overall quality of the generated speech, and the paper does not provide a detailed analysis of this effect or a thorough discussion on mitigating this issue.\n\n2. Lack of Large-Scale Experiments: The paper presents results on datasets with a limited number of speakers (e.g., VCTK and ESD). Conducting experiments on larger and more diverse datasets, encompassing a broader range of speaking styles, would strengthen the validity of the findings and provide more robust insights into the performance of DISSC across different scenarios.\n\n3. Limited Generalization: The proposed method appears to be limited to a closed set of speakers seen during training. This restricts the applicability of DISSC to unseen speakers, which is a common requirement in real-world scenarios. Addressing this limitation and demonstrating the ability to generalize to unseen speakers would significantly enhance the practical utility of the proposed approach.", "questions_to_authors": "1. Can you provide more details about the generalization capabilities of DISSC? Specifically, how does the model perform on unseen speakers, and what strategies have been employed to address the closed-set limitation?\n\n2. The paper mentions slight content loss during de-duplication. Can you elaborate on the impact of this content loss on the overall quality of the generated speech? Are there any strategies or techniques considered to mitigate this issue?\n\n3. Can you provide more insights into the failure cases of DISSC? An error analysis highlighting the limitations and potential weaknesses of the proposed method would be valuable for understanding its performance and future improvements.", "paper_topic_and_main_contributions": "This paper addresses the problem of speaking style conversion (SSC) in the field of natural language processing (NLP). Speaking style conversion involves transforming the prosodic features of a speech signal, such as pitch and rhythm, while preserving the phonetic content and speaker identity. The main contribution of this paper is the introduction of a novel approach called \"DISSC\" (Discrete Self-Supervised Speech Conversion) for speaking style conversion using discrete self-supervised speech representations, the introduction of new evaluation metrics, and the demonstration of superior performance compared to existing baselines. These contributions advance the research in the field of speech synthesis and have potential applications in various NLP-related tasks.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper address the problem of speaking style conversion and proposed a approach DISSC. 

Pros:
- comprehensive results for comparing with baseline methods
- Showing evaluation criteria for speaking style conversion.
Cons:
- Large scale speaker experiment would be more helpful
- There is some content loss with de-duplication.
Decision: Accept-Findings