Bag of Tricks for Unsupervised Text-to-Speech
Authors: Yi Ren
Chen Zhang
Shuicheng YAN
Venue: ICLR 2023 poster (2023)
Keywords: speech synthesis
unsupervised learning

Abstract: Unsupervised text-to-speech (TTS) aims to train TTS models for a specific language without any paired speech-text training data in that language. Existing methods either use speech and corresponding pseudo text generated by an unsupervised automatic speech recognition (ASR) model as training data, or employ the back-translation technique. Though effective, they suffer from low robustness to low-quality data and heavy dependence on the lexicon of a language that is sometimes unavailable, leading to difficulty in convergence, especially in low-resource language scenarios. In this work, we introduce a bag of tricks to enable effective unsupervised TTS. Specifically, 1) we carefully design a voice conversion model to normalize the variable and noisy information in the low-quality speech data while preserving the pronunciation information; 2) we employ the non-autoregressive TTS model to overcome the robustness issue; and 3) we explore several tricks applied in back-translation, including curriculum learning, length augmentation and auxiliary supervised loss to stabilize the back-translation and improve its effectiveness. Through experiments, it has been demonstrated that our method achieves better intelligibility and audio quality than all previous methods, and that these tricks are very essential to the performance gain.

Published as a conference paper at ICLR 2023
BAG OF TRICKS FOR UNSUPERVISED TTS
Yi Ren', Chen Z Shuicheng Yan'
SEA AI Lab, Zhejiang University
3 com, 2c9902 ju. edu. cn, yanscesea.com com
ABSTRACT
Unsupervised text-to-speech (TTS) aims to train TTS models for specific lan- guage without any paired speech-text training data in that language. Existing methods either use speech and corresponding pseudo text generated by an un- supervised automatic speech recognition (ASR) model as training data, or employ the back-translation technique. Though effective, they suffer from low robust- ness to low- uww-quality data and heavy dependence on the lexicon of a language that is sometimes unavailable, leading to difficulty in convergence, especially in low- resource language scenarios. In this work, we introduce a bag of tricks to enable effective unsupervised TTS. Specifically, 1) we carefully design voice conver- sion model to normalize the variable and noisy information in the low-quality speech data while preserving the pronunciation information; we employ the non-autoregressive TTS model to overcome the robustness issue; and 3) we ex- plore several tricks applied in back-translation, including curriculum learning, length augmentation and auxiliary supervised loss to stabilize the back-translation and improve its effectiveness. Through experiments, that our method achieves better intelligibility and audio quality than all previous methods, and that these tricks are very essential to the performance gain.
has been demonstrated
INTRODUCTION
Text to speech (TTS), or speech synthesis, has been a hot research topic (Wang et al., 2017; Shen et al., 2018; Ming et al., 2016; Arik et al., 2017; Ping et al., 2018; Ren et al., 2019a; Li et al., 2018; Ren et al., 2021a; Liu et al., 2021; Ren et al., 2021b) and has broad industrial applications as well. However, previous TTS has been developed dominantly for majority languages like English, Mandarin or German, while seldom for minority languages and dialects (low-resource languages), as supervised TTS requires hours of single-speaker and high-quality data to retain a good performance, but collecting and labeling such data for low-resource languages are very expensive and need substantial amount of manpower.
Recently, some works exploit unsupervised (Ni et al., 2022; Liu al., 2022b) semi- semi-unsupervised learning (Tjandra et al., 2017; Ren et al., 2019b; Liu et al., 2020; Xu et al., 2020) to enable speech synthesis for low-resource languages, some of which are summarized in Table Semi-supervised methods rely on a small amount of high-quality paired data in the target language to initialize the model parameters and employ back tack-translation leverage the unpaired data. But high-quality paired data in minor languages are usually collected via recording in professional studios or tran- scribing by native speakers, and hence very costly and sometimes even unaffordable to attain. In contrast, unsupervised methods train an unsupervised automatic speech recognition model (ASR) (Baevski et al., 2021; Liu et al., 2022a) to generate pseudo labels for the unpaired speech data, and then use the pseudo labels and speech paired data to train the TTS model. However, their per- formance tends to be bounded by the performance of the unsupervised ASR model, which is extremely difficult and unstable train some low-resource languages, especially for those with- (G2P) tools (Baevski et al., 2021; Liu et al., 2022a)². Besides,
out lexicon or grapheme-to-phonee
Baevski et al. (2021) claimed their method "requires phonemization of the text for the language of interest", and Liu al. (2022a) claimed "when switching to entirely letter-hased system without a lexicon, the unit error rate increases substantially".

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of some semi-supervised and unsupervised TTS methods. "G2P" denotes tool; "Paired (tgt)" and "Paired (other)" mean using paired data in the target language and other languages; "BT" denotes fack-translation; "NAR" denotes cooreetoreeresivvee architecture for TTS model. "Semi." denotes semi-supervised and "Unsup." denotes unsupervised
Dataset settings
Methods
Type
G2P
BT
NAR
Multispeaker
Paired (tgt)
Paired (other)
Noisy
Ren al. (2019h)
Semi.
X
few
Xu et al. (2020)
Semi.
few
/
Liu et al. (2022h)
Unsup.
/
/
X
X
X
X
X
Ni et al. (2022)
Unsup.
x
X
X
X
X
X
Ours
Unsup
x
X
the unpaired speech samples used in existing unsupervised methods clean and ready for gen-
eral TTS model training, such as CSS10 (Park & Mulc, 2019), LibriTTS (Zen et al., 2019) and LJSpeech (Ito, 2017). However, in real low-resource language scenarios, there is no guarantee that enough clean data can be obtained.
In this work, aim train unsupervised TTS model in low-resource language (the target language) with unpaired data, rather than any paired speech and text data, in that language, and also paired data in other rich-resouree languages for initialization. Such training data are easily acces- sible. For example, the unpaired speech and text in the target language can be crawled from video or news websites in the countries using that language; the paired data in rich-resource languages can be obtained from some ASR and TTS datasets. Besides, these crawled speech data are from different speakers. Under such a task setting, we need to address the following challenges in order to achieve our goal. 1) Low-quality multi-speaker data. The speech data to be used for unsu- pervised training in our problem are often multi-speaker and low-quality, with much variable and noisy information like timbre, background noise, etc., hindering model convergence and meaning- ful speech-text alignment. This significantly increases the difficulty of the TTS model training. 2) Back-translation stability. Previous semi-supervised TTS methods (Xu et al., 2020; Ren et al., 2019b) leverage the unpaired data with back-translation, but only achieving limited performance and sometimes difficult to converge, especially in unsupervised settings. 3) Robustness. Previous http:-/upprriisedduuuuerrere
TTS methods (Xu et al., 2020; Ren et al., 2019b; Ni et al., 2022: Liu al., 2022b) use an auto-regressive architecture (Li al., 2018: Shen al., 2018), which suffers from word missing and repeating issues, especially when the supervision signal is very weak. 4) Lack of lexicon. For low-resource languages, usually difficult to obtain existing lexicons or G2P tools.
We propose several practical tricks to address these issues and enable unsupervised TTS without any paired data in the target language and bridge the performance gap between the unsupervised and supervised TTS. Specifically, 1) we normalize the variable and noisy information in the low-quality training data. We propose a cross-lingual voice conversion model with flow-based enhanced prior, which converts the timbre of all sentences in different languages to one same speaker's 's voice while preserving the pronunciation information. 2) We explore some tricks including curriculum learning, length augmentation and auxiliary supervised loss to improve the effectiveness of fack-translation. 3) To strengthen model robustness, we employ the non-autoregressive (NAR) TTS model and use the alignment extracted from the ASR model in the model training. By applying such bag of tricks, we can successfully train effective TTS model with noisy and multi-speaker data and without any lexicons.
trktsaaanlatoon process to guide the NAR TTS
Through experiments, it has been verified that our method can achieve both high-quality and high- intelligibiliiy TTS. in terms of MOS and of word error rate (WER) and character error rate (CER) evaluated by external ASR, respectively. We compare our method to existing unsupervised TTS baselines (Ren et al., 2019b; Xu et al., 2020; Ni et al., 2022; Liu et al., 2022b) and find it significantly outperforms them both audio quality and intelligibility under the same experimental settings. We conduct some analyses the proposed tricks, which demonstrate the importance and necessity of
"The ASR model is the byproduct of fack-translation which does need any extra paired data.
2

---

Published as a conference paper at ICLR 2023
these tricks to achieve of-tt--f-thh--eeee unsupervised TTS. The samples generated by our models can be found at httttpp://unuupettsstiiic.. github. io.
2 RELATED WORKS
2.1 SUPERVISED SPEECH SYNTHESIS
In the past few years, with the development of deep learning, neural network-based TTS has thrived (Wang et al., 2017; Tachibana et al., 2018; ei al., 2019; Ren et al., 2019a; 2021a; Lańcucki, 2020), where the text-to-speech mapping is modeled by deep neural networks using encoder-decoder architectures. Early methods by Wang et al. (2017) and Ping et al. (2017) generate the mel- spectrogram autoregressively. However, they suffer from slow inference and low robustness issues, egg. word skipping and repeating. To tackle these issues, later works explore non- non-autoregressivee (NAR) speech generation. FastSpeech (Ren et al., 2019a) is the first non-autoregressive TTS archi- tecture, which adopts the duration predictor and length regulator to bridge the length gap between the speech and the text sequence. After that, many methods are proposed, such FastSpeech (Ren et al., 2021a), Glow-TTS (Kim et al., 2020) and EATS (Donahue et al., 2021), achieving not only better audio quality but also fast inference and good robustness. Recently, some NAR models leveraging variational auto-encoder (VAE) to model the variation information in the latent space are developed, like VITS (Kim et al., 2021b) and PortaSpeech (Ren et al., 2021b), and they quickly become popular. In this work, we also employ non-autoregressive architecture and VAE structure to achieve robustness against luwlquality data.
2.2 Low LOW-RESOURCE SPEECH SYNTHESTS
Supervised speech synthesis requires high-quality paired speech and text data for training, which are costly to attain, especially for low-resource languages. To broaden the application scope of TTS systems, several low-resource TTS models are developed, which only need few or even not any high-quality paired data. Instead, they use unpaired text and audio data to train TTS models in semi-supervised or unsupervised way, which are much straightforward and cheap to obtain.
Semi-supervised TTS. Ren al. (2019b) adopt back- translatnnntoonn and pre-training to leverage un- paired data, generating pseudo text/speech samples with ASR/TTS models and training them with the augmented data iteratively. However, as a proof of concept, Ren et al. (2019h) only verify the feasibility of semi-supervised TTS in single-speaker dataset. Later, LRSpeech (Xu et al., 2020) supports multi-speaker and noisy datasets and is closer to real application. However, these semi-supervised methods still require a few pairs of high-quality speech and text data, which are expensive for low-resource languages since they often need to be recorded in professional studios.
Unsupervised speech synthesis. Unsupervised speech synthesis does not any paired training data from the target speaker and language, which has attracted growing attention recently. As the earliest unsupervised TTS works, Liu al. (2022b) and Ni et al. (2022) both an unsupervised ASR model to transcribe the TTS speech data to pseudo text and train with the augmented data to build an unsupervised TTS system. However, they heavily rely on the unsupervised ASR technique, whose training procedure is very unstable and heavily relies on lexicons. Therefore, these methods difficult apply to other low-resource languages. Besides, when switching to a multi-speaker setup, the gap between supervised and these unsupervised TTS methods becomes larger than single- speaker setup (Liu et al., 2022b). recent ArXiv paper (Lian al., 2022) trains a non mon-paralled voice conversion model using unpaired speech data as the acoustic model and specific module to map the text sequence to the speech discrete representation sequence, but this module has to be trained with an external dataset with the same language in the unpaired dataset. Thus this method is hardly applicable to real low-resource language scenarios due to the difficulty of collecting such a large paired dataset in this language.
PROPOSED BAG OF TRICKS
Suppose we have an unpaired speech dataset Stow and a text dataset T Tiow in the target low-resource language Tooms together with a paired speech-text dataset Spich and Trich in another rich-resource
3

---

Published as a conference paper at ICLR 2023
language Lrich auxiliary supervised training data. We assume that Liow and Lrich share some common characters, such as Indonesian and French share some Latin alphabets. Stow is a multi- speaker speech dataset whose audio quality is extremely low, it is difficult to obtain enough single-speaker clean data for the low-resource language. As for the auxiliary training data, since there are many public speech audios available in rich-resource languages, we do not impose any restrictions on the quality of these speech data. Our method aims to train the TTS model in the language Liow using the above datasets. Besides, we need another clean speech dataset Sref to provide the target timbre in our voice conversion model and it can be part of Srich or Stow- In this section, we first describe the overall training pipeline. Then we introduce model designs and some tricks used in each stage of the pipeline.
3.1 OVERALL TRAINING PIPELINE
ASR
TTS
eteps
As shown in Figure 1, the training pipeline of our method consists of 3 stages: voice conversion, su- pervised warm-up training and unsupervised back- translation training. We put the detailed pseudo-code algorithm of our training pipeline in Appendix A and describe each stage in the following paragraphs.
MAE
ppecudo
Lpoeudo
AdvLossa
TTS
staps
ASR
t ~ Tow
t ~ Sime
Stage 1: Voice conversion. The low-resource speech dataset Stow contains many speakers and can be very noisy. We consider the variable and noisy information, e.g.. background noise, speaker timbre, accent and some specific prosody, as the text-independent information in speech. Although some variable information is essential for certain TTS tasks like emotional, expressive and person- alized TTS, would be an obstacle for unsuper- vised TTS. The core purpose of unsupervised TTS is to solve the information matching problem between two modalities, i.e., speech and text, which are ac- tually aligned by the pronunciation (or called con- tent). The variable information in speech may in- terfere with the crossmodal pronunciation informa- tion matching in the unsupervised training stage and make the model struggle to find aligned clues in the text for this variable information. Therefore intuitively, if we reduce the information gap be- tween speech and text, our TTS and ASR model can achieve crossmodal pronunciation information matching faster, and the unsupervised training process can then be stabilized. To this end, we apply the cross-lingual voice conversion as the first stage. We train the voice conversion model on the datasets Stow, Srich and clean dataset Sref providing the reference speaker timbre and can be in any language. Then we can normalize the variable and noisy speech information of audios in Stow and Spich using the voice conversion model and denote the generated datasets as Stow and S Srich
Tiow Low
Unsupervised TTS & ASR
Sine
Voice Conversion
Spich
initialize
Syef
Stick
Supervised TTS & ASR
Trich
Figure 1: The overall pipeline of our method. The top part of the figure shows the iterative mack-translation in our method. The modules marked in green with solid border are train- able and those in purple with dotted border are fixed. "XE" denotes cross-entropy loss; "MAE" and "AdvLoss" denote mean abso- lute error and adversarial loss.
Stage 2: Supervised warm-up training. It is still very difficult to directly train unsupervised TTS from scratch even though we have normalized the variable information in the speech dataset Stow- To warm up the models for next unsupervised training, we train sequeenceto-sequeeee-bbeee model, which is required in the following trkkslaasllt stage, and a non-autoregressive TTS model using the auxiliary paired dataset Srich in rich-resource language. This stage can provide a better initialization for the model since there exist certain commonalities between written and spoken formats in different languages'.
ASR
Stage 3: Unsupervised back-translation training. Back-translation, originating from neural ma- chine translation, is one of the most effective ways leverage monolingual data for translation. In unsupervised TTS, back-translation (Sennrich et al., 2016; He et al., 20 61;6; Ren et al., 2019b) lever- ages the dual nature (He et al., 2016; Qin, 2020) of TTS and ASR tasks and develops the capability of
For a low-resource language Trows it is usually not difficult to find a rich-resource language Torich which is close to Taww-
4

---

Published as a conference paper at ICLR 2023
M,
Disc
Disc
Pre-trained
Pixel-level Losses
GAN Loss
Pixel-Ievel Losses
GAN Loss
Trainable
Speech Decoder
Speech
Language ID
Decoder
Language ID
I.R
Train
Inference
Train
Inferenco:
Duration ASR
Content Encoder
Tiferrence
Posterior D
Prior D
KI.
DP
Posterior D2
Prior Dr
Speaker Encoder
Posterior Encoder
Normalizing Flow
HuBERT
Text Encoder
Posterior Encoder
Normalizing Flow
4 1()
Text Input
3p N(0
Mi
M2
(a) Voice conversion model
(b) TTS model
Figure 2: The voice conversion and TTS architecture in our method. In subfigure (a), "Disc" denotes discriminator; training, M1, M2 and M3 are all the same; in inference, Mj is the content provider mel-spectrogram from the source speaker and M2 is the target speaker's 's reference mel mel-spectrogram. In subfigure (b), "LR" and "DP" denote length regulator and duration predictor.
transforming text to speech (TTS) and speech to text (ASR). We transform speech sequence s into a text sequence Epscudo using the ASR model, and then train the TTS model on the transformed pair also train the ASR model the transformed pair (Specudo: t) generated by
(fpecudo s). Similarly, the TTS model. The back-translation has two training directions, and as the training directions shift, the accuracy of ASR and performance of TTS can be boosted iteratively. We show the performance improvements as the back-translation training progresses in Appendix D.I.
3.2
VARIATIONAL VOICE CONVERSION MODEL WITH ENHANCED PRIOR
The new voice conversion (VC) model in Stage 1 is aimed at normalizing the variance and noisy information in low-quality audios. It is based on self-supervised learning (SSL.) audio representa- tion (Polyak al., 2021; van Niekerk et al., 2022), which has been proved to very effective in disentangling the content and timbre information. As shown in Figure 2a, the overall architecture of our VC model is like an autoencoder. In training, the mel-spectrogram M1 and M2, which are the same here, are fed into several information extraction modules to generate disentangled rep- resentations. Specifically, 1) the pre-trat pre-trained speaker encoder extracts the sentence- speaker embedding: 2) the pre-trate trained HuBERT (Hsu al., 2021) extracts the frame-level SSL discrete rep- resentations containing content (pronunciation) information; and 3) the posterior encoder extracts the residual information. After the information decomposition, the speech decoder takes all the rep- resentations as input and reconstructs the mel-spectrogram using mean absolute eттог (MAE) and multi-length adversarial loss (Lodv) following Ye al. (2022) and Chen al. (2020). inference, we replace Mj with the reference speech which provides the target speaker timbre. In this way, the generated speech can preserve the pronunciation information in M2 and transfer its timbre to Mj.
However, besides the common merits of general VC including preserving content and converting timbre, our model should also have the below properties to ensure its performance in our pipeline. 1) Our model should be cross-lingual. It should be able to normalize Stow and Srich which are in different languages.
Our model should generate high-quality results. As the normalized speech will be fed to the next two stages the training target, its result would bound that of the whole pipeline. 3) Our model should be robust to noisy and low-quality audio. Upon previous SSL-based VC methods, we enable these properties of our model with several improvements:
Multilingual HuBERT. To enable the model to be cross-lingual, we employ a multilingual Hu- BERT (Lee et al., 2021; Popuri et al., 2022) to extract the SSL discrete representations the content information, which is pre-trained on speech in multiple languages. We find it also generalizes well to other unseen languages (see Appendix D.2). HuBERT does not need any paired data speaker
5

---

Published as a conference paper at ICLR 2023
information, which is consistent with our task setting. Besides, we add a language ID input to the speech decoder to indicate the language we need to generate, which can accurately model the pro- nunciation differences among different languages given the same discrete representation, and make up for the limited capacity of multilingual representations.
Variational encoder with flow-based enhanced prior. To improve model robustness and audio quality, inspired by previous successful work in TTS (Ren et al., 2021b; Kim et al., 2021a), we introduce a variational encoder with flow-based enhanced prior. In training, this encoder can "store" the residual information, e.g., some irregular noises, time-varying timbre and prosody, that cannot be encoded by other information extraction modules to the posterior distribution Dy, and uses a normalizing flow to reshape the prior distribution Dp which needs to be close to Dg in terms of kl.-divergence. With normalizing flows, the KI.-divergence no longer offers a simple closed-form solution. So we estimate KI Kl-ddivegeenc via Monte-Carlo method as in Ren et al. (2021b) and Kim al. (2021a). The reason why we need the normalizing flow that simple Gaussian prior distribution results in strong constraints on the posterior, which pushes the posterior distribution towards the mean and limits diversity, while the distribution shaped by normalizing flows is more flexible and provides the decoder with stronger prior. Besides, it also provide the sampled random variables with temporal dependency.
Information bottleneck. HuBERT representation is kind of fow-bitrate representation for speech content and does not contain much non-lexical information such as speaker identity and emotion. still cannot ensure it fully
However, due to its discrete space bottleneck and way of training, disentangles the timbre information, and the remaining timbre information may degrade the voice conversion quality in the inference stage. To further erase the speaker identity information, we need to choose an appropriate input dimension for the content encoder (i.e., embedding layer for HuBERT tokens), which can neither be too large nor too small. A large dimension may lead to leakage of fine tine-grained identity information from the content encoder and a small one may result in loss of pronunciation information. We put more details of our VC model in Appendix B.
3.3 TTS AND ASR MODELS
Previous unsupervised TTS works use an autoregressive (AR) TTS architecture such as Tacotron 2 (Shen et al., 2018) and TransformerTTS (Li al., 2018), which automatically find the speech- text alignment. However, such an AR TTS architecture is not robust and prone to word missing and repeating problems as stated in Ren et al. (2019a). In this work, as shown in Figure 2b, we adopt a non-autoregressive (NAR) TTS architecture (Ren et al., 2019a; 2021a). We mainly follow PortaSpeech (Ren et al., 2021b), except that we replace the post-net in PortaSpeech with multi- length adversarial training (Ye et al., 2022; Chen et al., 2020) to simplify the training pipeline while keeping the naturalness of the generated mel-spectroggamm Instead of obtaining the ground-truth duration information from Montreal Forced Aligner (MFA) (McAuliffe et al., 2017) as many non- autoregressive TTS models (Ren et al., a0; Ye et al., 2022) do, we extract the speech-text align- ment from the attention matrix generated by the ASR model, which simplifies the training pipeline in our teck-rranslttoon stage and removes the dependency upon external tools. Specifically, inspired by GlowTTS (Kim et al., 2020), we extract the speech-text alignment by finding the monotonic path of maximum probability over the attention matrix of our ASR model using the Viterbi decoding. To enable the TTS model generate speech in a different language, we add language embedding to the decoder and an extra language ID input is required to specify the language of the target speech.
architecture with an LSTM-based encoder and decoder. To generate more monotonic aligmment for TTS training, we employ the location- sensitive attention (Shen al., 2018). Different from the TTS model, our ASR model is universal to all languages and does not need any language embedding as input, which can generalize to new languages better in our scenario. We put more details and model configurations of TTS and ASR models in Appendix B.2 and B.3.
Our ASR model is based on requence-to-sequee
3.4 TRICKS IN BACK- TRANSLATION
is very critical step for unsupervised TTS training to leverage unpaired speech and text data. In this subsection, we describe some trasslattlttnnn strategies that can significantly improve the effectiveness and efficiency of unsupervised TTS training.

---

Published as a conference paper at ICLR 2023
Curriculum learning. After warming up the ASR model in Stage 2 using Srich and Trich, can force the ASR model to transcribe the audio in Liow to the text in Lrich by initializing the language embedding of Liow with that of Lrick- Considering the results of ASR are taken as the input TTS. we select some good transcriptions, whose pronunciation is very similar to the ground- truth, for TTS training and discard bad ones in each round of back-translation. Apparently, we cannot directly calculate the eттоr rate between the transcription and ground-trui -ruud text since we have no corresponding text for each audio. Therefore, to filter good recognition results during iterative back-translation, we design a metric called focus rate (FF to evaluate the confidence of ASR results, which is defined as J t E A... In its definition, N denotes the number of mel-spectroogram frames; As,j is ASR attention weights at the position of the i-th -ee-spectro frame and the j-th text token and satisfies Σ A, = 1; P. is the text token index corresponding to the i-th mel-spectrogram frame in the monotonic path of maximum probability decoded by the Viterbi algorithm (Forney, 1973). A higher FF means greater probability lies in the decoded monotonic path and implies better speech-text monotonic alignment and higher confidence in the transcribed results. In each round of the pseudo text generation process, use F to select good ASR results whose JF is greater than a fixed threshold Fihree- Besides, we also store F for each pseudo text Epscudo and replace Epscdd with new result in the next mack-translation round only if F is increased.
Length augmentation. At the beginning of training in the low-resource language, short utterances (text and speech) are easier for TTS and ASR models to fit and they are generally better at generating short utterances rather than long ones. Besides, our curriculum learning strategy approximates the quality of the generated text and filters bad results, forcing our model to keep more short utterances than long ones. Consequently, our model becomes biased towards short utterances and may perform very poorly for long sentences. To fix this issue, we introduce a length augmentation strategy. In particular, we randomly concatenate two utterances (t', t Mss" s²) and their generated results M(tpscudo
(s) psccudo "pscudo M(t pscudo> (psccudo) with some probability Pear and obtain the generated pairs (t gcat
"pscudo) and (sta, ( (peeuato) for back-translation training. Length augmentation helps the TTS and ASR models generate long sentences better and become more robust to some long text inputs in inference.
Auxiliary supervised losses. If we only employ the fack-translation loss in Stage 3, the model may fail to find the correct speech-text alignment, leading to unsatisfied results and unstable training, especially the beginning of training. To solve this problem, apart from the back-translation loss in the target low-resource language, we also keep the supervised training losses in the auxiliary rich-resource language the same as those in Stage 2. We call them "auxiliary supervised losses". Specifically, in the process of training, we intersperse the rich resource language supervised training, for both TTS and ASR, into the back-translation steps with some probability Pauz-
4 EXPERIMENTS AND RESULTS
In this section, we conduct experiments to evaluate the effectiveness of our proposed method for unsupervised TTS. We first describe the experiment settings, show the results of our method, and conduct some analyses of our method.
4.1 EXPERIMENTAL SETUP
Datasets. We choose the speech and text data from Common Voice dataset (Ardila et al., 2019) for training and English and Indonesian as the target low-resource languages'. We use French as the rich-resource language unless otherwise stated. The experimental results of using other languages as the rich-resource language put in Appendix D.2. We split the target language data into two halves. We take unpaired speech data from the first half and text data from the second, so as to guar- antee the speech and text data are disjoint. We use LJSpeech (Ito, 2017) as the Sref provide the speaker timbre and suppress the background noise for the voice conversion model. For evaluation, we choose 100 audio/text pairs in LJSpeech for English" and 100 audio/text pairs in Common Voice (Indonesian subset) for Indonesian. For the speech data, we convert the raw waveform into mel-
A We choose English as one of the target languages since can understand English and it is easy evaluate, although English is not low-resource language. Wee use LJSpeech hecause it has fewer errors in text and speech pairing, while the data in Common Voice are very noisy and have much wrongly labeled text.
7

---

Published as a conference paper at ICLR 2023
Table 2: The comparison between our method and other existing unsupervised TTS methods. Liu et al. (2022b) and Ni et al. (2022) are same in our settings.
English
Indonesian
Methods
MOS
CER
WER
MOS
CER
WER
Supervised
4.11:10.07
0.016
0.052
4.07.10.08
0.024
0.068
Ren et al. (2019b)
3.32.10.12
0.393
0,684
3.25:10.11
0.392
0.702
et al. (2020)
3.42:10.10
0.376
0.645
3.39:10.13
0.389
0.695
Liu et al. (2022b); Ni et al. (2022)
3.49:10.11
0.305
0.555
3.52:1:0.09
0.299
0.536
Ours
3.82:40.09
0.145
0.320
3.98:+0.10
0.034
0.083
spectrograms with 80 ms frame size, 20 ms frame hop following Hsu et al. (2021). More details listed Appendix C.I.
Training and evaluation. We train our VC, TTS, and ASR models NVIDIA A100 GPU witch batch size 128. We use the Adam optimizer with 31 = 0.9, 82 0.98, é = 10-9 and learning rate 2e-4. The training takes nearly days. The output mel-spectrograms using a HiFi-GAN (Kong et al., 2020) pre-trained on LJSpeech (Ito, 2017). The focus rate Firess Nateps Peat and Paus in mack-translation are set to 0.2, 20k, 0.2 and 0.2. For evaluation, we mainly use MOS (mean opinion score) for audio quality, WER (word eтror rate), and CER (character error rate) for the intelligibility of the voice (French & Steinberg, 1947) verify if we can generate a reasonable speech sequence. For mean opinion score evaluation, we keep the text content consis- tent among different models so as to exclude other interference factors and only examine the audio quality. We randomly choose 20 sentences from the test set and each audio is listened by least 20 testers following Ren et al. (2019a; 2021a), who are all native English/Indonesian speakers. For WER and CER, we first transcribe the sentences from the generated speech using open-soureed or commercial ASR and calculate these metrics between them and the ground-truth text in the test set. We use WeNet (Yao al., 2021; Zhang et al., 2022) for English for fair comparison with future works, since commercial ASR could be changed in the future; but we choose Azure ASR service for Indonesian since we cannot find any Indonesian open-sourced ASR that accurate enough. In analytical experiments, we also show the CER of our ASR model, which also indicates the perfor- mance of our system since ASR and TTS are dependent on each other and boosted iteratively.
converted to waveform
4.2
RESULTS AND ANALYSES
4.2.1 PERFORMANCE
We compare our method with previous works including Ren al. (2019b), Xu et al. (2020), Liu et al. (2022b) and Ni et al. (2022). For fair comparison, we make some modifications to all baseline methods including unifying the training dataset, TTS acoustic model and vocoder (more detailed modifications of each baseline method are put in Appendix C.2). The results are shown in Table 2. We also evaluate the outputs of a supervised TTS model trained with paired target language data for reference, and its results can be regarded as the upper bound. From the table, it be seen that our method achieves the best performance in both speech quality (MOS) and intelligibility (CER and WER) in English and Indonesian. And very surprisingly, our method can performance of the supervised model in Indonesian. A possible reason is that Indonesian is easier to pronounce than English. These observations prove the effectiveness of our proposed tricks for unsupervised TTS.
approach the
4.2.2 ABLATION STUDY
To analyze the effectiveness of each trick and component, we conduct some ablation stud- ies on English. In addition to generated speech quality (MOS) and intelligibility (CER and WER), we also analyze the character error rate of our ASR model (CHR(ASK)). The results shown in Table 3. 1) From Row 2, it be seen that our model can achieve better performance after normalizing the speech variance including timbre and noise in our dataset.
speech-to-textt
8

---

Published as a conference paper at ICLR 2023
2) From Row 3, it be seen that the NAR TTS architecture improves speech quality and in- telligibility by a large margin, as NAR TTS is more robust to noisy speech and reduces some bad cases in generated speech. 3) From Row 4, it can be seen that back-trane transsatallti is essential to unsupervised TTS, which is consistent with the findings of previous works (Xu et al., 2020; Ren et al., 2019b). 4) From Row 5, we can see that curriculum learning can improve the training effectiveness since it can filter out bad pseudo transcripts and improve the training set quality for back-translation. 5) From Row 6, it can be seen that our length augmentation strategy can improve the robustness long text inputs in inference. 6) From Row 7, we find that auxiliary supervised training loss can improve the perfor- mance of both ASR and TTS by stabilizing the training. From the table, comparing other rows with Row 1 that shows our model with all tricks, we have several observations.
Table 3: Ablation studies on the components of our method. "Norm" denotes normalization via voice conversion model; "NAR" denotes non-autoregreesive TTS architecture; "CL" de- notes curriculum learning: "Aug" denotes the length augmenta- tion; "Aux" denotes the auxiliary supervised training loss; "BT" denotes back-translation.
No.
Settings
MOS
CER
WER
CER (ASR)
Ours
3.82
0.145
0.320
0.445
w/o. Norm
3.21110.12
0.431
0.789
0.684
w/o. NAR
3.18:40.09
0.496
0.756
0.627
w/o. BT
3.39:1.110
0.376
0.706
w/o. CL
3.72:10.08
0.225
0,493
0.481
w/o, Aug
3.73:10.09
0.163
0.358
0.479
7
w/o. Aux
3.70+00.09
0.227
0.472
0.575
4.2.3 ANALYSES ON VOICE CONVERSION MODEL.
With verified effectiveness of normalization via the voice con- version model as demonstrated in Section 4.2.2, we conduct more analyses on our proposed voice conversion model, includ- ing the effects of different infor- mation bottleneck channels and the flow-based enhanced prior. The results are shown in Table 4. It can be observed that an appro- priate size of bottleneck chan- nels is crucial for the perfor- 5
Table 4: The comparison between unsupervised TTS with others whose training speech is normalized with different voice conversion methods. "Var. Enc." represents our variational en- coder with flow-based enhanced prior; "Chn" denotes the content information bottleneck channels.
No.
Var. Enc.
Chn.
MOS
CER
WER
CER (ASR)
3.82-0.09
0.145
0.320
0.445
2
x
16
3.75=10.10
0.155
0.351
0.500
3.69=
0.191
0.384
0.496
3.79-+0.09
0.146
0.342
0.478
128
3.70-t0.11
0.199
0.396
0.507
mance of the voice conversion model, with large bottleneck resulting in timbre information leakage and small bottleneck lead- ing to pronunciation information loss. Besides, our flow-based enhanced prior can improve the quality of converted speech, since it has make fewer assumptions about the prior distribution mentioned in Section 3.2.
5 CONCLUSION
In this work, we proposed an unsupervised method for TTS by leveraging low-quality and noisy un- paired speech and text data in the target language and paired data in other rich-resource languages. Our method encloses several practical tricks to realize unsupervised seeeech, including nor- malizing variable and noisy information in speech data, curriculum learning, length augmentation, and auxiliary supervised training. We have also found that the non-autoregressive TTS architecture can significantly relieve robustness issues in unsupervised settings. We conducted experiments on Common Voice dataset, taking English and Indonesian as the target languages, and have found that method can achieve high audio quality in terms of MOS, and high intelligibility in terms of WER and CER, demonstrating remarkable effectiveness. Further analyses have well verified the importance of each trick of our method.

---

Published as a conference paper at ICLR 2023
REFERENCES
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common voice: A massivelymultilingee speech corpus. arXiv preprint 1112:::9122066700 2019.
Sercan Ömer Arik, Mike Chrzanowski, Adam Coates, Gregory Frederick Diamos, Andrew Gibian- sky, Yongguo Kang, Xian Li, John Miller, Andrew Y. Ng, Jonathan Raiman, Shubho Sengupta, and Mohammad Shoeybi. Deep voice: Real-time neural text-to-speech. In Proc. of ICML, Pro- ceedings of Machine Learning Research, 2017.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint 16X.::6607.04400 2016.
Alexei Baevski, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. Unsupervised speech recogni- tion. In Advances Neural Information Processing Systems, 2021.
Ond rej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Ne- gri. Aurelie Neveol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi. Karin Verspoor, and Marcos Zampieri. Findings of the 2016 con- ference on machine translation. In Proceedings of the First Conference on Machine Translation, pp. 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL, http::/ www wwwweb..bb.....................l org/ aaaaaattooooooooo ogy /ww1l//w//////////
w1 6 6/W1 2300
Jiawei Chen, Xu Tan, Jian Luan, Tao Qin, and Tie- Yan Liu. Hifisinger: Towards high-fidelity neural singing voice synthesis. arXiv preprint arXiv:2009.01776, 2020.
Jeff Donahue, Sander Dicleman, Mikolaj Binkowski, Erich Elsen, and Karen Simonyan. End-to-end adversarial text-to-speech. In ICLR, 2021.
David Forney. The viterbi algorithm. Proceedings of the IEEE, 61(3):268-22) 1973.
Norman R French and John C Steinberg, Factors governing the intelligibility of speech sounds. The journal of the Acoustical society of America, (1), 1947.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd international conference on Machine learning, pp. 369-376, 2006.
Di He, Yingee Xia, Tao Qin, Liwei Wang. Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma. Dual learn- ing for machine translation. In Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, 2016.
Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov, and Abdelrahman Mohamed. Hubert: Self-supervised speech representation learning by masked prediction of hidden units. IEEF/ACM Transactions on Audio, Speech, and Language Processing, 29:3451-3460, 2021.
Keith Ito. The lj speech dataset. httpss://Keeihhttc.cco///////ee/
2017.
Jachyeon Kim, Sungwon Kim, Jungil Kong, and Sungroh Yoon. Glow-tts: A generative flow for text-to-speech via monotonic alignment search. In NIPS, 2020.
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In International Conference on Machine Learning, pp- 5530-5540. PMLR, 2021a.
10

---

Published as a conference paper at ICLR 2023
Jachyeon Kim, Jungil Kong, and Juhee Son. Conditional variational autoencoder with adversarial learning for end-to-end text-to-speech. In ICML, 2021b.
Jungil Kong, Jachyeon Kim, and Jaekyoung Bae. Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis. In Advances Neural Information Processing Sys- tems 33: Annual Conference Neural Information Processing Systems 2020, NeurIPS 2020,
December 6-12, 2020, virtual, 2020.
Adrian Lańcucki. Fastpitch: Parallel text-to-speech with pitch prediction. ArXiv preprint, 2020.
Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, and Wei-Ning Hsu. Textless speech-to-speech trans- lation real data. arXiv preprint arXiv:21 2021.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, Ming Liu, and Ming Zhou. Close to human quality Its with transformer. ArXiv preprint, 2018.
Naihan Li, Shujie Liu, Yanqing Liu, Sheng Zhao, and Ming Liu. Neural speech synthesis with transformer network. In The ThirttyThird AAAI Conference on Artificial Intelligence, AAAI 2019, The Thirty-First Innovative Applications of Artificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Advances in Artificial Intelligence, EAA! 2019, Honolulu, Hawaii, USA, January 27- February 1, 2019, 2019.
Jiachen Lian, Chunlei Zhang, Gopala Krishna Anumanchipalli, and Dong Yu. Utts: Unsu- pervised tts with conditional disentangled sequential variational auto-encoder. arXiv preprint arXiv:2206.02512, 2022.
Alexander H. Liu, Tao Tu, Hung-yi Lee, and Lin-Shan Lee. Towards unsupervised speech recog- nition and synthesis with quantized speech representation learning. In 2020 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2020, Barcelona, Spain, May 4-8, 2020, 2020.
Alexander H Liu, Wei-Ning Hsu, Michael Auli, and Alexei Baevski. Towards end-to-end unsuper- vised speech recognition. ArXiv preprint, 2022a.
Alexander H Liu, Cheng-I Jeff Lai, Wei-Ning Hsu, Michael Auli, Alexei Baevskiv, and James Glass. Simple and effective unsupervised speech synthesis. ArXiv preprint, 2022b.
Jinglin Liu, Chengxi Li, Yi Ren, Feiyang Chen, Peng Liu, and Zhou Zhao. Diffsinger: Singing voice synthesis via shallow diffusion mechanism. ArXiv preprint, 2021.
Michael McAuliffe, Michaela Socolof, Sarah Mihuc, Michael Wagner, and Morgan Sondereg- ger. Montreal forced aligner: Trainable text-speech alignment using kaldi. In Proc. of INTER- SPEECH, 2017.
Huaiping Ming. Dong- Yan Huang. Lei Xie, Jie Wu, Minghui Dong, and Haizhou Li. Deep bidi- rectional LSTM modeling of timbre and prosody for emotional voice conversion. In Proc. of INTERSPEECH, 2016.
Junrui Ni, Liming Wang, Heting Gao, Kaizhi Qian, Yang Zhang, Shiyu Chang, and Mark Hasegawa- Johnson. Unsupervised text-to-speech synthesis by unsupervised automatic speech recognition. ArXiv preprint, 2022.
Kyubyong Park and Thomas Mule. Css10: A collection of single speaker speech datasets for 10 languages. arXiv preprint arXiv: 2019.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice Scaling text-to-speech with convolutional sequence learning, arXiv preprint arXiv:1710.076551 2017.
Wei Ping, Kainan Peng, Andrew Gibiansky, Sercan O. Arik, Ajay Kannan, Sharan Narang, Jonathan Raiman, and John Miller. Deep voice 3: 2000-speaker neural text-to-speech. In International Conference Learning Representations, 2018

---

Published as a conference paper at ICLR 2023
Adam Polyak, Yossi Adi, Jade Copet, Eugene Kharitonov, Kushal Lakhotia, Wei-Ning Hsu, Ab- deirahman Mohamed, and Emmanuel Dupoux. Speech resynthesis from discrete disentangled self-supervised representations. arXiv preprint 3XX::22044000355,0000 2021.
Sravya Popuri, Peng-Jen Chen, Changhan Wang, Juan Pino, Yossi Adi, Jiatao Gu, Wei-Ning Hsu, and Ann Lee. Enhanced direct speech-to-speech translation using self-supervised pre-training and data augmentation. arXiv preprint arXiv:2204.02967 2022.
Tao Qin. Dual Learning. Springer, 2020.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tin-Yan Liu. Fastspeech: Fast, robust and controllable text to speech. In Advances in Neural Information Processing Sys- 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, 2019a.
Yi Ren, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. Almost unsupervised text to speech and automatic speech recognition. In Proc. of ICML, Proceedings of Machine Learning Research, 2019b.
Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Yan Liu. Fastspeech 2:
Fast and high-quality end-to-end text to speech. In Proc. of ICLR, 2021a.
Yi Ren, Jinglin Liu, and Zhou Zhao. Portaspeech: Portable and high-quality generative text-to- speech. Advances in Neural Information Processing Systems, 34:13963-13974, 2021b.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proc. of ACI,, 2016.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representa- tions. In Proc. of NAACL-HLT, 2018.
Jonathan Shen, Ruoming Pang. Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RI-Skerrv Ryan, Rif A. Saurous, Yannis Agiomyrgian- nakis, and Yonghui Wu. Natural TTS synthesis by conditioning wavenet MEL spectrogram predictions. In 2018 JEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, 2018
Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara. Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP 2018, Calgary, AB, Canada, April 15-20, 2018, 2018
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. Listening while speaking: Speech chain by deep learning. In 2017 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). IEEE, 2017.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing in- gredient for fast stylization. arXiv preprint arXiv: 1607.08022 2016.
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. In 9th ISCA Speech Synthesis Workshop.
Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW, 2016.
Benjamin van Niekerk, Mare-André Carbonneau, Julian Zaïdi, Matthew Baas, Hugo Seuté, and Her- man Kamper. A comparison of discrete and soft speech units for improved voice conversion. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 6562-6566. IEEF, 2022.
12

---

Published as a conference paper at ICLR 2023
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances Neural Infor- mation Processing Systems 30: Anmual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, 2017.
Changhan Wang, Morgane Riviere, Ann Lee, Anne Wu, Chaitanya Talnikar, Daniel Haziza, Mary Williamson, Juan Pino, and Emmanuel Dupoux. Voxpopuli: large-scale multilingual speech corpus for representation learning, semi-supervised learning and interpretation. arXiv preprint arXiv:2101.003390 2021.
Yuxuan Wang, R. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Zhifeng Chen, Samy Bengio, Quoe V. Le, Yannis Agiomyrgian- nakis. Rob Clark, and Rif A. Saurous. Tacotron: Towards end-to-end speech synthesis. In Proc. of INTERSPEECH, 2017.
Jin Xu, Xu Tan, Yi Ren, Tao Qin, Jian Li, Sheng Zhao, and Yan Liu. Lrspeech: Extremely low- resource speech synthesis and recognition. In KDD '20: The 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Virtual Event, CA. USA, August 23-27, 2020, 2020.
Zhuoyuan Yao, Di Wu, Xiong Wang, Binbin Zhang, Fan Yu, Chao Yang. Zhendong Peng, Xiaoyu Chen, Lei Xie, and Xin Lei. Wenet: Production oriented streaming and non-streaming end-to-end speech recognition toolkit. Proc. Interspeech, Brn, Czech Republic, 2021. IEEE.
Zhenhui Ye, Zhou Zhao, Yi Ren, and Fei Wu. Syntaspeech: Syntax-aware generative adversarial text-to-speech. arXiv preprint arXiv:2204.1 1792, 2022.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron J. Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. Libritts: A corpus derived from librispeech for text-to-speech. In Proc. of INTERSPEECH, 2019.
Binbin Zhang. Di Wu, Zhendong Peng, Xingehen Song, Zhuoyuan Yao, Hang Lv, Lei Xie, Chao Yang, Fuping Pan, and Jianwei Niu. Wenet 2.0: More productive end-to-end speech recognition toolkit. arXiv preprint arXiv:2203. 2022.
13

---

Published as a conference paper at ICLR 2023
Appendices
A TRAINING ALGORITHM
The detailed unsupervised training algorithm is shown in Algorithm 1.
Algorithm Unsupervised TTS Training
I: Input:
paired dataset in rich-resource language Sich and Tia: unpaired speech and text data in low- resource language Skow and Tios' single-speaker speech dataset See containing reference speaker for voice conversion; pre-trained multilingual HuBERT model Ma; pre-trained speaker encoder Mak.
Initialize: multilingual TTS model MYIS and ASR model MASKi current unsupervised training step = 0; total unsupervised training steps Totat: number of steps for each TTS or ASR stage Natep-
3: Train our proposed voice conversion model with Sac. Swew and Sref, and use the M, and Ma to extract HuBERT and speaker representations.
Convert timbre all speech samples in Sech Saw to that of the speech Siet and obtain converted Sia and Skow
{Sec. 3.2]
5: Train MASR and MrTs using Sia and Tras-
6: repeat
7: Convert all samples in Show to pseudo text Epucudo- Select pseudo training pairs (fpsedos 8) satisfying F Fihres and obtain (T peedds. -S").
(Curriculum learning in Sec. 3.4)
9: for N in 0 to Naeps do
10:
if Random() S Preau: then
Sample D (Lrch, 8) from (Tich, Skun)- else
(Auxiliary Sec. 3.4]
12:
13:
Sample D + (f.pccdo: 8) from (Tepuudo, S*).
14:
if Random() Pont then
15:
Sample D' (Ipsentos 8²) from (T pootto S¹). D) (Concat(tpacuu pocudoo (poudo Concat(s, 8²)) end end
16:
{Length augmentation in Sec. 3.4}
18:
19:
Train Mrrs using D.
20:
end for
21:
for 2 in O Naps do
22:
Convert samples Too to pseudo speech Specudo and obtain (Spacda Tiow)) if Randomm) S Paur then
23:
24:
Sample D +- (#rich f) from (Si Tric)-
(Auxiliary loss Sec. 3.4}
25:
else
26:
Sample D (Spaauda: from (Speudoy Tiow).
27:
if RRnddmo) Peut then
28:
Sample D + (sprestos B)) from (Spensor, T Tom.) end end if
29:
+- (Concat Specda, spicud)). ) Concat(t, (2)
(Length augmentation Sec. 3.4]
30:
31:
32:
Train MASR using D.
33:
end for
34:
+- ++1
35:
until > Trotal
B MODEL DETAILS AND CONFIGURATIONS
In this section, we put more details of models including voice conversion (VC), text-to-speech (TTS), and automatic speech recognition (ASR) models, and also the hyper-parameters used in our experiments.
B.
VC MODEL
Our proposed VC model takes two -eeesppctss (content provider M, and timbre reference M2) inputs and outputs the converted mel-spectrogram Mj. Firstly, M, is fed into tre-trained
14

---

Published as a conference paper at ICLR 2023
speaker encoder' to extract the speaker embedding Hap.- Secondly, M2 is fed into pre-trained multilingual HuBERT", which is pre-trained with three languages, and extract the HuBERT discrete frame-level representation Hing. Thirdly, M2 is taken to the posterior encoder, which generates multivariate Gaussian distribution as the posterior in our variational VC model. Instead of di- rectly employing Gaussian distribution, we introduce a small volume-preserviigg normalizing flow model the prior distribution. A latent z is sampled from the posterior distribution (in training) or prior distribution (in inference). Finally, we add Hapk. Hiing z and language embedding of M2 together (all of them have the same channel size C= 192) and feed the result hidden states into the speech decoder to generate the target speech. Besides, we introduce mulli--ength discriminator to distinguish between the output generated by the model and the ground truth neel-ppectrogram.
The loss terms of the voice conversion model consist of 1) reconstruction loss of mel-spectrotram IMAE: mean absolute error between the generated and ground-truth mel-spectogram; 2) the KL- divergence of prior and posterior distributions: L = log (z( log (z(z), where z ~ go (z(x;; and 3) the adversarial training loss introduced by the multi-length discriminator: Ladv. The final weighted total loss is Lital X1 LMAE + A2 LKI. + La In our experiments, we set À A2
A3 = 1.0.
The detailed structure of each module is introduced in the following subsubscctions.
B.1.1 MULTILINGUAL HUBERT
Multilingual HuBERT (Lee et al., 2021) is trained on English (En), Spanish (Es), and French (Fr) 100k subsets of the VoxPopuli dataset (Wang al., 2021). VoxPopuli contains unlabeled speech data for 23 languages, and Lee et al. (2021) use the 4.5k hrs of unlabeled speech for En, Es, and Fr, totaling 13.5k hours. We extract the HuBERT features from the 11-th layer of the third HuBERT model and discretize them using the pre-trained K-means model to obtain the discrete representations Hring-
B.1.2 POSTERIOR ENCODER
The structure of posterior encoder is similar with the encoder in the variational generator of Por- taSpeech (Ren et al., 202 10). which is composed of ID-convolution with stride 4 followed by ReLU activation (Glorot al., 2011) and layer normalization (Ba al., 2016), and non-causal WaveNet (Van Den Oord et al., 2016), as shown in Figure 3a. The number of encoder layers, WaveNet channel size and kernel size are 8, 192 and The outputs of posterior encoder is the parameters (H, and og) of the posterior distribution N ((p,, Ty) and the latent 2 is sampled from N ((1g Gg), whose latent size set to 32.
B.1.3 VOLUME- PRESERVING (VP) NORMALIZING FLOW
Following Kim et al. (2021b) and Ren et al. (2021b), the prior distribution generator since it does not need to consider the Jacobian term when calculating the data log-likelihood and is powerful enough for modeling the prior, as shown in Figure 3c. The normalizing flow transforms simple distributions (e.g.. Gaussian distribution) to complex distribu- tions through a series of K invertible mappings, which is a stack WaveNet (van den Oord et al.) residual blocks with dilation 1. Then we take the complex distributions as the prior of the speech decoder. When introducing normalizing flow-based enhanced prior, the optimization objective of the mel-spectrogram generator becomes:
polume-preserving normalizing flow as
log px) (xlx) (x (T.(g pp(2)) L(o, 0,0),
where é, θ and Ø denote the model parameters of the posterior encoder, speech decoder and the normalizing flow-based enhanced prior, respectively. Due to the introduction of normalizing flows, the KL term in Equation no longer offers a simple closed-form solution. So estimate the expectation w.r.t. go (z(x)) via Monte-Carle method by modifying the KL term:
(L(gg(z)x)) pa (z)) = Eg x) [log qs (z|x) log pz)))
(2)
Shttps com http:://gg....coooooooooooeeeeer
ob/m wtt.oo/mmmaaanaaeaaaeeeeeeeeeeee /t/e//ppepppp///m mesaample/// lespmpech///se
to co__ppechh/docc/teettlt
md
15

---

Published as a conference paper at ICLR 2023
(H Tg)
,hg 0g
TConv1D++ReLU+LN
Non-causal WaveNet
H Hing Null
Stride=4
Non-causal WaveNet
Hing Null
Conv1D+ReLUHAe
Stride=4
(a) Posterior encoder. the voice conversion model, the condition is "Null", TTS model, the condition is linguistic fcature Hing-
(b) Speech decoder.
Zpoet
gorior
[a + 21, x2
D++1
Layer Norm
Projection
Conv1D
Non-causal WaveNet
Residual Coupling Layer
Nuil
Layer Norm
24 [21, #2
Multi-head Self-Attention
Ffow
N(0,1) 1)
(c) normalizing flow.
(d) Content encoder layers.
Figure 3: Modules in our voice conversion model.
In training, the posterior distribution N(,,, o.)) is encoded by the encoder of the posterior encoder. Then z is sampled from the posterior distribution using reparameterization and is passed to the speech decoder. In the meanwhile, the posterior distribution is fed into the VP normalizing flow to convert to a standard normal distribution (the middle dotted line). inference. VP normalizing flow converts a sample in the standard normal distribution into a sample and we pass the to the speech decoder.
Our VP normalizing flow consists of 4 flow steps, each of which has 4 WaveNet layers, whose channel size and kernel size are set 64 and
B.1.4 CONTENT ENCODER
The content encoder is stacks of feed-forward Transformer (Vaswani al., 2017) layers with relative position encoding (Shaw et al., 2018), shown in Figure 3d. The information bottleneck is located in the first layer of the content encoder (the embedding layer of HuBERT tokens). We set the channel size of each embedding to 16 default.
B. SPEECH DECODER
The speech decoder, as shown in Figure 3b, consists of non-causal WaveNet and 1D transposed convolution with stride 4, also followed by ReLU and layer normalization. The number of decoder layers, WaveNet channel size and kerne size are 192 and 5.
16

---

Published as a conference paper at ICLR 2023
B.1.6 MULTI-LENGTH DISCRIMINATOR
True/False
Linear
Discriminator,
Instance
Norm
X
N
Conv1D
window
window 3
LeakyReLU
window 2
Dropout
window;
(a) Multi Aulli-window clips in mel-spectrogram.
(b) Discriminator structure.
Figure 4: Multi-length Discriminator.
Inspired by Ye et al. (2022), our multi-length discriminator is an ensemble of multiple CNN-based discriminators, which evaluate the mel-spectrogram based on random windows of different lengths, as shown Figure 4. In our experiments, we train three CNN-based discriminators which observe random mel-spectrogram clips with lengths of 32, 64, and 128 frames. The structure of the CNN- based discriminator is shown in Figure 4b. It consists of N+1 2D-convolutional layers, each of which is followed by Leaky ReLU activation and drop-out layer. The latter N convolutional layers are acditionally followed by an instance normalization (Ulyanov et al., 2016) layer. After the con- volutional layers, linear layer projects the hidden states of the mel-spectrogram slice scalar, which is the prediction that the input mel-spectrogram is true or fake. In our experiments, we set N=2 and the channel size of these discriminators 32.
B.2 TTS MODEL
Our TTS model architecture follows PortaSpeech (Ren et al., 2021b) except that 1) we replace the post-net in PortaSpeech with multi- length adversarial training, which is the same the adversarial training in our voice conversion model in Appendix B.1. 2) We add a language embedding layer to the speech decoder, indicating the language of the speech that will be generated. 3) We use simple character encoder like FastSpeech (Ren et al., 2019a; 2021a) instead of the mixed linguistic encoders for simplicity. The detailed model architecture and syper-parametter of posterior encoder, normalizing flow, speech decoder and multi-length discriminators are the same as those in the voice conversion model in Appendix B.I. The structures of text encoder and duration predictor are the same as those in Ren et al. (2021b), with channel size 192, kernel size 5 and number of layers 4.
B.3 ASR MODEL
We adopt the architecture of Tacotron (Shen et al., 2018) for our ASR model, since its location- sensitive attention can generate close-to-diagonal and monotonic alignment between speech and text. We replace the character/phonmme embedding of the encoder Tacotron 2 with a speech CNN- based pre-net with stride 4 to enable speech information encoding. For the decoder side, we use the character embedding as the input layer and the softmax as the output layer to adapt the decoder the character sequence. We set the hidden size of encoder RNN to 512 and the number of convolution stacks to 5; the hidden size of decoder RNN and attention RNN are both set to 1024; the channel of decoder attention is set to 512. We also train model and jointly beam search decode the recognition results using the ASR model PASK( (X SS) and language model (LM(X) to maximize the probability logpASK (X (X[S) ALM (X), where ALM is the weight of language model and it is set to 0.2 this work. The beam size of beam search decoding is set 5.
Transformer-basee (Vaswani et al., 2017) language
17

---

Published as a conference paper at ICLR 2023
C MORE EXPERIMENTAL DETAILS
In this section, we describe more experimental details for reproducibility.
C.I DATASETS
We select subsets of English, French and Indonesian from Common Voice (Ardila et al., 2019) dataset. We choose subsets of about 200k utterances for English and French and all data (about 20k utterances) for Indonesian. We randomly select 100 utterances in English and Indonesian for validation and another 100 utterances in Indonesian for testing. The test of English is randomly selected from LJSpeech (Ito, 2017). We split the target language data into two halves according to utterance ID: take unpaired speech data from those with odd ID and text data from those with even ID, as to guarantee the speech and text data are disjoint.
C.2 BASELINES
For fair comparison, we make some modifications to all baseline methods as follows:
We adopt training data consisting of paired French (as auxiliary rich-resource language) data and unpaired English/Indoneian (as target low-resource language) data. Specifically, for Ren et al. (2019h) and Xu et al. (2020), we warmup ASR and TTS in these methods us- ing rich-resource language data before back-translation for Liu et al. (2022b) and Ni al. (2022), we initialize the unsupervised ASR model using modified CTC loss (Graves et al., 2006) with rich-resource language data before unsupervised training and also initialize the TTS model using this data.
We directly use character sequence as TTS input without any lexicon and G2P tools.
We extract the speaker embeddings using the same pre-trained speaker encoder" and add them to the TTS model indicate the speaker information (timbre) since our dataset is multi-speaker.
We replace all baseline TTS models with NAR architecture the same as our method since AR architecture is very sensitive noisy audio and cannot produce any meaningful results in our settings.
We use the same voice conversion model (described in Section 3.2) to convert ground-truth audios and all baselines' outputs to the same person's timbre from Sref-
We the same vocoder, HiFi-GAN (Kong al., 2020), convert mel-sppettoggram the waveform.
D MORE EXPERIMENTAL RESULTS
D.1 PERFORMANCE CHANGES IN FACK-TRANSLATION TRAINING
To verify the accuracy of ASR and the performance TTS can be boosted iteratively as the training directions shift, plot the accuracy of TTS and ASR results in Figure 5. From the figure, see that with the training of the model (the training directions shift every Nateps 20000 steps), the error rates of ASR and TTS results gradually drop until convergence.
D.2 USE OTHER RICH-RESOURCE AUXILIARY LANGUAGES
We explore how different rich-resource auxiliary languages affect the target language's per- formance. addition to French, we use other languages including German, Dutch, Spanish, and Portuguese as the rich-resource auxiliary language to train our unsupervised TTS model. For fair comparison, we use the same training data size for all rich-resource languages (80k pairs speech- text subset each language from Voice, We choose English as the target low-resource language. The results are shown in Table 5. It can be seen that using German as the rich-resource
Phttps://gi thub. com resembl Resemb' yzer
18

---

Published as a conference paper at ICLR 2023
0.7
0.55
CERIASRI
0.3
0.6
CER
WER
0.50
0.5
0.2
0.4
0.45
50
100
150
200
250
100
200
100
200
Step
Step
Step
(a) CER of ASR results.
(b) CER of TTS results.
(c) WER of TTS results.
Figure Accuracy of unsupervised English TTS and ASR models.
language achieves the best performance. The possible reason is that the pronunciation distance be- tween English and German closer than other languages as they both belong to west Germanic languages. Though Dutch also belongs to west Germanic languages, it does not perform very well, which might due to its bad data quality. Then we combine data from all these languages and find that it achieves very strong results and outperforms others that use only Besides, we observe that our method performs very well not only in English, French and Spanish, which are used to pre-traie trained the multilingual HuBERT, but also in other unseen languages, which verifies the generalization of our voice conversion model and the whole unsupervised TTS pipeline.
auxiliary language.
Table 5: Performances of unsupervised English TTS and ASR models which are initialized with supervised training in different rich-resource auxiliary languages.
Auxiliary languages
CER
WER
CER (ASR)
French
0.195
0.377
0.475
German
0.113
0.264
0.370
Dutch
0.257
0.522
0.547
Spanish
0.207
0.423
0.539
Portuguese
0.225
0.468
0.540
All
0.068
0.168
0.288
D.3 ANALYSES FOCUS RATE FF
To verify the effectiveness of focus rate F we propose in Section 3.4, we calculate F and CER on English test set in model training process. We plot the curves to explore the correlation between them in Figure 6. From the figure we can see that the focus rate F is negatively related the indicator for filtering ASR
recognition accuracy, which means reasonable to transcriptions (higher F indicates lower CER).
0.220
0.20
CER
CER(ASR)
0.215
0.18
0.16
0.210
50
100
150
200
250
Step (k)
Figure 6: F and CER evaluated on English test set in the training process.
19

---

Published as a conference paper at ICLR 2023
Table 6: The results of our method trained with different text unpaired dataset.
Datasct
MOS
CER
WER
Supervised
4.11
0.016
0.052
Ours (Common Voice text)
3.82:110.09
0.145
0.320
Ours (WMT text)
3.74.10.08
0.181
0.380
D.4 USE OTHER TEXT UNPAIRED DATASET
We train our model using the audio data from the Common Voice English subset which is the same as the original version the paper and the text data from WMT16 (Bojar et al., 2016) English training set to make the domains of unpaired audio and text very different. We keep the test set the same as the original paper (LJSpeech subset). The results are shown Table 6. From the table, it can be seen that the performance drops bit (~0.036 and ~006 increasing in CER and WER) due to the domain gap between the text and speech unpaired data.
E POTENTIAL NEGATIVE SOCIETAL IMPACTS
Unsupervised TTS lowers the requirements for speech synthesis service deployment (only needs unpaired speech and text data) and synthesizes high-quality speech voice, which may cause unem- ployment for people with related occupations such as broadcasters and radio hosts. In addition, there the potential for harm from non-consensual voice cloning the generation of fake media and the voices of the speakers in the recordings might be overused than they expect.
20


Reviews:
[{"review_id": "Y8xgy2hSTj9", "reviewer": "Reviewer_G8SV", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper improves upon the unsupervised speech synthesis. The motivation of such work is to serve to low resource languages where the supervised approach might not be feasible due to lack of annotated data.\n\nThe paper proposes and implements several modifications (bag of tricks) to the existing methods to improve the performance. The tricks are: the variational normalization to tackle the noisy information; the curriculum learning; non-autoregressive TTS. \n\nThe main contribution of the paper is that they present a high-quality method for unsupervised TTS that allows training without the lexicon.  ", "summary_of_the_review": "In summary, while there are some minor issues with the experimentation and the writing, the paper is very clearly written and contains some important results. I am convinced that the paper is well suited for the conference. Accept.", "strength_and_weaknesses": "# Strengths\n\n- Despite the complexity of the methods for TTS, the paper is able to clearly explain prior and the proposed methods.\n- The motivation of the work is clear and introduced well\n- Judging by the samples, the model performs very well\n\n\n# Weaknesses\n\n- I found the ablation study unconvincing: the MOS error bars intersect", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "QUuKMHj7khj", "reviewer": "Reviewer_sG8y", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the important problem of learning text-to-speech synthesis from noisy unpaired data. To order to achieve this, the authors propose several practical tricks, including normalizing variable and noisy information in speech data, curriculum learning,\nlength augmentation and auxiliary supervised learning. Experimental results on public datasets confirm the effectiveness of the proposed tricks.", "summary_of_the_review": "Overall I think this is a good paper, that is likely to prove quite useful for the development of zero-shot or few-shot text-to-speech synthesis solution.\n", "strength_and_weaknesses": "Strength:\n\nThe main contributions are clearly stated and supported by the experiments.\n\nMajor works on the similar topic are widely covered and referenced.\n\nThe samples are quite convincing.\n\nEvaluation is thorough enough to support the arguments with in-depth ablation studies.\n\nWeaknesses:\n\nThe complicated training algorithm makes the proposed method hard to reproduce. It would be helpful if the authors can provide brief guidelines such as how to tune the hyper parameters. It seems that the performance of this system heavily depends on the performance of the self-supervised voice conversion. It is better to compare the proposed VC approach with SOTA VC models such as YourTTS, although YourTTS is trained with the need of text scripts.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KIL2jGUwhX", "reviewer": "Reviewer_oNy6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents an unsupervised TTS pipeline that is achieved by assembling several pre-trained models and various methods in a pipeline, along with a few new tricks, to achieve impressive unsupervised TTS results.\n\nThe main contributions of this paper is examining different aspects of the proposed pipeline and quantifying the contributions of each component, as well as showing that when properly integrated together, this pipeline can achieve impressive results on the setup chosen by the paper.", "summary_of_the_review": "This paper proposes a pipeline method to perform unsupervised TTS that does not rely on pseudo-labels produced by unsupervised ASR. However it does rely on several pre-trained models to achieve good results (as seen in ablations) which are pipelined together to build the final system. The main value of this paper, in my opinion, is showing how each component in the pipeline contributes to the final quality of the system and, together with released code for reproducibility, is certainly valuable to the community.", "strength_and_weaknesses": "Strengths:\n- The final results are very good, especially for Indonesian\n- There is an extensive ablation study showing the effectiveness of each component\n- There are some innovations, such as the focus rate F\n\nWeaknesses:\n- The method is quite complex, combining several pre-trained models together with complex techniques, though it may be justified given the results achieved\n- Many of these tricks can also be applied to unsupervised ASR (thus improving unsupervised TTS based on these methods), such as voice conversion, access to a high quality rich resource language close to the target language, etc, and these drive significant improvements in accuracy of this method, as shown in table 3. While the authors have made attempts to make use of these methods for the baselines (as described in appendix C2), it is not very clear how effective these adaptations were without any ablation studies, nor is it clear how much effort the authors put in to make sure the baselines were modified in a an optimal way.\n\nOther:\n- While using the focus rate F to judge the quality of back-translation results is interesting, is there any study on how well F maps to quality (e.g. using a paired dataset for evaluation of F)?\n- In the given setup, speech and text are from same domain (CommonVoice dataset). Does this method still work if text and speech are from different domains? e,g, conversational speech + wikipedia text\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster