Translatotron 2: Robust direct speech-to-speech translation
Authors: Ye Jia
Michelle Tadmor Ramanovich
Tal Remez
Roi Pomerantz
Venue: ICLR 2022 Submitted (2022)
Keywords: Speech-to-speech translation
voice transferring
end-to-end

Abstract: We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists of a speech encoder, a phoneme decoder, a mel-spectrogram synthesizer, and an attention module that connects all the previous three components. Experimental results suggest that Translatotron 2 outperforms the original Translatotron by a large margin in terms of translation quality and predicted speech naturalness, and drastically improves the robustness of the predicted speech by mitigating over-generation, such as babbling or long pause. We also propose a new method for retaining the source speaker's voice in the translated speech. The trained model is restricted to retain the source speaker's voice, but unlike the original Translatotron, it is not able to generate speech in a different speaker's voice, making the model more robust for production deployment, by mitigating potential misuse for creating spoofing audio artifacts. When the new method is used together with a simple concatenation-based data augmentation, the trained Translatotron 2 model is able to retain each speaker's voice for input with speaker turns.

Under review as a conference paper at ICLR 2022
TRANSLATOTRON 2:
ROBUST DIRECT SPEECH-TO-SPEECE TRANSLATION
Anonymous authors Paper under double-blind review
ABSTRACT
We present Translatotron 2, a neural direct speech-to-speech translation model that can be trained end-to-end. Translatotron 2 consists a speech encoder, a phoneme decoder, mel-spectrogram synthesizer, and an attention module that connects all the previous three components. Experimental results suggest that Translatotron 2 outperforms the original Translatotron by large margin in terms of translation quality and predicted speech naturalness, and drastically improves the robustness of the predicted speech by mitigating over-generation, such babbling or long pause. We also propose a new method for retaining the source speaker's s voice in the translated speech. The trained model restricted to retain the source speaker's 's voice, but unlike the original Translatotron, it is not able to generate speech a different speaker's voice, making the model more robust for production deployment, by mitigating potential misuse for creating spoofing audio artifacts. When the new method is used together with a simple concatenation based data augmentation, the trained Translatotron 2 model is able to retain each speaker's s voice for input with speaker turns.
INTRODUCTION
Speech-to -peeechto-speecc translation (S2ST) is highly beneficial for breaking down communication barriers between people not sharing a common language. Conventional S2ST systems are often composed of cascade of three components: automatic speech recognition (ASR), text-to-text machine translation
(MT), and text-to-ppeech (TTS) synthesis (Lavie et al. 1997 Wahlster 2000 Nakamura et al.) (0066). Very recently, direct speech-to-tes to-text translation (ST) is rapidly emerging. and has outperiormed the cascade of ASR and MT (Weiss et al.) 2017 Jia et al. [2199]] Di Gangi et al. 2019 McCarthy et al.)
2020. Wang et al. 2021b Ansari et al. [020 Anastasopoulos et al. | 2021p, which makes the cascade of ST and TTS as S2ST feasible (Jia et al., 2019b). However, works on direct S2ST is very limited.
Compared to cascaded systems, direct S2ST has the potential benefits of: 1) retaining paralinguistic and non-linguistic information during the translation, such as speaker's voice (Jia et al. 2019b), emotion and prosody: 2) working on languages without written form (Tjandra et al.) 2019 Zhang et al. 2021: Lee et al. a021a);; 3) reduced computational requirements and lower inference latency; 4) avoiding error compounding across sub-systems; 5) casier on handling contents that do not need to be translated, such names and proper nouns (Jia et al. 2019b).
Translatotron (Jia et al., 2019b) is the first model that able to directly translate speech in one language speech in another language. It is also able to retain the source speaker's voice in the translated speech. However, its translation quality underperforms a cascade baseline by a large margin, and its synthesized translated speech suffers from robustness issues, such as babbling and long pause, which is an issue shared with the Tacotron 2 TTS model (Shen et al. 2018 He et al. 2019 Zheng al., 2019; Battenberg et al. 22200] Shen et al. 2020). since they share the same attention- based approach for synthesizing speech.
In this work we present Translatotron 2. The main contributions include:
1. We propose Translatotron 2, a novel direct S2ST model that is able to be trained end-to-end;
2. We conduct experiments suggesting that Translatotron 2 significantly outperforms Trans- latotron, and is comparable to a cascade system, in terms of translation quality, speech naturalness, and speech robustness:

---

Under review as a conference paper at ICLR 2022
3. We propose new method for voice retention in S2ST without relying on any explicit speaker embedding or ID. The trained model is only able to retain the source speaker's 's voice but not able to generate speech in different speaker's s voice. making it more robust for production deployment by mitigating potential misuse for creating spoofing audio artifacts;
4. We propose a simple concatenation data augmentation, ConcatAug, to enable Translatotron to retain each speaker's voice when the input includes speaker turns;
5. We conduct experiment suggesting that Translatotron 2 is efficient on multilingual direct S2ST, in which obtaines translation quality very close to an ST baseline.
Audio samples from Translatotron 2 are available in the supplementary material.
2 RELATED WORKS
S2ST Until very recently, S2ST systems are typically composed of cascade of ASR, MT, and TTS components (Lavie et al. 1997 Wahlster 2000 Nakamura et al. 2006 ITU 2016). Translatotron (Jia et al. 2019h) is the first direct S2ST model, which is sequence-to-sequeee se-ueuuence model trained end- to-end in multi-ojjective task. It has shown reasonable translation quality and speech naturalness, but still underperformed a baseline of ST + TTS cascade by a large margin. It also demonstrated the capacity of retaining speaker's 's voice during the translation, by leveraging speaker encoder separately trained in a speaker verification task (Wan et al.) 2018 Dia et al.) 2018).
A few recent works proposed cascade S2ST systems using learned discrete representation as the intermediate representation instead of text or phoneme. Tjandra et al. (2019) introduced such an S2ST system that first translated the source speech into discrete representation of the target speech which was predicted from a separately trained VQ-VAE (Oord et al. 2017), and then used the VQ- VAE decoder predict the target speech spectrogram from the discrete representation. Zhang al. (202I) additionally trained the VQ-VAE jointly with a supervised phoneme recognition objective in different languages. [ee et al, (2021a) used a separately trained vocoder to directly predict waveform from the discrete representation without relying on spectrogram; for the best performance, this vocoder included a duration predictor and an upsampler, akin to a generative TTS model. All these works require multiple components being trained in multiple steps, but are not able to be trained end-to-end.
Kano et al. (202T) introduced an end-to-end S2ST model with a cascade of three auto-regressive decoders, and used pre-trained MT and TTS models teacher models to facilitate the training of the end-to-end model. requires pre-trained ASR, MT, and TTS models, and the end-to-end model itself has to be trained in multiple steps.
Unfortunately, despite that these recent works generated translated speech in novel ways without relying on TTS subsystems, most of these works (except for et al. (2019b)) focused only on the translation quality, but did not assess the perceptual quality (Wagner et al. 2019) of the translated speech (e.g. naturalness), which is critical to S2ST.
TTS Translatotron uses a decoder similar to Tacotron 2 (Shen et al., 2018 Jia et al. 2018), which is an attention-base auto-regressive decoder. Due to the flexibility of the attention mechanism, they both suffer from robustness issues such as over-generation. Recent TTS models such as PeetSp (Ren et al. 2019; 2021).
Tacotron (NAT) (Shen et al., 2020; Jia et al.] 2021) and Parallel Tacotron EElis et al.) 2021b a), demonstrate that replacing the attention module with duration- curation-based upsampler yields more robust synthesized speech, as quantitatively evaluated at a large scale in (Shen al. 2020). The synthesizer component in this work resembles these works.
Voice conversion and anti-spoofing The performance of voice conversion (VC) has progressed rapidly in the recent years, and is reaching a quality that is hard for automatic speaker verification (ASV) systems to detect (Yi et al. 2020). ASVspoof 2019 (Todisco et al.) 2019 Wang et al. 2020) found that it was challenging to detect spoof audios generated from et al. (2018), which uses the same speaker encoder-bace based approach as in the original Translatotron. Such progress poses concerns on related techniques being misused for creating spoofing artifacts. We designed Translatotron 2 with the motivation of avoiding such potential misuse.
2

---

Under review as a conference paper at ICLR 2022
spectrogram (English)
phoneme (English)
spectrogram (English)
Softmax
Conv
Synthesizer
Decoder
RNN
Altention
Upsampler
durations
Duration Predictor
Encoder
spectrogram (Spanish)
concal(hidden,, attention)
Translatation
reiierr Translalation
Figure 1: A Translatotron 2 model that translates Spanish speech into English speech.
TRANSLATOTRON
The motivation of the architectural design of Translatotron to improve on three weaknesses existing in the original Translatotron: 1) The valuable auxiliary supervision textual features is not fully utilized during training; 2) Long- sequence-to-longe -elloo--equuene modelling with attention is difficult because of the flexibility of the attention mechanism; 3) Attention- basodd speech generation known to suffer from robustness issues such as over-generation and under-generation.
The proposed Translatotron 2 model solves these problems by using architecture composed of source speech encoder, a target phoneme decoder, and a target mel-spectrogram synthesizer. These three components connected together by single attention module. See Figure for an illustration. The model is jointly trained with a speech-to-speech translation objective and a speech-to-phonneme translation objective. As the result, the auxiliary supervision is fully utilized, and the attention is used only for modeling shorter target phoneme sequence but is not directly involved in speech generation.
The following subsections describe the architecture of each components as used our main ex- periments. As shown in Appendix D using alternative architectures for these components do not significantly impact the performance of Translatotron 2.
3.1 ENCODER
The encoder of Translatotron takes mel-spectrogram of the source speech as the input, and produces hidden representation. We use Conformer (Gulati et al. 2020) the architecture for the encoder. It first subsamples the input mel-spectrogram with a convolutional layer, and then processes it with a stack of Conformer blocks. Fach Conformer block composed of a bred-forward layer, self- attention layer, convolution layer, and second feed-forward layer. SpecAugment (Park et al. 2019) applied at the training time.
3.2 DECODER
The autoregressive decoder is assisted with an attention module. It takes the encoder output the source values for the attention, and predicts phoneme sequences corresponding to the target translated speech. We stack of LSTM cells the decoder, along with multi-head attention (Vaswani jet al. 2017). The combination of the encoder, the decoder, and the attention module is similar to typical ST model, except that it predicts phonemes instead of subword tokens.

---

Under review as a conference paper at ICLR 2022
3.3 SYNTHESIZER
The synthesizer takes the hidden output from the decoder, well as the context output from the attention module as its input, and synthesizes the target seeesppcto.. It is similar to the decoders in typical neural TTS models, such as in (henn et al. (2018) 2020); Ren et al. (202I). The predicted mel-spectrogram can be converted into waveform using either an estimation algorithm such as Griffin neural vocoder.
& Lim (1984)
We experimented with a number of different non-attentive architectures for the synthesizer, including autoregressive ones and parallel ones, and found that autoregressive synthesizers worked best. We followed the architecture and the hyperparameters of the mel-spectrogram decoder from NAT (Shen) et al. 2020). It first predicts duration for each of its input elements using bidirectional LSTT stack, then upsamples the input sequence with Gaussian weights based on the predicted duration, and finally uses an auto-regressive LSTM stack to predict the target mel-spectrogram. The predicted mel- spectrogram is further refined with a residual convolutional post-net. This synthesizer is trained jointly with -pectroggamm reconstruction loss and duration prediction loss. Figure [b) illustrates the architecture of this synthesizer.
Unlike Shen et al. 2020), we do not use per-phoneme duration labels for training the duration predictor, nor do we apply the FVAE-based alignment. Instead, only an 1,2 loss on the total predicted duration of the entire sequence is used (i.c. the "naïve approach" unsupervised duration modelling in [Shen et al. (2020)). We anticipate that adopting these approaches could further improve the performance of Translatotron 2, which we leave as future work.
VOICE RETENTION
The original Translatotron (Jia et al. 2019b) demonstrated the capacity of being able to retain the source speaker's voice in the translated speech, by conditioning its decoder on a speaker embedding generated from separately trained speaker encoder. In fact, it is capable of generating the translated speech in a different speaker's voice, as long as clip of the target speaker's recording used as the reference audio to the speaker encoder, the embedding of the target speaker is directly available. While this is impressively powerful, it can potentially be misused for generating spoofing audio with arbitrary content, posing concern for production deployment.
In this work, we use a different approach for retaining the source speaker's 's voice in the translated speech. The trained model is restricted to retaining the source speaker's 's voice, but not able to generate speech in a different speaker's voice.
4.1 TRAINING TARGETS IN SOURCE SPEAKER'S 'S VOICE
To retain speakers' voices across translation, we train S2ST models on parallel utterances with the same speaker's voice on both sides. Such dataset with human recordings on both sides is extremely difficult to collect, because it requires a large number of fluent bilingual speakers. Instead, we use a TTS model that is capable of cross-lingual voice transferring to synthesize such training targets.
We modified the PnG NAT (Jia et al. 2021) TTS model by incorporating a separately trained speaker encoder (Wan et al. 2018 in the same way Jia et al. (2018), and trained it on the LibriTTS corpus (Zen et al. 2019). The result TTS model is capable of zero-shot voice transferring, but synthesizes in better quality and more robust than Jia et al. (211)). We used this model to synthesize target speech in the source speaker's voice in our experiments. Other TTS models capable of cross-lingual voice modelling, such as Zhang al. (209)) Chen al. (2019): Xin al. (2021). could also be utilized.
4.2 SPEAKER TURNS
Theoretically, because the target-spectrogram synthesizer in both Translatotron 2 and Translatotron are directly conditioned on the source- sprrtte-pectroramm encoder output, the encoder output may be capable of preserving voice information locally in together with linguistic information, and the decoders and synthesizers may be capable of utilizing such local information for translating linguistic information while preserving local voice information. As result, such direct S2ST models may be capable of
4

---

Under review as a conference paper at ICLR 2022
Table Datasets for experiments with single-speaker target.
Conversational
Fisher
Vo 2
(Jia 2019a
Post et al. 2013)
(Wagg 2021a)
Languages
es-ten
fr, de, Read, short-form
Domain
Read, short-form 16-48 kHz
Telephone conversation
Source sample rate
kHz
48 kHz
Utterance pairs
979k
120k
321k
Source hours
1,400
127
476
Target hours
619
96
296
Target synthesized by
Tacotron + Griffin-L.im
Parallel WaveNet
PnG NAT + WaveRNN
retaining each source speaker's voice on input with speaker turns. However, proper training data is required to enable such models learn so.
ConcatAug To enable direct S2ST models to preserve each speaker's voice for input with speaker turns, augmented the training data by randomly sampling pairs of training ex- amples and concatenating the source speech, the target speech, and the target phoneme se- quences into new training examples. The result new examples contain two speakers' voices both the source and the target speech, which enables the model to learn examples with speaker turns. See Figure 2 for an exam- ple of such concatenation and the prediction from Translatotron on it.
".Per hater Me paliee
75
Such augmentation does not only enable the model to learn voice retention on speaker turns, but also increases the diversity of the speech content as well as the complexity of the acoustic conditions in the training examples, which can further improve the translation qual- ity of the model, especially on small datasets (as shown in Section 5.1). Narayanan et al. (2019) uses similar augmentation more complicated fashion, for improving ASR performance on multi-speaker inputs.
Tiretedd
Figure 2: Sample mel-spectroggrms on input with speaker turns. The input concatenation of an utterance from male speaker followed by another utterance from a female speaker. Translatotron preserves the voices of each speaker in the translated speech.
in a
EXPERIMENTS
We conducted experiments on three datasets, including two multilingual--Enghnn dataset. All datatsets use TTS synthesized target speech with 24 kHz sample rate. The phonemes used only at training time were converted from the transcripts using a propri- etary G2P system. See Table T for the details of each dataset. We evaluated the translation quality, naturalness and robustness of the predicted speech, as well as speaker similarity for voice retention.
datasets and a
Unless described otherwise, in all the experiments, Translatotron models use 16-layer Conformer encoder with 144 dimension following Gulati et al. (2020), a 4-layer LSTM decoder, and a RNN- based synthesizer following Shen et al. (2020). A comprehensive table hyperparameters in available in Appendix A All models were implemented using the Lingvo framework (Shen et al.) 2019).
5.1 TRANSLATION QUALITY
The same two datasets from Jia et al. (2019b) were used for evaluating the translation quality of Translatotron 2 when predicts translated speech is in single female speaker's voice. In contrast to
5

---

Under review as a conference paper at ICLR 2022
Table S2ST performance for outputting in single speaker's voice. BLEU scores were computed with reference for the Conversational test set, and with 4 references for the Fisher test set.
Conversational
Fisher
BLEU
MOS
UDR (%)
BLEU
MOS
UDR (%)
Translatotron
55.6
4.21 1 0.06
0.16
37.0
3.98 0.08
0.07
ConcatAug
55.1
4.19 0.06
0.13
3.79 = 0.09
0.14
Translatotron
50.4
4.15
0.07
0.69
26.9
3.70 = 0.08
0.48
Cascade (ST + TTS)
58.8
4.31 + 0.06
0.21
43.3
4.04 = 0.08
0.13
Training target
81.9
3.37 + 0.09
0.43
88.6
3,95
0.07
0.07
Reported in Jia et al. 2019b:
Translatotron
42.7
4.08 + 0.06
25.6
3.69 0.07
Cascade (ST TTS)
48.7
4.32
0.05
41.4
4.09 = 0.06
Training target
74.7
3.71 + 0.06
85.3
3.96 = 0.06
Reported discrete representation-bust cascade systems: Zhang et al 2021 (trained w/o text)
9.4
Lee et at
2021a (trained w/ text)
37.2
Jia et al. (211b),, we did not augment the source speech to add background noise or reverberation, and consistently use 80-channel speettrcro as input and 128-channel spectroorn as output for all Translatotron models. The predicted -eeesppctro is converted into waveform using the Griffin-Lim algorithm (Griffin & Lim 1984).
Following Jia et al. (209b)), the translation quality is measured by BLEU on ASR transcribed text (lower case, punctuation marks). Because ASR makes errors, such BLEU be thought lower bound of the translation quality. We used an ASR model from Park et al. (2020), trained on LibriSpeech (Panayotov al. (2015) and LibriLight (Kahn al. (0200) corpora. We retrained the baseline Translatotron strictly following Dia et al. (2019b) and re- re-evaluated it with this ASR model.
As shown in Table 2 the translation quality from Translatotron 2 outperformed the original Transla- totron by large margin and got very close to a strong cascade system. Applying ConcatAug further improved the translation quality of Translatotron on the smaller Fisher dataset.
The original Translatotron uses the phoneme sequences from both the source and the target sides as auxiliary training tasks. Translatotron uses only the target phonemes, yet yield better translation quality than Translatotron especially on the smaller Fisher dataset, indicating it is more data efficient.
Manual error case analysis revealed high consistency between the target speech prediction and the target phoneme prediction, suggesting more headroom for improving translation quality lies in the encoder and the decoder of Transsltooronn 2, instead of the synthesizer. Potential approaches to take include utilizing beam search, self-supervised pre-training (Baevski al. [2020] Wang et al. (0221b. self-training (Park et al. 2020; Wang et al. 2021b), and weakly supervised data (Jia et al. 2019a).
5.2 SYNTHESIS NATURALNESS
The naturalness of the predicted speech is evaluated by subjective listening test, reporting 5-scale mean opinion scores (MOS) with 95% confidence interval on 1,000 randomly sampled predictions. WaveRNN-based neural vocoder (Kalchbrenner et al., 2018) was used for converting the mel- spectrograms predicted from S2ST models into waveforms.
The result reported in Table 2 The naturalness of the speech predicted from Translatotron significantly better than from the original Translatotron, and is getting close to cascade system, which uses Tacotron 2, one of the state-of-the-art TTS models, to synthesize the predicted text translation into speech.
Consistent with Jia et al.) (2019b), despite that the training targets in the Conversational dataset is synthesized with lower quality Griffin-Lim vocoder, the trained S2ST model is able to synthesize with significantly better naturalness than the training target when a higher quality neural vocoder is used inference time.

---

Under review as a conference paper at ICLR 2022
Table S2ST performance with voice retention using the approach in Section Speaker similarity MOS is evaluated between the synthesized English speech and the human Spanish recording. Note: 1) the BLEU evaluation set a subset of the same in Table 2) Training targets are human recordings in Jia et al. (2019b) and synthesized speech in this work.
BLEU
Naturalness (MOS)
Similarity (MOS)
Translatotron
57.3
3.24 0.08
2.33 + 0.08
Cug
56.8
2.94 0.08
2.12 + 0.07
Translatotron
48.5
2.55 0.09
2.30 = 0.07
ConcatAug Training target
51.3
2.76 0.09
2.19 = 0.07
81.3
3.40 0.08
2.55 = 0.07
Results fromJia al. 2019b):
Translatotron
36.2
3.15 0.08
1.85 1 0.06
Training target
59.9
4.10 + 0.06
5.3 ROBUSTNESS
We specifically evaluated the robustness issue of werrgeenraation in the predicted speech, such as babbling or long pause, measured by unaligned duration ratio (UDR) (Shen et al.) 2020) with 1-second threshold The ASR transcribed text is used for alignment, using a confidence islands-based forced alignment model (Chiu et al. 2018
The result is shown in Table On the Fisher set, the UDR from Translatotron 2 is about times lower than from the original Translatotron, and is about the same as the training target. On the Conversational set, the UDR from Translatotron 2 is more than 4 times lower than from the original Translatotron, and is even about 3 times lower than the training targets. Note that the training target in the Conversational set is synthesized by the Tacotron 2 TTS model (see Table T which by itself suffers from over- wver-geneaation (He et al.) 2019 Zheng et al., 2019; Battenberg et al., 2020 [hen et al., 2020). The result suggests that Translatotro 2 drastically improved robustness than the original Translatotron, and is also robust to small ratio of disfluency in the training targets.
5.4 VOICE RETENTION
To evaluate the ability of retaining speakers' voices while translating their speech from one language to another, we augmented the Conversational dataset by synthesizing the target speech using voice-transferring TTS model as described in Section 4.1 Examples with source speech shorter than I second were discarded for the stability of voice transferring. The result dataset contains parallel utterances with similar voices on both sides. S2ST models were trained on this dataset without any explicit conditioning on speaker embeddings or IDs (i.e. no speaker encoder for the original Translatotron). Translatotron 2 models used Conformer encoder with larger dimension (256) than in previous experiments, since its output was expected carry more acoustic information for voice retention. Following Jia et al. (2019b), we also reduced the synthesizer's is pre-net dimension to 16 to encourage to infer voice information from the encoder output instead of the teacher-forcing label.
5-scale subjective MOS on both naturalenss and speaker similarity was evaluated with 1,000 random samples or pairs of samples from the test se reported with 95% confidence interval. As Table shows, both Translatotron 2 and Translatotron performed well using the new approach for voice retention. They both obtained about the same speaker similarity MOS as the original Translatotron and significantly better translation quality. Translatotron 2 further outperformed Translatotron in terms of translation quality, which is consistent with the experimental results for translating into a single speaker's voice. It is worth to note that with the new approach, the speaker similarity from S2ST models capped by the same of the training targets, which by itself is low. This can be partially due the performance of the voice-transferring TTS model used for synthesizing the training targets, and partially due to the fact that cross-lingual speaker similarity evaluation is more challenging to raters
'Under sender-generationn (i.e. WDR from Shen al. 2020 does not apply hecause of the nature of translation. Related errors are reflected the BI EE evaluation.
"This dataset was created by crowd-sourcing without collecting speaker identities for privacy protection. The set (12.7 hours source recordings) may contain both speakers seen and unseen during training.
7

---

Under review as a conference paper at ICLR 2022
Table 4: Voice retention performance on speaker turns. The speaker similarity MOS between the leading/tralling 1.6-second segment from the predicted speech (English) and the entire 1st/2nd source speaker's speech (Spanish) is measured. (t / indicates that higher/lower values are better.)
1st source speaker
2nd source speaker
Leading seg. 1
Trailing seg.
Leading seg.
Trailing seg.
Translatotron 2
2.22 + 0.07
2.15 + 0.07
2.04 + 0.07
2.00 + 0.07
+ ConcatAug
2.44 + 0.07
1.82 1 0.07
1.76 + 0.07
2.51 + 0.08
Translatotron
1.87 + 0.06
1.90 1 0.07
2.06 1 0.07
2.05 1 0.07
ConcatAug
2.18 + 0.07
1.71 + 0.06
1.93 0.07
2.35 0.07
Training target
2.58 + 0.08
1.62 1 0.06
1.83 + 0.07
2.44 1 0.07
(some rating comments are purely based on language difference), as also observed in Zhang et al. (019). Obtaining better quality training targets, such as human recordings instead of synthesized speech, may further improve the performance of S2ST models trained with the new approach.
5.4.1 SPEAKER TURNS
Speaker similarity evaluation with speaker turns the entire model prediction is challenging because would require speaker diarization on the predicted speech, and the potential content re-ordering during translation as well as potential model prediction error adds extra difficulty. We approximated by considering the leading/trailing short segments in the predicted speech as corresponding to each of the two speakers inputs with single speaker turn.
The evaluation set was constructed by applying the same concatenation augmentation as described in Section 4.2] on the original test set, so that each utterance contains two speakers' voices. Examples with target speech shorter than 2 seconds before concatenation were discarded. We conducted subjective speaker similarity MOS evaluation between the two entire original utterances before concatenation and the leading/trailing 1.6-second segment of the predicted speech.
The results are shown in Table 4 can be seen that the impact of the concatenation augmentation consistent on Translatotron and Translatotron 2. When the concatenation augmentation was not used during training, for each source speaker, the similarity compared to the leading/trailing segment in the predicted speech was about the same; and for each segment in the predicted speech, the similarity compared to the first/second source speaker was also close. This suggests that the predicted speech was in single speaker's voice, which was optimized for both source speakers at the same time. When the concatenation augmentation was used, both models obtained significantly higher speaker similarity on matched pairs than mismatched pairs, indicating that the models successfully separated two speakers and retained voice for each of them respectively. It can also be seen that Translatotron 2 obtained significantly higher speaker similarity than Translatotron on matched pairs, indicating the effectiveness of Translatotron 2.
Such quantitative evaluation cannot reflect how the predicted speech transits from one speaker's voice into another speaker's 's voice. Listening to the audio samples verified that the voice change happened instantly without blurry rather than transitionally, suggesting that the encoder outputs preserved the voice information locally in together with the linguistic information, and the synthesizer and the decoders were able to utilize such local information for translating the linguistic while retaining the corresponding voice information locally. A sample of such speaker turn with the prediction from Translatotron 2 trained with concatenation augmentation is visualized in Figure 2
Although Translatotron 2 bears some similarity to cascade systems in terms of the cascade of the decoder and the synthesizer, such voice retention capacity, especially on speaker turns, is very challenging for the latter, as it would require separate speaker diarization and voice encoder.
While ConcatAug is effective on enabling S2ST models to support voice retention on speaker turns, and further improve the translation quality and the speech naturalness models with lower performance (e.g. trained on small datasets), it may negatively impact the speech naturalness and similarity models with strong performance, shown in Table 2 and Table 3 This may be
1.6 seconds is the minimum duration for computing d-vectors for the objective evaluation in Appendix B
8

---

Under review as a conference paper at ICLR 2022
Table 5: Multilingual X- S2ST performance on high-resource languages from CoVoST measured by BLEU on ASR transcribed text. The same checkpoints from each model were used for evaluating all language pairs. Note: BLEU scores are not directly comparable between S2ST and ST.
Source language
fr
de
Translatotron 2
27.0
18.8
27.7
22.5
Translatotron
18.9
10.8
18.8
13.9
ST WWan et al.
202
27.0
18.9
28.0
23.9
Training target
82.1
86.0
85.1
89.3
explained by the fact that the augmented utterances sound less natural and supporting speaker turns may sacrifice model capacity on single-speaker cases.
5.5 MULTILINGUAL S2ST
We also conducted experiments to evaluate the performance of multilingual En S2ST models. We trained both a Translatotron model and Translatotron model on the high-resource languages from the CoVoST 2 corpus (Wang et al. 202 using TTS synthesized target speech in single female speaker's 's voice. The original Common Voice (Ardila et al. 2020) data split was followed. The models not explicitly conditioned on language IDs. We modified the original Translatotron model to the same Conformer encoder and SpecAugment as in Translatotron 2 for fair comparison. Similarly, we only used the target phoneme sequence (in English) for both Translatotron and Translatotron 2. Translatotron used a decoder with 6-layer LSTM with 1024 dimension; Translatotron used decoder with 6-layer LSTM with 512 dimension and synthesizer of 2-layer LSTM with 1024 dimension. Both used the same convoluational post-net net as Shen et al. (2020). The total number of parameters Translatotron was about 10% fewer than in Translatotron.
The translation quality as measured by BLEU on the ASR transcribed text is shown in Table 5 The BLEU scores from Translatotron 2 significantly outperformed the same from Translatotron. Although the results are not directly comparable between S2ST and the close numbers suggest that Translatotron achieved translation quality comparable to the baseline ST model. This indicates that Translatotron 2 is also highly effective for multilingual S2ST.
CONCLUSION
We proposed Translatotron 2, neural direct S2ST model that can be trained end-to-end. The major differences compared to the original Translatotron are: 1) the output from the auxiliary target phoneme decoder is used as an input to the spectrogram synthesizer; 2) the spectrogram synthesizer is duration-based while still keeping the benefits of the attention mechanism. Experiments conducted on three different datasets, including multilingual S2ST, suggested that Translatotron outperformed the original Translatotron by large margin in terms of translation quality and predicted speech naturalness, and drastically improved the robustness of the predicted speech.
We also proposed new method for retaining the source speaker's voice in the translated speech. In contrast to the original Translatotron, S2ST models trained with the new method restricted to retain the source speaker's voice, but not able generate speech in different speaker's voice, which makes the model free from potential abuse such as creating spoofing audios, thus more robust for production deployment. When the new method is used together with a simple concatenation data augmentation, the trained Translatotron 2 model is able to retain each speaker's voice for input with speaker turns.
ETHICS STATEMENT
Preserving voices during S2ST is desired as it helps communication. However, progress on high quality voice cloning poses societal concerns on misuse, such as creating "deepfake" spoofing audios.
'BLELL .FU for S2ST is computed case-insensitively and without punctuation marks hecause the transcript is from the output ASR. On the other side, ASR errors in evaluation underestimates the performance of S2ST.

---

Under review as a conference paper at ICLR 2022
This work includes a new voice retention method for S2ST that reduces the potential for such misuse, such models are deployed production.
REPRODUCIBILITY STATEMENT
We conducted experiments multiple datasets, including public datasets. We also provided comprehensive list hyperparameters in Appendix A to help reproducing the experimental results.
REFERENCES
Antonios Anastasopoulos, Ondrej Bojar, Jacob Bremerman, Roldano Cattoni, Maha Elbayad, Mar- cello Federico, Xutai Ma, Satoshi Nakamura, Matteo Negri, Jan Nichues, et al. Findings of the IWSLT 2021 evaluation campaign. In International Conference on Spoken Language Translation (IWSLT), 2021.
Ebrahim Ansari, Amittai Axelrod, Nguyen Bach, Ondřej Bojar, Roldano Cattoni, Fahim Dalvi, Nadir Durrani, Marcello Federico, Christian Federmann, Jiatao Gu, et al. Findings of the IWSLT 2020 evaluation campaign. In International Conference Spoken Language Translation (IWSLT), 2020.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M Tyers, and Gregor Weber. Common Voice: A massively- multilingual speech corpus. In Proceedings of Language Resources and Evaluation Conference (LREC), 2020.
Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Eric Battenberg. RJ Skerry-Ryan Soroosh Mariooryad, Daisy Stanton, David Kao, Matt Shannon, and Tom Bagby. Location-relative attention mechanisms for robust long-form speech synthesis. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020.
Mengnan Chen, Minchuan Chen, Shuang Liang, Jun Ma, Lei Chen, Shaojun Wang, and Jing Xiao. Cross-lingual, Interspeech, 2019.
text-to text-to-speech synthesis using neural speaker embedding. In Proc.
Chung-Cheee Chiu, Anshuman Tripathi, Katherine Chou, Chris Co. Navdeep Jaitly, Diana Jaun- zeikare, Anjuli Kannan, Patrick Nguyen, Hasim Sak, Ananth Sankar, Justin Tansuwan, Nathan Wan, Yonghui Wu, and Xuedong Zhang. Speech recognition for medical conversations. In Proc. Interspeech, 2018.
Mattia A Di Gangi, Matteo Negri, and Marco Turchi. One-to-many multilingual end-to-end speech translation. IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019.
Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Skerry-Ryan and Yonghui Wu. Parallel Tacotron 2: A non-autoregressive neural TTS model with differentiable duration modeling. In Proc. Interspeech, 2021a.
Isaac Elias, Heiga Zen, Jonathan Shen, Yu Zhang, Ye Jia, Ron Weiss, and Yonghui Wu. Paral- lel Tacotron: Non-autoregressive and controllable TTS. In IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), 2021b.
Daniel Griffin and Jae Lim. Signal estimation from modified short-time Fourier transform. IEEE Transactions Acoustics, Speech, and Signal Processing, 32(2):236-22 243, 1984.
Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han. Shibo Wang, Zhengdong Zhang, Yonghui Wu, and Ruoming Pang. Conformer: Convolution-augmentee transformer for speech recognition. In Proc. Interspeech, 2020.
10

---

Under review as a conference paper at ICLR 2022
Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi In- aguma, Naoyuki Kamo, Chenda Li, Daniel GtariiaRoomets Jiatong Shi, et al. Recent developments on ESPnet toolkit boosted by Conformer. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021
Mutian He, Yan Deng, and Lei He. Robust sequence-to-ssqqeene acoustic modeling with stepwise monotonic attention for neural TTS. In Proc. Interspeech, 2019.
ITU. ITU-T E.75: Functional requirements for network-based speech-to-speech translation services, 2016. International Telecommunication Union.
Ye Jia, Yu Zhang, Ron J Weiss, Quan Wang, Jonathan Shen, Fei Ren, Zhifeng Chen, Patrick Nguyen, Ruoming Pang. Ignacio Lopez. Moreno, and Yonghui Wu. Transfer learning from speaker verification to multispeaker text-to-speech synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2018.
Ye Jia, Melvin Johnson, Wolfgang Macherey, Ron J Weiss, Yuan Cao, Chung- Chiu, Naveen Ari, Stella Laurenzo, and Yonghui Wu, Leveraging weakly supervised data to improve end-to-end speech-to-text translation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019a.
Ye Jia, Ron Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Zhifeng Chen, and Yonghui Wu. Direct speech-to-speech translation with sequence-to-seqqeeee model. Proc. Interspeech, 2019b.
Ye Jia, Heiga Zen, Jonathan Shen, Yu Zhang, and Yonghui Wu. PnG BERT: Augmented BERT on phonemes and graphemes for neural TTS. In Proc. Interspeech, 2021.
Jacob Kahn, Morgane Rivière, Weiyi Zheng, Evgeny Kharitonov, Qiantong Xu,
Mazaré, Julien Karadayi, Vitaliy Liptchinsky. Ronan Collobert, Christian Fuegen, Tatiana Likhoma- nenko, Gabriel Synnaeve, Armand Joulin, Abdelrahman Mohamed, and Emmanuel Dupoux. Libri- light: A benchmark for ASR with limited no supervision. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
Nal Kalchbrenner, Erich Elsen, Karen Simonyan, Seb Noury, Norman Casagrande, Edward Lockhart, Florian Stimberg, Aaron van den Oord, Sander Dieleman, and Koray Kavukcuoglu. Efficient neural audio synthesis. In Proceedings of International Conference on Machine Learning (ICML), 2018.
Takatomo Kano, Sakriani Sakti, and Satoshi Nakamura. Transformer-based direct speech-to-speech translation with transcoder. In IEEE Spoken Language Technology Workshop (SLT), 2021.
Alon Lavie, Alex Waibel, Lori Levin, Michael Finke, Donna Gates, Marsal Gavalda, Torsten Zeppenfeld, and Puming Zhan. JANUS-III: Speech-to-speech translation in multiple languages. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1997.
Ann Lee, Peng-Jen Chen, Changhan Wang, Jiatao Gu, Xutai Ma, Adam Polyak, Yossi Adi, Qing He, Yun Tang, Juan Pino, et al. Direct speech-to-speech translation with discrete units. arXiv preprint 10Xi:22077005044, 2021a.
Yoonhyung Lee, Joongbo Shin, and Kyomin Jung. Bidirectional variational inference for non- autoregressive text-to-speech. In International Conference on Learning Representations (ICLR), 2021b.
Arya D McCarthy, Liezl Puzon, and Juan Pino. SkinAugment: Auto-encoding speaker conversions for automatic speech translation. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020.
Satoshi Nakamura, Konstantin Markov, Hiromi Nakaiwa, Genichiro Kikui, Hisashi Kawai, Takatoshi Jitsuhiro, Jin-Song Zhang, Hirofumi Yamamoto, Eicchrro Sumita, and Seichhi Yamanoto. The ATR multilingual speech-to-speece translation system. IEEE Transactions on Audio, Speech, and Language Processing, 2006.
11

---

Under review as a conference paper at ICLR 2022
Arun Narayanan, Rohit Prabhavalkar, Chung- Chung-Cheng Chiu, David Rybach, Tara N Sainath, and Trevor Strohman. Recognizing long-fore form speech using streaming end-to-end models. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), 2019.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu, Neural discrete representation learning. In Advances Neural Information Processing Systems (NeurIPS), 2017.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. LibriSpeech: an ASR corpus based on public domain audio books. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
Daniel S Park, William Chan, Yu Zhang, Chung-Cheng Chiu, Barret Zoph. Ekin D Cubuk, and Quoc V Le. SpecAugment: A simple data augmentation method for automatic speech recognition. In Proc. Interspeech, 2019.
Daniel Park, Yu Zhang, Ye Jia, Wei Han, Chung-Chon Chiu, Bo Li, Yonghui Wu, and Quoe V Le. Improved noisy student training for automatic speech recognition. In Proc. Interspeech, 2020.
Matt Post, Gaurav Kumar, Adam Lopez, Damianos Karakos, Chris Callison-Burch, and Sanjeev Khudanpur. Improved speech-to-text translation with the Fisher and Callhome Spanish- Englishh speech translation corpus. In International Conference on Spoken Language Translation (IWSLT), 2013.
Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie- Yan Liu. FastSpeech: Fast, robust and controllable text to speech. In Advances Neural Information Processing Systems (NeurIPS), 2019.
Yi Ren, Chenxu Hu, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. FastSpeech 2: Fast and high-quality end-to-end text-to-speech. In International Conference on Learning Representations (ICLR), 2021.
Jonathan Shen, Ruoming Pang, Ron J. Weiss, Mike Schuster, Navdeep Jaitly, Zongheng Yang, Zhifeng Chen, Yu Zhang, Yuxuan Wang, RJ Skerrv-Ryan Rif A. Saurous, Yannis Agiomyrgiannakis. and Yonghui Wu. Natural TTS synthesis by conditioning WaveNet on Mel spectrogram predictions. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
Jonathan Shen, Patrick Nguyen, Yonghui Wu, Zhifeng Chen, et al. Lingvo: A modular and scalable 2019.
framework for sequence-to-sequencc modeling. arXiv preprint
Jonathan Shen, Ye Jia, Mike Chrzanowski, Yu Zhang, Isaac Elias, Heiga Zen, and Yonghui Wu. Non-Attentive Tacotron: Robust and controllable neural TTS synthesis including unsupervised duration modeling. arXiv preprint arXiv:2010.04301, 2020.
Andros Tjandra, Sakriani Sakti, and Satoshi Nakamura. speech-to-speech translation between untranscribed unknown languages. In IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). 2019.
Massimiliano Todisco. Xin Wang, Ville Vestman, Md Sahidullah, Héctor Delgado, Andreas Nautsch, Junichi Yamagishi, Nicholas Evans, Tomi Kinnunen, and Kong Aik Lee. ASVspoof 2019: Future horizons in spoofed and fake audio detection. In Proc. Interspeech, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeuriPS), 2017.
Petra Wagner, Jonas Beskow, Simon Betz, Jens Edlund, Joakim Gustafson, Gustav Eje Henter, Sébastien Le Maguer, Zoñia Malisz, Êva Székely, Christina Tännander, al. Speech synthesis eval- uation state-of-the-ar assessment and suggestion for novel research program. In Proceedings of the 10th Speech Synthesis Workshop (SSW10). 2019.
Wolfgang Wahlster. Verbmobil: Foundations of speech-to-sppech translation. Springer, 2000.
12

---

Under review as a conference paper at ICLR 2022
Li Wan, Quan Wang, Alan Papir, and Ignacio Lopez Moreno. Generalized end-to-end loss for speaker verification. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2018.
Changhan Wang, Anne Wu, and Juan Pino. CoVoST 2: A massively multilingual speech-to-text translation corpus. In Proc. Interspeech, 2021a.
Changhan Wang, Anne Wu, Juan Pino, Alexei Baevski, Michael Auli, and Alexis Conneau. Large- scale self-and semi-supervised learning for speech translation. In Proc. Interspeech, 2021b.
Xin Wang, Junichi Yamagishi, Massimiliano Todisco, Héctor Delgado, Andreas Nautsch, Nicholas Evans, Md Sahidullah, Ville Vestman, Tomi Kinnunen, Kong Aik Lee, et al. ASVspoof 2019: A large-scale public database of synthesized, converted and replayed speech. Computer Speech & Language, 64:101 114, 2020.
Ron Weiss, Jan Chorowski, Navdeep Jaitly, Yonghui Wu, and Zhifeng Chen. Sequence-to-sequeece models can directly translate foreign speech. In Proc. Interspeech, 2017.
Detai Xin, Tatsuya Komatsu, Shinnosuke Takamichi, and Hiroshi Saruwatari. Disentangled speaker and language representations using mutual information minimization and domain adaptation for cross-lingual TTS. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
Zhao Yi, Wen-Chin Huang, Xiaohai Tian, Junichi Yamagishi, Rohan Kumar Das, Tomi Kinnunen, Zhenhua Ling, and Tomoki Toda. Voice conversion challenge 2020: Intra-lingual semi-paran marilealllel and cross-lingual voice conversion. In Proc. Joint Workshop for the Blizzard Challenge and Voice Conversion Challenge, 2020.
Heiga Zen, Viet Dang, Rob Clark, Yu Zhang, Ron Weiss, Ye Jia, Zhifeng Chen, and Yonghui Wu. LibriTTS: A corpus derived from LibriSpeech for text-to-speech. In Proc. Interspeech, 2019.
Chen Zhang, Xu Tan, Yi Ren, Tao Qin, Kejun Zhang, and Tie- Yan Liu. UWSpeech: Speech to speech translation for unwritten languages. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
Yu Zhang, Ron J Weiss, Heiga Zen, Yonghui Wu, Zhifeng Chen, RJ Skerry-Ryan, Ye Jia, An- drew Rosenberg, and Bhuvana Ramabhadran. Learning to speak fluently in a foreign language: Multilingual speech synthesis and cross-lan voice cloning. In Proc. Interspeech, 2019.
Yibin Zheng, Xi Wang, Lei He, Shifeng Pan, Frank Soong, Zhengqi Wen, and Jianhua Tao. Forward-backwarr decoding for regularizing end-to-end TTS. In Proc. Interspeech, 2019.
13

---

Under review as a conference paper at ICLR 2022
A
TABLE OF HYPER-PARAMETI
Table 6: Model hyper-parametes used in the experiments (xn means layers).
Conversational (Single voice)
Conversational (Voice retained)
Fisher
CoVoST 2
Input
Sample rate (Hz)
8,000
48,000
16,000
48,000
Mel channels
80
Mel lower band (Hz))
125
Mel upper band (Hz)
3,800
7,600
7,600
Frame size (ms)
25
Frame step (ms)
10
Output
Sample rate (Hz)
24,000
Mel channels
128
Mel lower band (Hz)
20
Mel upper band (Hz)
12,000
Frame size (ms)
50.0
Frame step (ms)
12.5
SpecAugment
Freq blocks
2
Time blocks
10
Freq block length ratio
0.33
Time block max length ratio
0.05
Encoder
Conformer dims
144 x
144 x
144 x
16
Attention heads
4
4
4
Conv kernal size
32
32
32
32
Subsample factor
4
4
4
Attention
Output dim
256
512
512
512
Hidden dim
512
512
512
Attention heads
8
Dropout prob
0.1
0.2
0.2
0.2
Decoder
LSTM dims
256 4
512 6
512 4
512 x
Zoncout prob
0.1
0.1
0.1
0.1
Phoneme embedding dim
96
256
256
256
Label smoothing uncertainty
0.1
Loss weight
1.0
1.0
1.0
1.0
Duration predictor
Si-LSTM (dim x layers)
64 x 2
128 x 2
128 x 2
128 x
Loss weight
1.0
1.0
1.0
1.0
Synthesizer
LSTT dims
1024 x 2 0.1
LSTM zoncout prob
Pre-net dims
128 2
128 x 2
128 2
x
Pre-net dropout prob
0.5
Post-net (kernel, channels) x layers
512) X 4 (5, 128)
Loss weight
0.1
Training
Batch size
1,024
768
768
768
L regularization weight
10
10
10
10
Learning rate (Transformer schedule)
5.0
3.75
4.0
4.0
Warm-up up steps (Transformer schedule) Training steps
10K
20K
10K
10K
120K
130K
220K
150K
14

---

Under review as a conference paper at ICLR 2022
ehdigerail
90
30
STC VS STC
tgt STC
tgt tgt
s2st VS src
s2st tgt
Figure 3: Affinity matrices of d-vector similarity among 100 random examples. ("s2st" refers the predictions from Translatotron 2.)
Table 7:
Objective d-vector similarity between the predicted translated speech (English) and the source human speech (Spanish) speaker turns. The similarity between the leading/trailing 1.6- second segment from the predicted speech and the entire 1st/2nd source speaker's speech is measured. (1 / means higher/lower values are better.)
1st source speaker
2nd source speaker
Leading seg. f
Trailing seg. 4
Leading seg. 4
Trailing seg.
Translatotron
0.21
0.19
0.21
0.19
Concat aug.
0.20
0.14
0.14
0.21
Translatotron
0.20
0.22
0.27
0.29
Concat aug-
0.32
0.16
0.14
0.35
Training target
0.48
0.17
0.15
0.48
B OBJECTIVE SPEAKER SIMILARITY ANALYSIS
Subjective speaker similarity evaluation is costly and has long turnaround. We explored into alter- native objective evaluation using separately trained speaker encoders, such as d-vector (Wan et al.. 2018). We evaluated the voice retention performance using the cosine similarity of the d-vectors.
We first checked the scenario that each input contains a single speaker's recording. Figure visualizes the affinity matrices of d-vector similarity among different input utterances for Translatotron 2 model. The outstanding higher similarity values on the diagonals indicate that the model able to retain the source speaker's voice in the predicted speech.
We then conducted detailed evaluation for the voice retention performance for speaker turns. The experiment setting up was identical to Section 5.4.1 except that the speaker similarity was measured by d-vector similarity instead of subjective MOS evaluation. The d-vectors for each source speaker were computed on the entire original utterance before concatenation; the d- d-vectors for each speaker in the prediction is approximated by computing on the leading/trailing 1.6 seconds of predicted speech.
The results shown in Table 7 Consistent with the MOS evaluation results in Table when the concatenation augmentation was not used, the d-vector similarity to each source speaker is about the same regardless if was compared to the leading or trailing segments, indicating that the predicted speech was in a single speaker's voice and the model was unable to separate different speakers in the input, but rather optimized for both source speakers at the same time. When the concatenation augmentation was used, the d-vector similarity was significantly higher between matched pairs than between unmatched pairs, indicating that the models were able to separate different speakers in the input and retain their voice in the predicted speech respectively.
However, when these similarities are compared among different models, it seems suggesting that Translatotron performed better than Translatotron 2, which is in contradictory to the subjective evaluation results in Table 4 By carefully listening to the audio samples, we found that such discrepancy may be due to that the d-vector model was also sensitive to non-voice related acoustic characteristics, such as reverb and channel noise in the audios. This is likely a consequence of the fact that in the large-scall training set for the d- devector model used in the evaluation, each speaker is typically associated with a particular recording condition, e.g. recording device and room. Because the encoder output from the Translatotron model was of significantly larger dimension than from the
15

---

Under review as a conference paper at ICLR 2022
Table Translatotron performance on the Conversational dataset using an autoregressive synthe- sizer and non-autoregressive synthesizer.
Synthesizer
BLEU
MOS
RNN
55.6
4.21
0.06
Conformer
54.5
3.61 0.09
Table 9: Ablation studies on the CoVoST dataset (on high-resource X- pairs). + indicates using replacing component.
Source language
fr
dc
Translatotron (w/ SpecAugment)
17.7
9.9
17.7
13.1
+ Conformer encoder
18.9
10.8
18.8
13.9
NAT decoder
4.0
2.1
3.5
2.5
Translatotron 2
27.0
18.8
27.7
22.5
Conformer encoder
26.4
18.1
26.4
21.8
NAT synthesizer SpecAugment
18.3
27.0
25.9
17.9
25.9
21.8
Training target
82.1
86.0
85.1
89.3
Translatotron 2 model (2048 vs 256), it was capable of carrying more non-voice acoustic information and thus obtained better d-vector similarity, which not necessarily indicating higher speaker similarity.
These results suggest that while such speaker encoder-base objective analysis reveals insightful indications about the performance of the S2ST models, can be less reliable compared subjective MOS evaluation. Such reliability also highly depends on the training details of the speaker encoder model being used, especially the training corpus.
C NON-AUTOREGREESSWV SYNTHESIZER
Following recent non-autoregressive TTS works (Ren et al. 2021 Guo al. 2021 Lee al. b021b Elias et al. 2021a) a) we explored using non-autoreeressivee synthesizer in Translatotron 2, which may have significantly lower latency inference time. The experimental results in Table suggested that despite producing comparable BLEU on ASR transcribed text from the Translatotron 2 predictions, using non-autoregressivv synthesizer produced significantly worse naturalness of the predicted speech. This is consistent with the observation in TTS in Shen et al. 2000
D ABLATION STUDIES
To understand the importance of each components in the Translatotron 2 model, we conducted ablation studies the CoVoST multilingual X En dataset described in Section 5.5 All models in the ablation used the same input and output features, SpecAugment setting, and learning rate schedule described in Section 5.5 and Appendix A No auxiliary training target on the source text or phonemes were used. For models using an RNN-based encoder, we first applied the same 4x time-wise subsampling as used in the Conformer encoder, then used a 8-layer bidirectional LSTM stack with a cell size of 256. The number of parameters in this RaNebased encoder is close to the same in the Conformer encoder. For the Translatotron model using a NAT decoder, the same architecture and hyperparameters as the synthesizer of Translatotron was used to replace the original attention-base decoder. For Translatotron 2 using a non-autoreegressvve synthesizer, a simple 6-layer Conformer stack with dimension of 512 and attention heads was used to replace the autoregressive synthesizer, same as in Appendix C This Conformer based non non-autoregressive synthesizer is similar to the Transformer- saasformer-basedd decoder in the FastSpeech 2 TTS model (Ren et al.) 2021). but performed better in our experiments. All the rest hyperparameters follow Appendix for Translatotron 2, and follow the Conversational model in Jia et al. (2019h) for Translatotron. All models were trained for
16

---

Under review as a conference paper at ICLR 2022
spedtrogrm
Decoder
phoneme durations
Upsampler
Duration Predictor
spesker embedding
PnG BERT
Speaker Encoder
speaker reference
graphemes
waveform
Figure Augmented PnG NAT TTS model for cross-lingual voice transferring.
200K steps with batch size of 768. The evaluation was done using the same checkpoints for all the 4 language pairs, picked by the highest average performance on the dev sets.
The results are shown Table As can be seen, while the use of Conformer, SpecAugment, and NAT decoder helps the performance of Translatotron 2, replacing them with alternative architectural choices or removing SpecAugment only reduced the performance by a small degree (< BLEU). Similarly, directly using these components in the original Translatotron does not bring its performance close to Translatotron 2. These results suggest that the performance improvement Translatotron comes from addressing the weaknesses existing in Translatotron (Section 3), rather than the architectural choices of each individual components.
Consistent with the result in Appendix using non-autoregresssvve synthesizer in Translatotron obtained comparable translation quality to using an autoregressive synthesizer which based on the NAT decoder. However, Appendix Shows, such non-autoregressive synthesizer may suffer from lower naturalness the predicted speech. Directly using the NAT decoder in the original Translatotron obtained the worst performance because its monotonic nature does not work well for translation task.
E CROSS-LINGUAL VOICE TRANSFERRING TTS
The TTS model used for synthesizing training targets in the source speaker's voice Section 4. modified from the PnG NAT (Tia et al. 2021) TTS model by incorporating separately trained speaker encoder (Wan et al., 2018) in the same way as Jia et al. (2018). The architecture of this TTS model illustrated in Figure We trained this model on the LibriTTS corpus (Zen al. 2019), following the hyperparameters in Jia et al. (2021): Shen et al. (020): Wan et al. (2018). The speaker encoder separately trained speaker verification task and is frozen during the TTS model training. At synthesis time, the phonemes and graphemes in the target language and the reference speech in the source language are fed into model as inputs; and the model produces speech the target language with the voice from the source speech transferred.
17


Reviews:
[{"review_id": "rPAsVPKi1iO", "reviewer": "Reviewer_UaLy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of the paper include the follows.\n(1)\tCompared to other direct S2ST works which mostly focus on translation quality, this work also aims to improve perceptual quality such as naturalness of the translated speech, which is also very important to the overall quality and user experience of a S2ST system.\n(2)\tAlthough Translatotron 2 also uses an encoder (conformer)-decoder (a stack of LSTMs) with attention modules, which is similar to a ST model, the difference is that instead of predicting subword units as a standard ST model, Translatotron 2 uses this framework to predict target phonemes, and then uses the phonemes as input to the spectrogram synthesizer. Note that the original Translatotron uses both source and target side phoneme predictions as auxiliary training tasks but Translatotron 2 only predicts and uses target side phonemes, yet achieving a better BLEU than Translatotron, especially on small dataset.  \n(3)\tExperimental results showed Translatotron2 improves BLUE and naturalness (MOS) and robustness (UDR), compared to Translatotron, on conversational and smaller datasets, and also significant BLEU improvement (on ASR transcripts) on the multi-lingual CoVoST 2 dataset.\n\nThe main weaknesses of the paper include the follows.\n(1)\tThe innovations from Translatotron 2 over Translatotron are not very strong. The major innovations are the use of the encoder(conformer)-decoder(LSTM) with attention framework to predict target phonemes, instead of using both source and target phonemes, and use phonemes as input to the spectrogram synthesizer.  The synthesizer in Translatotron 2 replaces the attention module with a duration-based upsampler, which yields more robust synthesized speech. But this approach is similar to the previous published works such as (Shen et al., 2020). And, the synthesizer in Translatotron 2 is simplified as it only used the L2 loss on the total predicted duration of the entire sequence.  The more principled approaches such as using per-phoneme duration labels for training the duration predictor and applying FVAE alignment, as in (Shen et al., 2020), would be important to be investigated in this work, but they were left for future work.\n\n(2)\tSome experimental results need to be further discussed. For example, the effect of ConcatAug in Table 2, Table 3, and Table 4 is not discussed clearly. For example, ConcatAug hurts BLEU, MOS and UDR for the relatively larger conversational dataset, and it is useful to provide analysis for it.\n\n(3)\tAlthough there is a significant improvement from Translatotron 2 over Translatotron on translation quality, naturalness and robustness of predicted speech, as can be seen from Table 2, there is still a relatively significant gap between Translatotron 2 and cascaded (ST->TTS). It would be useful to provide analysis and provide insights on more focused future directions on direct S2ST.\n", "summary_of_the_paper": "This paper advances the previously proposed direct speech-to-speech translation (S2ST) model Translatotron.  Direct S2ST has many benefits but has quite limited research effort. The major contributions of this work can be summarized as follows. Translatotron 2 addressed the following major problems of Translatotron: 1)  translation quality, naturalness and robustness of predicted speech are significantly improved and now comparable to a cascaded system (i.e., speech-to-text translation followed by TTS).  2) Similar to Translatotron, the model retrains source speaker’s voice in predicted speech, but Translatotron 2 achieves this without relying on any explicit speaker embedding or speaker ID, hence Translatotron 2 will not be able to generate speech in a different speaker’s voice. This will address some ethical concerns for deployment. Translatotron 2 achieved these improvements through the following approaches. Compared to the previous Translatotron model,  1) Translatotron 2  uses  the output from the auxiliary target phoneme decoder as an input to the spectrogram synthesizer;  2) the spectrogram synthesizer is duration-based, while still keeping the benefits of the attention mechanism.", "summary_of_the_review": "This paper extends the previous direct S2ST Translatotron model with Translatotron 2 model, which significantly improves translation quality (BLEU), the naturalness (MOS) and robustness (UDR) of predicted speech compared to Translatotron,  through a few modifications. The paper also proposed an approach to train targets in source speaker’s voice and prohibits generation in a different speaker’s voice to mitigate spoofing audio artifacts.  Experimental results on S2ST datasets of different sizes (relative large and small) and multilingual S2ST demonstrate the significantly improved performance from Translatotron 2 over Translatotron, and also the effectiveness of the proposed approach to retrain the source speaker’s voice in the translated speech.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "csyupgb9eGC", "reviewer": "Reviewer_rKi2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n1) With the use of Conformer encoder with Spec Augment and giving auxiliary target phoneme decoder output as input to the spectrogram synthesizer , the results in Table2 indicate the significant improvement in transcription quality\n\n2) The voice retention capability of Translatotron 2 is significant\n\n3) The ability of Translatotron2 to retain voice in case of speaker turns without additional diarization systems is interesting\n\n4) The multilingual capability of Translatotron 2 is significant\n\nWeakness:\n\n1) While results are impressive, however, novelty is limited. For example: a) Conformer encoder and SpecAug are known to improve ASR performance; b) Using auxiliary target phoneme decoder output as input to the spectrogram synthesizer makes the architecture similar to cascade system; c) Robustness issue of Translatotron is addressed here by using Non-Attentive Tacotron spectrogram synthesizer \n\n2) In both Table 2 and Table 3,  two results for the Translatotron model are shared, one of which is reported from the original paper. For the second results that are reported, it appears that the authors trained their own Translatotron model for the experiments, however, this should be stated explicitly in the paper \n\n3) In Section 4.2, pause duration used to concatenate two consecutive speakers is missing in the description of  ConcatAug method. This information will be useful to replicate the experiments. Besides, it will justify the rationale of using 1.6 sec leading and trailing segments to measure voice retention in speaker turn experiments.\n", "summary_of_the_paper": "Translatotron2, a speech-to-speech neural based translation system is proposed. The work is a modification of Translatotron (also a speech-to-speech translation system) and tries to address some of the  issues in Translatotron:\n\na) Translation quality fairly below a cascaded baseline\n\nb) synthesized translated speech suffers from robustness issues, such as babbling and long pauses\n\nc) Voice retention relies on explicit speaker embedding which can potentially be misused for generating spoofing audio with arbitrary content\n\nCompared to Translatotron, following modifications are made in Translatotron2. The experiments suggest that Translatotron 2 significantly outperforms Translatotron, and is comparable to a cascaded system, in terms of translation quality, speech naturalness and speech robustness.\n\na)  Output from the auxiliary target phoneme decoder is used as an input to the spectrogram synthesizer \n\nb) Conformer Encoder with SpecAugment is used, which is known to improve the speech to text performance\n\nc) Duration-based  Spectrogram synthesizer is used. Specifically, architecture and hyperparameters similar to Non-Attentive Tacotron spectrogram synthesizer is used, which is known to improve the robustness issue in Tacotron2 \n\nd) To retain speakers’ voices across translation, Translatotron2 is trained on parallel utterances with the same speaker’s voice on both sides. This enables the trained model to restrict generation to only the source speaker’s voice, mitigating the risk of potential misuse for creating spoofing audio artifacts. Parallel utterances are synthesized using a TTS model with cross lingual voice transfer capacity.\n\nIn addition, authors also trained Translatotron 2 with examples that contain two speakers’ voices in both the source and the target. The experiments suggest that Translatotron 2 has capability to retain voice identity when the input contains speaker turns.  For multilingual experiments, Translatotron 2 was trained with 4 high resource languages  and BLEU scores suggest Translatotron2 outperforms Translatotron in multilingual settings as well.\n\n", "summary_of_the_review": "The paper is well written.  The experiment results of  improvement in  translation quality , naturalness and robustness are convincing. The voice retention capability in the speaker turn setting of Translatotron 2 is appealing.  However, the overall  paper’s novelty is limited mainly because the paper brings in the best systems available in literature together to improve speech-to-speech translation systems performance. Thus, the paper might be more appropriate for a speech technology focused conference. \n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "aIoB6OTpwee", "reviewer": "Reviewer_aQuX", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper proposed several non-trivial improvement to a previous S2ST model, which significantly improves translation quality and speech naturalness, as well as mitigates audio spoofing concerns in production. Although some improvements are apparently inspired by existing work, they can still be considered somewhat novel and significant contribution.\n\nMost of the claimed improvements are substantiated by experiments. However, some new questions arise from the experiment results themselves.\n1. The reason to translation quality improvement (5.1, 5.5) is not discussed, and is not clear from the proposed approaches. Some ablation study would be helpful here.\n2. Voice retention in speaker turns (5.4.1) seems somewhat tailored towards ConcatAug, and its relevance to real-world S2ST application is not clear. (How often will a third person use S2ST to translate dialogue between two other persons?) Instead it would be more interesting to see experiment scenarios related to anti-spoofing as discussed at the beginning of the paper.\n\nThe paper is generally written clearly and is easy to understand. But some important technical details are completely left out as reference to previous work, such as 3.3 and 4.1. It makes readers not closely following latest research in speech quite difficult to understand. In other words, the paper is not self-contained. In addition, requiring reviewers to search for these references during a double-blind review can easily reveal authors' identity or affiliation.", "summary_of_the_paper": "The paper proposed a speech-to-speech translation (S2ST) model that is and improvement to a previous work. The model is trained end-to-end from speech to speech, along with an auxiliary speech-to-phoneme task. The relevance of the two tasks is further exploited beyond parameter sharing, by feeding phoneme decoder's hidden layer output to spectrogram synthesizer's input, and the synthesizer is duration based. To retain speakers' voice from source language to target language, two data-centric approach is proposed. First, a zero-shot voice-transfer TTS model is trained, to transfer target speech into source speaker's voice. Second, to enforce local voice similarity within an utterance, two samples from two distinct speakers are randomly selected and concatenated to create new training samples. Experiments are conducted on speech-to-text translation (ST) datasets with TTS synthesized target speech, and are measured by both objective (BLEU) and subjective (MOS) metrics. The proposed model's translation quality is much closer to the cascaded ST+TTS oracle than baseline S2ST models, for both bilingual and multilingual S2ST tasks. Its generated speech is also more natural and more resembles the source speech.", "summary_of_the_review": "Contribution and novelty are moderately significant. Experiments firmly supports multiple claimed improvement to the previous model. Key technique are completely left out as reference to other work, making out-of-domain readers difficult to follow.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JQqswMw_Yy_", "reviewer": "Reviewer_gzba", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Please list both the strengths and weaknesses of the paper. When discussing weaknesses, please provide concrete, actionable feedback on the paper.\n\nThis paper describes an updated version of Translatron which purports to be a speech to speech translation system trained end-to-end.  The claimed benefits include the ability to use with languages without a written form and reduced error-compounding relative to a cascade system.  However, I find no support for these claims in the paper.  The system is not strictly end-to-end since it relies on an auxiliary loss function based on a phoneme transcription and nowhere does it explain where this transcription comes from (the original Translatotron paper is also quiet about this).   Since the experimental data was derived from text corpora my guess is that the phoneme transcription used in the experiments was generated by a standard G2P system.  Given the difficulty of automatic phone transcription, its not at all clear how you could in practice realise the claimed benefit of using with languages with only speech as training data and no text (or well-trained ASR system)  \n\nThe results presented for translation quality come close to those achieved by an ST+TTS cascade system but do not match it.  Given that deriving an accurate phoneme transcription is not much easier than deriving a text transcription, I fail to see the benefits of this system.  \n\nAlso the so-called Conversational dataset is mis-leading because this is actually crowd-sourced speakers reading conversational texts.  Read speech is quite different from spontaneous conversational speech. \n\nThe architectural changes between Translatotron and Translatotron 2 are described but not fully justified.  The paper contains various evaluations of voice quality and similarity but does not provide any ablation studies to show why the new architecture is better than the old and the relative contribution of each change. \n\nOne particular difficulty is understanding the BLEU results since this is measured automatically by applying ASR to the output.  I presume the reason why improving the output synthesis quality improves the BLEU score is not because the underlying translation is any better, but simply because the ASR made fewer errors?\n\nThe authors include an augmentation option in training to include more than one speaker turn with the goal of getting the target speech to track the changed input speech instead of being an average of the two.  The results show that without the augmentation, the target voices are indistinguishable, whereas with it there is a clear differentiation although the absolute similarities are quite poor.\n\nOne of the many difficulties that i had in understanding this paper was the lack detail on the training data and how the results would apply to a real-world use-case.  Firstly, I cant find any information on the relative sizes of the train and test sets.  Are speakers in the test set included in the training set - ie is this evaluated as a speaker independent translation system or a speaker dependent system.  For the speaker dependent case,  if I wanted to train a Translatotron for a single voice, how much speech data would I need?  \n\nOverall, there is much interesting stuff in this paper, but far too many unanswered questions.\n", "summary_of_the_paper": "This paper describes an updated version of Translatron which purports to be a speech to speech translation system trained end-to-end.  The new system now includes the ability to control the target voice to match the input voice without providing an explicit speaker id vector and the translation performance of the updated system now comes close to that of a cascade speech to text + TTS system, although it does not match or improve on it.  \n", "summary_of_the_review": "The paper describes an incremental improvement over a previous paper on the same topic.  However, the experimental details are not adequately described and the claimed benefits relative to a cascade system are not justified.\n\nI remain somewhat sceptical of the practical utility of the overall framework.  Nevertheless, following the authors response to questions and updated paper, I have increased my score to \"marginally above\" the threshold.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "879wRUTDEWD", "reviewer": "Reviewer_YR5V", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper gives a good picture of current developments in speech synthesis and voice conversion, especially those that are derived from the Tacotron family of models. The setup seems to be directly derived (at least in philosophy) from the Translatotron. In Translatotron, the main components were an encoder and two types of decoders - one to synthetize target spectrogram and the other to classify phonemes in source/target languages. The language decoders in Translatotron were treated in the manner of multi task learning - to add extra supervision to the setup rather than to be used directly for the task. In the current work, the language decoders seem to be performing a similar role, but the paper does not go too much in detail about the philosophical motivations. \n\nThe paper is generally very well written and results (also, samples) look good. However, I have some misgivings about \n\n1) The novelty of concepts. It seems that most of the ideas are already there in Translatotron, and this paper modifies architecture (and does not give much insight into these changes).\n2) I did not understand the presentation in section 4.1 - of retaining voice. The paper proposes an alternative way to produce source speaker's voice without conditioning with a speaker embedding, which they claim could be used for spoofing purposes. They also point to other works and especially, state that they can produce target voice with the help of a speaker encoder network and TTS. \n\n\"We modified the PnG NAT (Jia et al., 2021) TTS model by incorporating a separately trained speaker\nencoder (Wan et al., 2018) in the same way as Jia et al. (2018), and trained it on the LibriTTS corpus\n(Zen et al., 2019). ...\"\n\nI find that this could be explained in a clearer manner. Do they convert to text and then to speech? Moreover, the use of a speaker encoder seems to suggest that the embedding will add on to the context as in other works, but then wasn't the aim of the work to avoid using a speaker encoder for antispoofing purposes? Please clarify.\n\nOverall, I feel that the current work does not have sufficient novelty, and details are missing. ", "summary_of_the_paper": "This paper proposes a setup to perform Direct Speech To Speech Translation from one language to another. As such, we could view this as a voice conversion setup, but with the additional task of rendering translated voice in a different language. It seems to be created on the same lines as Translatotron, but with improved performance from different modeling choices. The architecture is encoder-decoder, with the decoder side consisting of two parts - a spectrogram synthesizer, and a phoneme/language translator. The phoneme translator is needed because it also needs to translate to a different language. \n\nThe experimental evaluation is quite good, consisting of cases to translate with the same voice as source (voice retention); samples with speaker turns (augmented dataset with two different speaker utterances concatenated together), and cross language translation. The paper shows improved results over Translatotron in all cases. \n\nAside from the results, the contributions seem to be incremental. They also emphasize a modified training setup that does not use speaker embeddings to avoid antispoofing. ", "summary_of_the_review": "Not enough novelty.\nSome components not explained clearly,", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposed a speech-to-speech translation (S2ST) model. The model is trained end-to-end from speech to speech, along with an auxiliary speech-to-phoneme task. Experiments firmly support multiple claimed improvements to the previous model.

However, most reviewers argue the novelty and clarity of this paper, making this paper cannot be accepted by ICLR-2022. We hope the authors can modify this paper accordingly.
Decision: Reject