Kyoto Speech-to-Speech Translation System for IWSLT 2023
Year: 2023

Abstract:
This paper describes the Kyoto speech-tospeech translation system for IWSLT 2023. Our system is a combination of speech-to-text translation and text-to-speech synthesis. For the speech-to-text translation model, we used the dual-decoder Transformer model. For the text-to-speech synthesis model, we took a cascade approach of an acoustic model and a vocoder.

heading: 1 Introduction
text: This paper describes the Kyoto speech-to-speech translation system for IWSLT 2023 (Agarwal et al., 2023). Our system is a combination of speech-totext translation and text-to-speech synthesis. For speech-to-text translation model, we used dualdecoder Transformer model following Le et al. (2020). For text-to-speech synthesis model, we took cascade approach of an acoustic model and a vocoder. We used FastSpeech 2 (Ren et al., 2021) as the acoustic model and HiFi-GAN (Kong et al., 2020) as the vocoder.

heading: 2 System Description
text: The speech-to-speech translation system is a combination of speech-to-text translation and text-tospeech synthesis.

heading: 2.1 Speech-to-Text Translation
text: We adopt the end-to-end speech-to-text translation architecture. The speech-to-text translation model is based on dual-decoder Transfomer (Le et al., 2020).
As shown in Figure 1, the model is a Transformer-based model, comprising two decoders - one for speech-to-text translation (ST) and the other for automatic speech recognition (ASR). The task of ASR and ST can be defined as follows:
• For ASR, the input sequence s = [s1, ..., sTs ] is a sequence of speech features. The out-
put sequence x = [x1, ..., xTx ] is the corresponding transcription, where Tx indicates the length of the transcription.
• For ST, the input sequence s = [s1, ..., sTs ] is the same with ASR and the output sequence y = [y1, ..., yTy ] is the corresponding translation in target language, where Ty indicates the length of the translation.
The model performs the multi-task learning of ASR and ST and the output distributions can be written as
Dasr-st = p(x,y|s)
=
max(Tx,Ty)∏
t=0
p(xt, yt|x<t,y<t, s) (1)
The training objective is a weighted sum of crossentropy losses for both tasks:
Lasr-st = αLasr + (1− α)Lst (2)
Different decoders can exchange information with each other with the interactive attention mechanism, which refers to replacing attention sublayers in the standard Transformer decoder with interactive attention sub-layers (Liu et al., 2020). In our models, the replaced sub-layers are the encoderdecoder attention sub-layers.
As illustrated in the lower part of Figure 1, an interactive attention sub-layer consists of one main attention sub-layer and a cross-attention sub-layers. The main attention sub-layer is the same as the replaced attention sub-layer. The cross-attention sub-layers receive query Q from the same decoder A and receive key K and value V from another decoder B. We adopt the parallel variation of dualdecoder Transformers where K and V are hidden states from the same layer in decoder B.
The final output is obtained by merging the output of the primary attention sub-layer Hmain with
357
the output of the cross attention sub-layer Hcross. We adopt linear interpolation as the merging function. Therefore the output representations of the interactive attention sub-layers are
Hdual = Hmain + λHcross (3)
where λ is a learnable parameter.
2.2 Text-to-Speech Synthesis We adopted the approach to cascade an acoustic model and a vocoder. We used FastSpeech 2 (Ren et al., 2021) as the acoustic model and HiFi-GAN (Kong et al., 2020) as the vocoder. FastSpeech 2 adopts Transformer-based architecture for the encoder and the Mel-spectrogram decoder, and the variance adapter between them predicts the duration, pitch, and energy of the audio. HiFi-GAN employs generative adversarial networks to generate waveforms from Mel-spectrograms. It is composed of one generator and two discriminators, a multiperiod discriminator, and a multi-scale discriminator. We used the PaddleSpeech toolkit (Zhang et al., 2022a) and the pretrained models provided by Zhang et al. (2022a) to generate waveforms.
Table 1: The size of the datasets and the ltered versions used for training the ST system. 3 Experiments 3.1 Speech-to-Text Translation 3.1.1 Datasets To train our ST system, we utilized two distinct datasets: MuST-C (Di Gangi et al., 2019) v2 with Chinese translations, and GigaST (Ye et al., 2022) which is the original dataset that was used to construct the GigaS2S dataset provided by the organizers. Both datasets offer unique advantages. While GigaST is in the same domain as the development and test data, MuST-C is not. In addition, GigaST is considerably larger than MuST-C. However, it is worth noting that the translations in GigaST were generated by a machine translation system and may not be of the same quality as those in MuST-C, which were translated by human. As a result, determining which dataset is more likely to yield better results requires further experimentation. To shorten the training time and improve performance, we ltered the extremely large GigaST dataset to select utterances with better translation quality. As the translations in GigaST are machinegenerated and there are no reference translations available, we evaluated the translation quality using the cosine similarity of sentence embeddings from the source and target sentences. We tested two different models for generating the embeddings: LASER1 and paraphrase-xlm-r-multilingual-v1 from Sentence Transformers2 (simply referred to as Sentence Transformers subsequently). The resulting similarity distributions are shown in Figure 2. We selected the top 10% of the data based on similarity scores (data that is on the right-hand side of the red line). Table 1 shows the sizes of MuST-C and GigaST before and after ltering.
1https://github.com/facebookresearch/LASER 2https://github.com/UKPLab/ sentence-transformers/tree/master/examples/ training/paraphrases

heading: 3.1.2 Training and Decoding
text: English sentences were normalized and tokenized using the Moses tokenizer (Koehn et al., 2007), and punctuations were stripped. Chinese sentences were tokenized using jieba.3 English and Chinese tokens were further split into subwords using the BPE method (Sennrich et al., 2016) with a joint vocabulary of 16, 000 subwords.
We used Kaldi (Ravanelli et al., 2019) to extract 83-dimensional features normalized by the mean and standard deviation computed on the training set. We removed utterances with more than 6, 000 frames or more than 400 characters and used speed perturbation (Inaguma et al., 2020) with factors of 0.9, 1.0, and 1.1 for data augmentation.
Our implementation was based on the ESPnetST toolkit (Inaguma et al., 2020). We used the same architecture for all the ST models with a 12- layer encoder and 8-layer decoders. The coefficient α in the loss function (Equation 2) was set to 0.3 in all the experiments. We used the Adam optimizer (Kingma and Ba, 2015) and Noam learning rate schedule (Vaswani et al., 2017) with 25, 000 warmup steps and a maximum learning rate of 2.5e− 3. We used a batch size of 48 per GPU and trained models on a single machine with 4 Tesla V100 GPUs. The models were trained for 25 epochs. We kept checkpoints after each epoch and averaged the five best models on the development set based on prediction accuracy. For decoding, the beam size was set to 5 for ST and 1 for ASR.

heading: 3.1.3 Results
text: We conducted experiments to investigate the impact of using different datasets for training the system. The results are presented in Table 2. Additionally, we evaluated the performance of the system when using different sentence embedding models for data filtering. Our findings reveal that LASER produces better results compared to Sentence Transformers. Notably, after filtering the data using LASER, the total number of hours of audio is higher compared to that obtained using Sentence Transformers. Given this observation, it might be more appropriate to perform filtering based on the length of the audio rather than the number of utterances.
Our experiments also revealed that training the model with GigaST alone yielded better results compared to using only the MuST-C dataset. Fur-
3https://github.com/fxsjy/jieba
thermore, we evaluated an approach in which we trained the model with one dataset and use the best checkpoint to initiate the training with the other dataset. However, we observed that this approach did not yield any improvement compared to training the model with GigaST alone.
Based on these findings, we adopted the translation generated by the ST system trained solely on GigaST filtered based on LASER for our submission.

heading: 3.2 Text-to-Speech Synthesis
text: We used pretrained models provided by Zhang et al. (2022a) trained on the AISHELL-3 dataset (Shi et al., 2021). The PaddleSpeech toolkit provides several models trained with the AISHELL-3 dataset, including FastSpeech 2 and HiFi-GAN. We used the best-performing model combination in terms of MOS reported in (Zhang et al., 2022a). For other configurations, such as grapheme-tophoneme conversion, we followed Zhang et al. (2022a).
The generated audio files have one channel, a sample width of 16 bit, and a frame rate of 24, 000. Because the predictions of speech-to-text translation sometimes contained English words that were preprocessed to empty strings by the grapheme-tophoneme conversion, some (less than 1 % of the test set) audio files could not be generated.

heading: 4 Conclusion
text: In this paper, we described our system, which is a combination of speech-to-text translation and textto-speech synthesis. For speech-to-text translation, we trained the Dual-decoder Transformer model with the GigaST dataset filtered based on the similarity of multilingual sentence embeddings. For the text-to-speech synthesis model, we took a cascade approach of an acoustic model and a vocoder and used a combination of FastSpeech 2 and HiFi-GAN.
In the future, we will try to perform multi-level pretraining based on transforming SpeechUT (Zhang et al., 2022b) with phonemes as unit. We will also try to use Encodec-based speech synthesis method similar to VALL-EX (Zhang et al., 2023) to increase the accurate representation of emotions and vocal patterns.

heading: Future Work
text: In the future, we will try to perform multi-level pretraining based on transforming SpeechUT (Zhang et al. , 2022b) with phonemes as unit. We will also try to use Encodec-based speech synthesis method similar to VALL-EX (Zhang et al. , 2023) to increase the accurate representation of emotions and vocal patterns