The HW-TSC’s Speech-to-Speech Translation System for IWSLT 2023
Year: 2023

Abstract:
This paper describes our work on the IWSLT2023 Speech-to-Speech task. Our proposed cascaded system consists of an ensemble of Conformer and S2T-Transformer-based ASR models, a Transformer-based MT model, and a Diffusion-based TTS model. Our primary focus in this competition was to investigate the modeling ability of the Diffusion model for TTS tasks in high-resource scenarios and the role of TTS in the overall S2S task. To this end, we proposed DTS, an end-to-end diffusionbased TTS model that takes raw text as input and generates waveform by iteratively denoising on pure Gaussian noise. Compared to previous TTS models, the speech generated by DTS is more natural and performs better in codeswitching scenarios. As the training process is end-to-end, it is relatively straightforward. Our experiments demonstrate that DTS outperforms other TTS models on the GigaS2S benchmark, and also brings positive gain for the entire S2S system.

heading: 1 Introduction
text: Compared to previous iterations of the IWSLT-S2S task (Anastasopoulos et al., 2022; Guo et al., 2022), this year’s task (Agarwal et al., 2023) is distinct, particularly in terms of data. The official training dataset provided is GigaS2S (Chen et al., 2021; Ye et al., 2022), which is substantially larger than previous S2S datasets, with a data size of 10,000 hours. Although the target text and speech are generated by MT and TTS systems, their quality is relatively high, making them suitable for initiating research on end-to-end S2S or TTS models in highresource scenarios.
Our strategy is similar to that of last year (Guo et al., 2022), where we used a cascaded S2S system, but our research focus has shifted. In last year’s work, we primarily studied the role of ASR and MT in the S2S system and attempted to optimize the context consistency of translation results. In this
year’s competition, we shifted our research focus to the TTS component. Therefore, we directly used the ASR and MT systems in our offline ST track (Wang et al., 2022a,b). Additionally, we no longer considered the issue of context consistency during inference.
Given the unprecedented success of the Diffusion Model (Ho et al., 2020; Rombach et al., 2022) in image generation over the past few years, we sought to explore its potential in speech synthesis. Thus, we proposed an end-to-end Diffusion TTS (DTS) model. Unlike previous TTS models, such as FastSpeech2 (Ren et al., 2021), which use phonemes as input and use a duration predictor to determine the duration and generate melspectrograms, DTS uses raw text as input, predicts the total audio length, and generates the waveform by iteratively denoising the output.
The structure of this paper is as follows: We first introduce the dataset used in this task, followed by a brief introduction to the ASR and MT models used. Then, we provide a detailed description of our proposed DTS model. Finally, we showcase the performance of each model on the GigaS2S dataset.

heading: 2 Method

heading: 2.1 Dataset
text: To train the ASR model, we combined five datasets and added corresponding domain tags to enable the model to generate speech in the desired style (Wang et al., 2022b). For the MT model, we aggregated all available en-de, en-zh, and en-ja translation data allowed for constrained offline tasks and added language tags to train a multilingual model. Finally, for the TTS model, we utilized the Chinese text and speech pairs from GigaS2S (Ye et al., 2022).

heading: 2.2 ASR
text: We trained our ASR models using a combination of five datasets: MuST-C V2, LibriSpeech, TED-
277
LIUM 3, CoVoST, and IWSLT. Table 1 provides statistics for these datasets. Our model uses an 80-dimensional filterbank feature, with input samples restricted to a frame size between 50 to 3000 and a token limit of 150 to ensure that the Transformer model’s encoder and decoder can process sequences of limited size.
To identify outliers, we calculated the speech speed of each sample based on the transcript length and frame size. We excluded samples with speeds outside the range of µ(τ)± 4× σ(τ), where τ = # frames # tokens .
We utilized an ensemble of two models to improve ASR performance: Conformer (Gulati et al., 2020) and S2TTransformer (Synnaeve et al., 2019). The encoder of Conformer incorporates a macaron structure at each layer based on the S2TTransformer’s encoder to enhance speech encoding capability. Our ensemble method involves averaging the probabilities output by both decoders at each decoding step during beam-search. To control the model’s generation style, we added prefix tags corresponding to the COVOST dataset for inference, making the model’s inference style closer to GigaS2S transcripts.

heading: 2.3 MT
text: For MT, we utilized the multilingual Transformer model that we developed for the offline track, training it on en-zh, en-de, and en-ja datasets. To ensure high-quality pairs, we first cleaned and removed duplicates from the data, then filtered it using LaBSE (Feng et al., 2022) to select domain-specific data. During training, we employed R-Drop (Liang et al., 2021) for additional regularization. Our Transformer (Vaswani et al., 2017) model consisted of a 25-layer encoder and a 6-layer decoder with a dimension of 1024 and an FFN dimension of 4096.

heading: 2.4 TTS

heading: 2.4.1 Modeling
text: The Denoising Diffusion Model (DDM) (Ho et al., 2020) models a continuous process of iteratively denoising Gaussian noise to restore the original sample. The model consists of two processes: the forward process of adding noise and the reverse process of denoising. These continuous processes are assumed to have Markovian properties and can be decomposed into T conditional distributions through a Markov chain, with x0 representing the original data (raw waveform in the TTS task) and xT representing pure noise.
In the forward process of DDM, q(x1:T |x0, c) is decomposed into a Markov process of T steps and conditioned on the input text c:
q(x1:T |x0, c) = T∏
t=1
q(xt|xt−1) (1)
q(xt|xt−1, c) = N (xt; √ 1− βtxt−1, βtI). (2)
The sampling of xt given xt−1 can be expressed as:
xt = √ 1− βtxt−1 + √ βtϵ, (3)
where ϵ ∈ N (0, I), and βt ∈ [0, 1] is a noise scheduler related to t. Therefore, each step of the forward process is adding a certain amount of Gaussian noise to the previously corrupted speech xt−1. Finally, x0 ultimately evolves into white noise that follows a Gaussian distribution. An important characteristic of the forward process is that xt ∼ q(xt|x0, c) for any t has a closed form:
q(xt|x0, c) = N (xt; √ ᾱtx0, (1− ᾱt)I} (4)
αt = 1 − βt, and ᾱt = ∏t
s=1 αs, so we can efficiently obtain xt for any t from x0 during training.
In the reverse process, the denoising process is similar to the forward process, and is also described as a T -step Markov process:
pθ(x0:T , c) = p(xT ) T∏
t=1
pθ(xt−1|xt) (5)
pθ(xt−1|xt, c) = N (xt−1;µθ(xt, t, c), σ2t I), (6)
where the µθ can be learned by neural networks and σ2t = 1− αt.
The training objective of the Diffusion Model is to maximize the log-likelihood of p(x0|c), which
is intractable, so optimization on the variational bound is used instead. (Ho et al., 2020) further simplify it to an unweighted version of L2 regression loss with respect to ϵ̂ and added noise ϵ. In our work, we predict the x0 with the model instead of the noise:
L(θ) = Et,x0,ϵ [ ||x̂θ(xt, t, c)− x0|| ] (7)
Here, t is uniformly sampled from the interval [0, T ].
During inference, the model iteratively samples xt−1 from xt:
xt−1 = 1√ αt
( xt −
1− αt√ 1− ᾱt
ϵθ(xt, t, c) + σtz )
(8)
ϵθ = 1√
1− ᾱt
( xt − √ ᾱtx̂θ(xt, t, c) ) (9)
where σt = √ 1− αt and z ∼ N (0, I). In our experiments, to allow for flexible determination of the maximum step T , we choose to use a continuous t ranging from 0 to 1. During training, t is uniformly sampled, and we use the cosine noise scheduler (Nichol and Dhariwal, 2021).
In addition to modeling the denoising process, DTS also needs to predict the length of the target audio in advance, as DTS is essentially a nonautoregressive (NAR) model. However, unlike previous TTS models that predict the duration of each phoneme, we directly model the total number of frames in the target audio, which is more convenient. Specifically, we use the text representation after average pooling, denoted as hc, as the input to the classifier ϕ to predict the length distribution. Then, we calculate the cross-entropy loss with the frame number Nx0 of x0.
Llength = CE(ϕ(hc; θ), Nx0) (10)

heading: 2.4.2 Model Architecture
text: The DTS model is essentially a parameterized denoising function x̂(xt, t, c) which takes xt, t as input, conditions on c, and predicts the x0 for the sampling of xt−1. The model makes some modifications on top of the Transformer model to make it more suitable for speech synthesis. As shown in Figure 1, the main modifications are as follows:
• On top of the Encoder, we add a two-layer FFN network to predict the length of the target audio.
• In the input part of the Decoder, we use two 1D convolutions with a proper setting of kernel size, stride, and padding, so the sequence length before and after convolution remains unchanged.
• As the Diffusion model depends on the time step t, we additionally introduce a Timestep Embedding, and use the same implementation as (Ho et al., 2020).
• To make the time step encoding more comprehensive, we add Layerwise time encoding at each layer and added to the encoded hidden states from the last layer.
• In the output part of the decoder, we add 2 1D deconvolutions to restore the hidden state back to the waveform. We use deconvolution because we found that using only linear projection leads to a lack of dependency between the generated waveform and the previous waveform, resulting in noticeable jitter, which can be significantly eliminated by using deconvolution.

heading: 3 Experiment

heading: 3.1 Experimental Setup
text: For the ASR and MT parts of our S2S system, we directly used the same setting as in the Offline track. For the TTS part, we trained the model on the GigaS2S dataset for 360k steps, with a maximum learning rate of 1e-4, warmup of 20000 steps, and a batch size of 32 samples per GPU. The maximum and minimum audio lengths were restricted to 25 seconds and 0.5 seconds, respectively. The model has 12 layers in the encoder and 16 layers in the decoder, with a hidden dimension of 512 and an FFN dimension of 2048. DTS can directly generate waveforms, but since audio waveforms are usually long, we pre-segment them into equally sized nonoverlapping frames. In this way, the model learns to generate the waveform frame by frame, and we only need to flatten these frames to get the final output. In our experiments, we used a frame length of 1200 and a sampling rate of 24000. When inference, we set the sampling step to 100. In addition to
the raw waveform, DTS can also learn to generate mel-spectrogram, simply by changing wave frames to spectrogram frames. This is also evaluated in our experiment.

heading: 3.2 Experimental Results
text: In the experiments, we tested the performance of each module in our S2S system separately. In addition to testing with the cascaded results as input, we also conducted independent tests with ground truth input. For the three modules, we mainly used the dev set of GigaS2S for evaluation. In terms of evaluation metrics, for ASR and MT, we used WER, BLEU and ChrF, respectively. For TTS, we used a Whisper-medium (Radford et al., 2022) model to transcribe the TTS-generated audio back into the text for automatic evaluation and calculated WER.
ASR Results We evaluated the results of two ASR models trained on the same corpus separately, as well as the ensemble version. As shown in Table 3, the ensemble results were slightly better.
MT Results In the evaluation of MT, we considered two scenarios: using ground truth transcripts as input and using the output of the previous ASR module as input. The experimental results showed that the robustness of MT was relatively good, even if there were errors in the ASR output, the difference in BLEU score was not significant as shown
in Table 4.
TTS Results In the TTS experiments, because the development set of GigaS2S contains codeswitching samples, we evaluated not only the WER of the entire set but also separately evaluated the cases without the code-switching. As for the models, we chose FastSpeech 2 as the baseline. In addition, we trained an additional DTS based on mel-spectrogram for comparison with the waveform-based DTS. Both FS2 and DTS-mel used the Griffin-lim vocoder. As shown in Table 2, DTS-Wave outperformed the other two models, especially on Chinese monolingual data.
Full Pipeline Results In addition to testing each module separately, we also tested the final metrics of the entire pipeline. We compared the difference between the speech generated by the three TTS models with the MT results as input by computing the BLEU and ChrF with the ground truth translation. Table 5 shows that there is a difference that existed, but it is not significant. Therefore, we can conclude that the quality of the speech generated by TTS does affect the final performance of S2S system in terms of automatic evaluation, but the impact is still limited.

heading: 4 Conclusion
text: In this paper, we present the system we developed for the IWSLT2023 speech-to-speech competition. The system includes relatively simple and effective ASR and MT modules, as well as a TTS module proposed by us based on the Diffusion Model. In the experiments, we demonstrate that the denoising diffusion process can effectively learn end-to-end TTS task, simplifying both training and inference. However, its generation speed is relatively slow. In our future work, we will continue to optimize its quality and generation efficiency, and further explore the application of diffusion in end-to-end S2S tasks.

heading: Future Work
text: In our future work, we will continue to optimize its quality and generation efficiency, and further explore the application of diffusion in end-to-end S2S tasks