SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Qingkai Fang $^{1,2}$ , Yan Zhou $^{1,2}$ , Yang Feng $^{1,2*}$
$^{1}$ Key Laboratory of Intelligent Information Processing
Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)
$^{2}$ University of Chinese Academy of Sciences, Beijing, China
{fangqingkai21b,zhouyan23z,fengyang}@ict.ac.cn
Direct speech-to-speech translation (S2ST) translates speech from one language into another using a single model. However, due to the presence of linguistic and acoustic diversity, the target speech follows a complex multimodal distribution, posing challenges to achieving both high-quality translations and fast decoding speeds for S2ST models. In this paper, we propose DASpeech, a non-autoregressive direct S2ST model which realizes both fast and high-quality S2ST. To better capture the complex distribution of the target speech, DASpeech adopts the two-pass architecture to decompose the generation process into two steps, where a linguistic decoder first generates the target text, and an acoustic decoder then generates the target speech based on the hidden states of the linguistic decoder. Specifically, we use the decoder of DA-Transformer as the linguistic decoder, and use FastSpeech 2 as the acoustic decoder. DA-Transformer models translations with a directed acyclic graph (DAG). To consider all potential paths in the DAG during training, we calculate the expected hidden states for each target token via dynamic programming, and feed them into the acoustic decoder to predict the target mel-spectrogram. During inference, we select the most probable path and take hidden states on that path as input to the acoustic decoder. Experiments on the CVSS  $\mathrm{Fr} \rightarrow \mathrm{En}$  benchmark demonstrate that DASpeech can achieve comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while preserving up to  $18.53 \times$  speedup compared to the autoregressive baseline. Compared with the previous non-autoregressive S2ST model, DASpeech does not rely on knowledge distillation and iterative decoding, achieving significant improvements in both translation quality and decoding speed. Furthermore, DASpeech shows the ability to preserve the speaker's voice of the source speech during translation.[23]
Direct speech-to-speech translation (S2ST) directly translates speech of the source language into the target language, which can break the communication barriers between different language groups and has broad application prospects. Traditional S2ST usually consists of cascaded automatic speech recognition (ASR), machine translation (MT), and text-to-speech (TTS) models [1, 2]. In contrast, direct S2ST achieves source-to-target speech conversion with a unified model [3], which can (1) avoid error propagation across sub-models [4]; (2) reduce the decoding latency [5]; and (3) preserve
*Corresponding author: Yang Feng.
2 Audio samples are available at https://ictnlp.github.io/daspeech-demo/.
3Code is publicly available at https://github.com/ictnlp/DASpeech.
non-linguistic information (e.g., the speaker's voice) during translation [6]. Recent works show that direct S2ST can achieve comparable or even better performance than cascaded systems [7, 8].
Despite the theoretical advantages of direct S2ST, it is still very challenging to train a direct S2ST model in practice. Due to the linguistic diversity during translation, as well as the diverse acoustic variations (e.g., duration, pitch, energy, etc.), the target speech follows a complex multimodal distribution. To address this issue, Jia et al. [6], Inaguma et al. [7] propose the two-pass architecture, which first generates the target text with a linguistic decoder, and then uses an acoustic decoder to generate the target speech based on the hidden states of the linguistic decoder. The two-pass architecture decomposes the generation process into two steps: content translation and speech synthesis, making it easier to model the complex distribution of the target speech and achieving state-of-the-art performance among direct S2ST models.
Although the two-pass architecture achieves better translation quality, two passes of autoregressive decoding also incur high decoding latency. To reduce the decoding latency, Huang et al. [9] recently proposes non-autoregressive (NAR) S2ST that generates target speech in parallel. However, due to the conditional independence assumption of NAR models, it becomes more difficult to capture the multimodal distribution of the target speech compared with autoregressive models<sup>4</sup>. Therefore, the trade-off between translation quality and decoding speed of S2ST remains a pressing issue.
In this paper, we introduce an S2ST model with both high-quality translations and fast decoding speeds: DASpeech, a non-autoregressive two-pass direct S2ST model. Like previous two-pass models, DASpeech includes a speech encoder, a linguistic decoder, and an acoustic decoder. Specifically, the linguistic decoder uses the structure of DA-Transformer [11] decoder, which models translations via a directed acyclic graph (DAG). The acoustic decoder adopts the design of FastSpeech 2 [12], which takes the hidden states of the linguistic decoder as input and generates the target mel-spectrogram. During training, we consider all possible paths in the DAG by calculating the expected hidden state for each target token via dynamic programming, which are fed to the acoustic decoder to predict the target mel-spectrogram. During inference, we first find the most probable path in DAG and take hidden states on that path as input to the acoustic decoder. Due to the task decomposition of two-pass architecture, as well as the ability of DA-Transformer and FastSpeech 2 themselves to model linguistic diversity and acoustic diversity, DASpeech is able to capture the multimodal distribution of the target speech. Experiments on the CVSS  $\mathrm{Fr}\rightarrow \mathrm{En}$  benchmark show that: (1) DASpeech achieves comparable or even better performance than the state-of-the-art S2ST model Translatotron 2, while maintaining up to  $18.53\times$  speedup compared to the autoregressive model. (2) Compared with the previous NAR S2ST model TranSpeech [9], DASpeech no longer relies on knowledge distillation and iterative decoding, achieving significant advantages in both translation quality and decoding speed. (3) When training on speech-to-speech translation pairs of the same speaker, DASpeech emerges with the ability to preserve the source speaker's voice during translation.
2.1 Directed Acyclic Transformer
Directed Acyclic Transformer (DA-Transformer) [11, 13] is proposed for non-autoregressive machine translation (NAT), which achieves comparable results to autoregressive Transformer [14] without relying on knowledge distillation. DA-Transformer consists of a Transformer encoder and an NAT decoder. The hidden states of the last decoder layer are organized as a directed acyclic graph (DAG). The hidden states correspond to vertices of the DAG, and there are unidirectional edges that connect vertices with small indices to those with large indices. DA-Transformer successfully alleviates the linguistic multi-modality problem since DAG can capture multiple translations simultaneously by assigning different translations to different paths in DAG.
Formally, given a source sequence  $X = (x_{1},\dots,x_{N})$  and a target sequence  $Y = (y_{1},\dots,y_{M})$ , the encoder takes  $X$  as input and the decoder takes learnable positional embeddings  $\mathbf{G} = (\mathbf{g}_1,\dots,\mathbf{g}_L)$  as input. Here  $L$  is the graph size, which is set to  $\lambda$  times the source length, i.e.,  $L = \lambda \cdot N$ , and  $\lambda$  is a hyperparameter. DA-Transformer models the translation probability  $P_{\theta}(Y|X)$  by marginalizing all
It is known as the multi-modality problem [10] in non-autoregressive sequence generation.
possible paths in DAG:
$$
P _ {\theta} (Y | X) = \sum_ {A \in \Gamma} P _ {\theta} (Y | A, X) P _ {\theta} (A | X), \tag {1}
$$
where  $A = (a_{1},\dots,a_{M})$  is a path represented by a sequence of vertex indexes with  $1 = a_{1} < \dots < a_{M} = L$ , and  $\Gamma$  contains all paths with the same length as the target sequence  $Y$ . The probability of path  $A$  is defined as:
$$
P _ {\theta} (A | X) = \prod_ {i = 1} ^ {M - 1} P _ {\theta} \left(a _ {i + 1} \mid a _ {i}, X\right) = \prod_ {i = 1} ^ {M - 1} \mathbf {E} _ {a _ {i}, a _ {i + 1}}, \tag {2}
$$
where  $\mathbf{E} \in \mathbb{R}^{L \times L}$  is the transition probability matrix. We apply lower triangular masking on  $\mathbf{E}$  to allow only forward transitions. With the selected path  $A$ , all target tokens are predicted in parallel:
$$
P _ {\theta} (Y | A, X) = \prod_ {i = 1} ^ {M} P _ {\theta} \left(y _ {i} \mid a _ {i}, X\right) = \prod_ {i = 1} ^ {M} \mathbf {P} _ {a _ {i}, y _ {i}}, \tag {3}
$$
where  $\mathbf{P} \in \mathbb{R}^{L \times |\mathbb{V}|}$  is the prediction probability matrix, and  $\mathbb{V}$  indicates the vocabulary. Finally, we train the DA-Transformer by minimizing the negative log-likelihood loss:
$$
\mathcal {L} _ {\mathrm {D A T}} = - \log P _ {\theta} (Y | X) = - \log \sum_ {A \in \Gamma} P _ {\theta} (Y | A, X) P _ {\theta} (A | X), \tag {4}
$$
which can be calculated with dynamic programming.
2.2 FastSpeech 2
FastSpeech 2 [12] is a non-autoregressive text-to-speech (TTS) model that generates mel-spectrograms from input phoneme sequences in parallel. It is composed of three stacked modules: encoder, variance adaptor, and mel-spectrogram decoder. The encoder and mel-spectrogram decoder consist of several feed-forward Transformer blocks, each containing a self-attention layer followed by a 1D-convolitional layer. The variance adaptor contains three variance predictors including duration predictor, pitch predictor, and energy predictor, which are used to reduce the information gap between input phoneme sequences and output mel-spectrograms. During training, the ground truth duration, pitch and energy are used to train these variance predictors and also as conditional inputs to generate the mel-spectrogram. During inference, we use the predicted values of these variance predictors. The introduction of variation information greatly alleviates the acoustic multi-modality problem, which leads to better voice quality. The training objective of FastSpeech 2 consists of four terms:
$$
\mathcal {L} _ {\mathrm {T T S}} = \mathcal {L} _ {\mathrm {L 1}} + \mathcal {L} _ {\mathrm {d u r}} + \mathcal {L} _ {\mathrm {p i t c h}} + \mathcal {L} _ {\mathrm {e n e r g y}}, \tag {5}
$$
where  $\mathcal{L}_{\mathrm{L1}}$  measures the L1 distance between the predicted and ground truth mel-spectrograms,  $\mathcal{L}_{\mathrm{dur}}$ ,  $\mathcal{L}_{\mathrm{pitch}}$  and  $\mathcal{L}_{\mathrm{energy}}$  compute the mean square error (MSE) loss between predictions and ground truth for duration, pitch, and energy, respectively.
3 DASpeech
In this section, we introduce DASpeech, a non-autoregressive two-pass direct S2ST model that generates target phonemes and target mel-spectrograms successively. Formally, the source speech sequence is denoted as  $X = (x_{1},\dots,x_{N})$ , where  $N$  is the number of frames in the source speech. The sequences of target phoneme and target mel-spectrogram are represented by  $Y = (y_{1},\dots,y_{M})$  and  $S = (s_1,\ldots ,s_T)$ , respectively. DASpeech first generates  $Y$  from  $X$  with a speech-to-text translation (S2TT) $^{5}$  DA-Transformer. Subsequently, it generates  $S$  with a FastSpeech 2-style decoder conditioned on the last-layer hidden states of the DA-Transformer. We first overview the model architecture of DASpeech in Section 3.1. In Section 3.2, we introduce our proposed training techniques that leverage pretrained S2TT DA-Transformer and FastSpeech 2 models and finetune the entire model for S2ST end-to-end. Finally, we present several decoding algorithms for DASpeech in Section 3.3.
Figure 1: Overview of DASpeech. The last-layer hidden states of the linguistic decoder are organized as a DAG. During training, the input to the acoustic decoder is the sequence of expected hidden states. During inference, it is the sequence of hidden states on the most probable path.
As shown in Figure 1, DASpeech consists of three parts: a speech encoder, a non-autoregressive linguistic decoder, and a non-autoregressive acoustic decoder. Below are the details of each part.
Speech Encoder Since our model takes speech features as input, we replace the Transformer encoder in the original DA-Transformer with a speech encoder. The speech encoder contains a subsampler followed by several Conformer blocks [15]. Specifically, the subsampler consists of two 1D-convolutional layers which shrink the length of input sequences by a factor of 4. Conformer combines multi-head attention modules and convolutional layers together to capture both global and local features. We use relative positional encoding [16] in the multi-head attention module.
Non-autoregressive Linguistic Decoder The linguistic decoder is identical to the decoder of DA-Transformer, which generates the target phoneme sequence from the source speech in parallel. Each decoder layer comprises a self-attention layer, a cross-attention layer, and a feed-forward layer. The decoder takes learnable positional embeddings as input, and the last-layer hidden states are organized as a DAG to model translations, as we described in Section 2.1.
Non-autoregressive Acoustic Decoder The acoustic decoder adopts the design of FastSpeech 2, which generates target mel-spectrograms from the last-layer hidden states of DA-Transformer in parallel. The model architecture is the same as that introduced in Section 2.2, except that the embedding matrix of the input phonemes is removed, since the input has changed from the phoneme sequence to the hidden state sequence.
DASpeech has the advantage of easily utilizing pretrained S2TT DA-Transformer and FastSpeech 2 models. We use the pretrained S2TT DA-Transformer to initialize the speech encoder and the non-autoregressive linguistic decoder, and use the pretrained FastSpeech 2 model to initialize the non-autoregressive acoustic decoder. Finally, the entire model is finetuned for direct S2ST. This pretraining-finetuning pipeline simplifies model training and enables the use of additional S2TT and TTS data. However, end-to-end finetuning presents a major challenge due to the length discrepancy between the output from the linguistic decoder and the input to the acoustic decoder. Specifically, the hidden state sequence output by the linguistic decoder has a length of  $L = \lambda \cdot N$ , while the input sequence required by the acoustic decoder should have a length of  $M$ , which is the length of the ground truth phoneme sequence. Therefore, it is necessary to determine how to obtain the input
In this work, speech-to-text translation refers to speech-to-phoneme translation if not otherwise specified.
sequence of acoustic decoder  $\mathbf{Z} = (\mathbf{z}_1, \dots, \mathbf{z}_M)$  from the last-layer hidden states of linguistic decoder  $\mathbf{V} = (\mathbf{v}_1, \dots, \mathbf{v}_L)$ . The following introduces our proposed approach: Expect-Path Training.
Expect-Path Training Intuitively, the  $i$ -th input element  $\mathbf{z}_i$  should be the hidden state of the vertex responsible for generating  $y_i$ . However, since there may be multiple vertices capable of generating each  $y_i$  due to numerous possible paths, we would like to consider all potential paths. To address this issue, we define  $\mathbf{z}_i$  as the expected hidden state under the posterior distribution  $P_{\theta}(a_i|X,Y)$ :
$$
\mathbf {z} _ {i} = \sum_ {j = 1} ^ {L} P _ {\theta} \left(a _ {i} = j | X, Y\right) \cdot \mathbf {v} _ {j}, \tag {6}
$$
where  $P_{\theta}(a_i = j|X,Y)$  refers to the probability of vertex  $j$  being the  $i$ -th vertex on path  $A$ , which means that  $y_i$  is generated by vertex  $j$ . We can compute  $P_{\theta}(a_i = j|X,Y)$  as follows:
$$
\begin{array}{l} P _ {\theta} \left(a _ {i} = j \mid X, Y\right) = \sum_ {A \in \Gamma} \mathbb {1} \left(a _ {i} = j\right) \cdot P _ {\theta} (A \mid X, Y) (7) \\ = \sum_ {A \in \Gamma} \mathbb {1} (a _ {i} = j) \cdot \frac {P _ {\theta} (Y , A | X)}{\sum_ {A ^ {\prime} \in \Gamma} P _ {\theta} (Y , A ^ {\prime} | X)} (8) \\ = \frac {\sum_ {A \in \Gamma} \mathbb {1} (a _ {i} = j) \cdot P _ {\theta} (Y , A | X)}{\sum_ {A \in \Gamma} P _ {\theta} (Y , A | X)}, (9) \\ \end{array}
$$
where  $\mathbb{1}(a_i = j)$  is an indicator function to indicate whether the  $i$ -th vertex of path  $A$  is vertex  $j$ . To calculate  $\sum_{A\in \Gamma}\mathbb{1}(a_i = j)\cdot P_\theta (Y,A|X)$  and  $\sum_{A\in \Gamma}P_{\theta}(Y,A|X)$  in Equation (9), we employ the forward-backward algorithm [17], which involves two passes of dynamic programming.
Forward Algorithm The forward probability is defined as  $\alpha_{i}(j) = P_{\theta}(y_{1},\dots,y_{i},a_{i} = j|X)$ , which is the probability of generating the partial target sequence  $(y_{1},\dots,y_{i})$  and ending in vertex  $j$  at the  $i$ -th step. By definition, we have  $\alpha_{1}(1) = \mathbf{P}_{1,y_{1}}$  and  $\alpha_{1}(1 < j \leq L) = 0$ . Due to the Markov property, we can sequentially calculate  $\alpha_{i}(\cdot)$  from its previous step  $\alpha_{i-1}(\cdot)$  as follows:
$$
\alpha_ {i} (j) = \mathbf {P} _ {j, y _ {i}} \sum_ {k = 1} ^ {j - 1} \alpha_ {i - 1} (k) \cdot \mathbf {E} _ {k, j}. \tag {10}
$$
Backward Algorithm The backward probability is defined as  $\beta_{i}(j) = P_{\theta}(y_{i + 1},\dots,y_{M}|a_{i} = j,X)$ , which is the probability of starting from vertex  $j$  at the  $i$ -th step and generating the rest of the target sequence  $(y_{i + 1},\dots,y_{M})$ . By definition, we have  $\beta_{M}(L) = 1$  and  $\beta_{M}(1\leq j < L) = 0$ . Similar to the forward algorithm, we can sequentially calculate  $\beta_{i}(j)$  from its next step  $\beta_{i + 1}(j)$  as follows:
$$
\beta_ {i} (j) = \sum_ {k = j + 1} ^ {L} \mathbf {E} _ {j, k} \cdot \beta_ {i + 1} (k) \cdot \mathbf {P} _ {k, y _ {i + 1}}. \tag {11}
$$
Recalling Equation (9), the denominator is the sum of the probabilities of all valid paths, which is equal to  $\alpha_{M}(L)$ . The numerator is the sum of the probabilities of all paths with  $a_{i} = j$ , which is equal to  $\alpha_{i}(j) \cdot \beta_{i}(j)$ . Therefore, the Equation (6) can be calculated as:
$$
\mathbf {z} _ {i} = \sum_ {j = 1} ^ {L} P _ {\theta} \left(a _ {i} = j \mid X, Y\right) \cdot \mathbf {v} _ {j} = \sum_ {j = 1} ^ {L} \frac {\alpha_ {i} (j) \cdot \beta_ {i} (j)}{\alpha_ {M} (L)} \cdot \mathbf {v} _ {j}. \tag {12}
$$
The time complexity of the forward-backward algorithm is  $\mathcal{O}(ML^2)$ . Finally, the training objective of DASpeech is as follows:
$$
\mathcal {L} _ {\mathrm {D A S p e e c h}} = \mathcal {L} _ {\mathrm {D A T}} + \mu \cdot \mathcal {L} _ {\mathrm {T T S}}, \tag {13}
$$
where  $\mu$  is the weight of TTS loss. The definitions of  $\mathcal{L}_{\mathrm{DAT}}$  and  $\mathcal{L}_{\mathrm{TTS}}$  are the same as those in Equations (4) and (5).
3.3 Inference
During inference, we perform two-pass parallel decoding. First, we find the most probable path  $A^{*}$  in DAG with one of the decoding strategies proposed for DA-Transformer (see details below). We then
feed the last-layer hidden states on path  $A^{*}$  to the non-autoregressive acoustic decoder to generate the mel-spectrogram. Finally, the predicted mel-spectrogram will be converted into waveform using a pretrained HiFi-GAN vocoder [18]. Since both DAG and TTS decoding are fully parallel, DASpeech achieves significant improvements in decoding efficiency compared to previous two-pass models which rely on two passes of autoregressive decoding. Considering the trade-off between translation quality and decoding efficiency, we use the following two decoding strategies for DA-Transformer in our experiments: Lookahead and Joint-Viterbi.
Lookahead Lookahead decoding sequentially chooses  $a_i$  and  $y_i$  in a greedy way. At each decoding step, it jointly considers the transition probability and the prediction probability:
$$
a _ {i} ^ {*}, y _ {i} ^ {*} = \underset {a _ {i}, y _ {i}} {\arg \max } P _ {\theta} \left(y _ {i} \mid a _ {i}, X\right) P _ {\theta} \left(a _ {i} \mid a _ {i - 1}, X\right). \tag {14}
$$
Joint-Viterbi Joint-Viterbi decoding [19] finds the global joint optimal solution of the translation and decoding path via Viterbi decoding [20]:
$$
A ^ {*}, Y ^ {*} = \underset {A, Y} {\arg \max } P _ {\theta} (Y, A | X). \tag {15}
$$
After Viterbi decoding, we first decide the target length  $M$  and obtain the optimal path by backtracking from  $a_{M}^{*} = L$ . More details can be found in the original paper.
Dataset We conduct experiments on the CVSS dataset [4], a large-scale S2ST corpus containing speech-to-speech translation pairs from 21 languages to English. It is extended from the CoVoST 2 [21] S2TT corpus by synthesizing the target text into speech with state-of-the-art TTS models. It includes two versions: CVSS-C and CVSS-T. For CVSS-C, all target speeches are in a single speaker's voice. For CVSS-T, the target speeches are in voices transferred from the corresponding source speeches. We evaluate the models on the CVSS-C French  $\rightarrow$  English (Fr  $\rightarrow$  En) and CVSS-T French  $\rightarrow$  English (Fr  $\rightarrow$  En) datasets. We also conduct a multilingual experiment by combining all 21 language directions in CVSS-C together to train a single many-to-English S2ST model.
Pre-processing We convert the source speech to  $16000\mathrm{Hz}$  and generate target speech with  $22050\mathrm{Hz}$ . We compute the 80-dimensional mel-filterbank features for the source speech, and transform the target waveform into mel-spectrograms following Ren et al. [12]. We apply utterance-level and global-level cepstral mean-variance normalization for source speech and target speech, respectively. We follow Ren et al. [12] to extract the duration, pitch, and energy information of the target speech.
Model Configurations The speech encoder, linguistic decoder, and acoustic decoder contain 12 Conformer layers, 4 Transformer decoder layers, and 8 feed-forward Transformer blocks, respectively. The detailed configurations can be found in Table 5 in Appendix A. For model regularization, we set dropout to 0.1 and weight decay to 0.01, and no label smoothing is used. We use the HiFi-GAN vocoder pretrained on the VCTK dataset [22] to convert the mel-spectrogram into waveform.
Training DASpeech follows the pretraining-finetuning pipeline. During pretraining, the speech encoder and the linguistic decoder are trained on the S2TT task for 100k updates with a batch of 320k audio frames. The learning rate warms up to 5e-4 within 10k steps. The acoustic decoder is pretrained on the TTS task for 100k updates with a batch size of 512. The learning rate warms up to 5e-4 within 4k steps. During finetuning, we train the entire model for 50k updates with a batch of 320k audio frames. The learning rate warms up to 1e-3 within 4k steps. We use Adam optimizer [23] for both pretraining and finetuning. For the weight of TTS loss  $\mu$ , we experiment with  $\mu \in \{1.0, 2.0, 5.0, 10.0\}$  and choose  $\mu = 5.0$  according to results on the dev set. We implement our model with the open-source toolkit fairseq [24]. All models are trained on 4 RTX 3090 GPUs.
In the multilingual experiment, the presence of languages with limited data or substantial interlingual variations makes the mapping from source speech to target phonemes particularly challenging. To address this, we adopt a two-stage pretraining strategy. Initially, we pretrain the speech encoder and
See VCTK_V1 in https://github.com/jik876/hifi-gan.