SlowGuess/ACL_2020_OCR huggingface dataset:
rayeren@zju.edu.cn
Jinglin Liu *
jinglinliu@zju.edu.cn
Xu Tan
xuta@microsoft.com
Zhou Zhao†
zhaozhou@zju.edu.cn
Sheng Zhao
Microsoft STC Asia
Sheng.Zhao@microsoft.com
Tie-Yan Liu
tyliu@microsoft.com
Non-autoregressive (NAR) models generate all the tokens of a sequence in parallel, resulting in faster generation speed compared to their autoregressive (AR) counterparts but at the cost of lower accuracy. Different techniques including knowledge distillation and source-target alignment have been proposed to bridge the gap between AR and NAR models in various tasks such as neural machine translation (NMT), automatic speech recognition (ASR), and text to speech (TTS). With the help of those techniques, NAR models can catch up with the accuracy of AR models in some tasks but not in some others. In this work, we conduct a study to understand the difficulty of NAR sequence generation and try to answer: (1) Why NAR models can catch up with AR models in some tasks but not all? (2) Why techniques like knowledge distillation and source-target alignment can help NAR models. Since the main difference between AR and NAR models is that NAR models do not use dependency among target tokens while AR models do, intuitively the difficulty of NAR sequence generation heavily depends on the strength of dependency among target tokens. To quantify such dependency, we propose an analysis model called CoMMA to characterize the difficulty of different NAR sequence generation tasks. We have several interesting findings: 1) Among the NMT, ASR and TTS tasks, ASR has the most target-token dependency while TTS has the least. 2) Knowledge distillation reduces the target-token dependency in target sequence and thus improves the accuracy of NAR models. 3) Source-target alignment constraint encourages dependency
of a target token on source tokens and thus eases the training of NAR models.
Non-autoregressive (NAR) models (Oord et al., 2017; Gu et al., 2017; Chen et al., 2019; Ren et al., 2019), which generate all the tokens in a target sequence in parallel and can speed up inference, are widely explored in natural language and speech processing tasks such as neural machine translation (NMT) (Gu et al., 2017; Lee et al., 2018; Guo et al., 2019a; Wang et al., 2019; Li et al., 2019b; Guo et al., 2019b), automatic speech recognition (ASR) (Chen et al., 2019) and text to speech (TTS) synthesis (Oord et al., 2017; Ren et al., 2019). However, NAR models usually lead to lower accuracy than their autoregressive (AR) counterparts since the inner dependencies among the target tokens are explicitly removed.
Several techniques have been proposed to alleviate the accuracy degradation, including 1) knowledge distillation (Oord et al., 2017; Gu et al., 2017; Guo et al., 2019a,b; Ren et al., 2019), 2) imposing source-target alignment constraint with fertility (Gu et al., 2017), word mapping (Guo et al., 2019a), attention distillation (Li et al., 2019b) and duration prediction (Ren et al., 2019). With the help of those techniques, it is observed that NAR models can match the accuracy of AR models for some tasks (Ren et al., 2019), but the gap still exists for some other tasks (Gu et al., 2017; Chen et al., 2019). Therefore, several questions come out naturally: (1) Why the gap still exists for some tasks? Are some tasks more difficult for NAR generation than others? (2) Why the techniques like knowledge distillation and source-target alignment can help NAR generation?
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 149-159
The main difference between AR and NAR models is that NAR models do not consider the dependency among target tokens, which is also the root cause of accuracy drop of NAR models. Thus, to better understand NAR sequence generation and answer the above questions, we need to characterize and quantify the target-token dependency, which turns out to be non-trivial since the sequences could be of different modalities (i.e., speech or text). For this purpose, we design a novel model called COditional Masked prediction model with Mix-Attention (CoMMA), inspired by the mix-attention in He et al. (2018) and the masked language modeling in Devlin et al. (2018): in CoMMA, (1) the prediction of one target token can attend to all the source and target tokens with mix-attention, and 2) target tokens are randomly masked with varying probabilities. CoMMA can help us to measure target-token dependency using the ratio of the attention weights on target context over that on full (both source and target) context when predicting a target token: bigger ratio, larger dependency among target tokens.
We conduct a comprehensive study in this work and obtain several interesting discoveries that can answer previous questions. First, we find that the rank of the target-token dependency among the three tasks is  $\mathrm{ASR} > \mathrm{NMT} > \mathrm{TTS}$ : ASR has the largest dependency while TTS has the smallest. This finding is consistent with the accuracy gap between AR and NAR models and demonstrates the difficulty of NAR generation across tasks. Second, we replace the target sequence of original training data with the sequence generated by an AR model (i.e., through knowledge distillation) and use the new data to train CoMMA; we find that the target-token dependency is reduced. Smaller target-token dependency makes NAR training easier and thus improves the accuracy. Third, source-target alignment constraint such as explicit duration prediction (Ren et al., 2019) or implicit attention distillation (Li et al., 2019b) also reduces the target-token dependency, thus helping the training of NAR models.
- We design a novel model, conditional masked prediction model with mix-attention (CoMMA), to measure the token dependency for sequence generation.
- With CoMMA, we find that: 1) Among the
three tasks, ASR is the most difficult and TTS is the least for NAR generation; 2) both knowledge distillation and imposing source-target alignment constraint reduce the target-token dependency, and thus reduce the difficulty of training NAR models.
2 CoMMA
In this section, we analyze the token dependency in the target sequence with a novel conditional masked prediction model with mix-attention (CoMMA). We first introduce the design and structure of CoMMA, and then describe how to measure the target token dependency based on CoMMA.
2.1 The Design of CoMMA
It is non-trivial to directly measure and compare the target token dependency in different modalities (i.e., speech or text) and different conditional source modalities (i.e., speech or text). Therefore, we have several considerations in the design of CoMMA: 1) We use masked language modeling in BERT (Devlin et al., 2018) with source condition to train CoMMA, which can help measure the dependency on target context when predicting the current masked token. 2) In order to ensure the dependency on source and target tokens can be comparable, we use mix-attention (He et al., 2018) to calculate the attention weights on both source and target tokens in a single softmax function.
The model architecture of CoMMA is shown in Figure 1. Specifically, CoMMA differs from standard Transformer (Vaswani et al., 2017) as follows: 1) Some tokens are randomly replaced by a special mask token  $\langle M\rangle$  with probability  $p$ , and the model is trained to predict original unmasked tokens. 2) We employ mix-attention mechanism (He et al., 2018) where layer  $i$  in the decoder can attend to itself and the layer  $i$  in the encoder at the same time and compute the attention weights in a single softmax function. We share the parameters of attention and feed-forward layer between the encoder and decoder. 3) Following He et al. (2018), we add source/target embedding to tell the model whether a token is from the source or target sequence, and also add position embedding with the positions of source and target tokens both starting from zero. 4) The encoder and decoder pre-net (Shen et al., 2018) vary in different tasks: For TTS, encoder pre-net consists of only embedding lookup table, and decoder pre-net consists of 2-layer dense network
(a) The main structure of CoMMA.
(b) The input module of CoMMA.
Figure 1: The architecture of conditional masked prediction model with mix-attention (CoMMA).
with ReLU activation. For ASR, encoder pre-net consists of 3-layer 2D convolutional network, and decoder pre-net consists of only embedding lookup table. For NMT, both encoder and decoder pre-net consist of only embedding lookup table.
CoMMA is designed to measure the target token dependency in a variety of sequence generations, including AR (unidirectional) generation, NAR generation, bidirectional generation or even identity copy. To this end, we vary the mask probability  $p$  (the ratio of the masked tokens in the whole target tokens<sup>1</sup>) in a uniform distribution  $p \sim U(0.0, 1.0)$  when training CoMMA. In this way,  $p = 1$  covers NAR generation,  $p = 0$  covers identity copy, and in some cases,  $p$  can also cover AR generation.
2.2 How to Measure Target Token Dependency based on CoMMA
To measure the target token dependency, we define a metric called attention density ratio  $R$ , which represents the ratio of the attention density (the normalized attention weights) on target context in mix-attention when predicting the target token with a well-trained CoMMA. We describe the calculation of  $R$  in the following steps.
First, we define the attention density ratio  $\alpha$  for a single target token  $i$  as
$$
\alpha_ {i} = \frac {\frac {1}{N} \sum_ {j = 1} ^ {N} A _ {i , j}}{\frac {1}{N} \sum_ {j = 1} ^ {N} A _ {i , j} + \frac {1}{M} \sum_ {j = N + 1} ^ {N + M} A _ {i , j}}, \tag {1}
$$
where  $A_{i,j}$  denotes the attention weights from token  $i$  to token  $j$  in mix-attention, and  $i \in [1,N]$  represents the target token while  $j \in [N + 1,N + M]$  represents the source token,  $M$  and  $N$  is the length of source and target sequence respectively,  $\sum_{j = 1}^{N + M} A_{i,j} = 1$ .  $\alpha_i$  represents the ratio of attention density on target context when predicting target token  $i$ .
Second, we average the attention density ratio  $\alpha_{i}$  over all the predicted tokens (with masked probability  $p$ ) in a sentence and get
$$
\frac {1}{| \mathcal {M} ^ {p} |} \sum_ {i \in \mathcal {M} ^ {p}} \alpha_ {i}, \tag {2}
$$
where  $\mathcal{M}^p$  represents the set of masked target tokens under mask probability  $p$  and  $|\mathcal{M}^p|$  denotes the number of tokens in the set.
Third, for a given  $p$ , we calculate  $R(p)$  over all test data and average them to get the final attention density ratio
$$
R (p) = \operatorname {A v g} \left(\frac {1}{\left| \mathcal {M} ^ {p} \right|} \sum_ {i \in \mathcal {M} ^ {p}} \alpha_ {i}\right). \tag {3}
$$
We vary  $p$  and calculate  $R(p)$  to measure the density ratio under different conditions, where a small  $p$  represents more target context that can be leveraged and a large  $p$  represents less context. In the extreme cases,  $p = 1$  represent NAR generation while  $p = 0$  represents to learn identity copy.
Given the proposed attention density ratio  $R(p)$  based on CoMMA, we can measure the target token dependency of the NAR model in different tasks,
<sup>1</sup>Considering the continuity of the mel-spectrogram frames in speech sequence, we mask the frames by chunk, each chunk with frame size 10.
Task	NMT	ASR	TTS
AR	Transformer (Vaswani et al., 2017)	Transformer ASR (Karita et al., 2019)	Transformer TTS (Li et al., 2019a)
NAR	NAT (Gu et al., 2017) w/ AC	NAR-ASR (Chen et al., 2019) w/ AC	FastSpeech (Ren et al., 2019)
which can help understand a series of important research questions, as we introduce in the following three sections.
3 Study on the Difficulty of NAR Generation
In this section, we aim to find out why the gap still exists for ASR and NMT tasks, while in TTS, NAR can catch up with the accuracy of AR model. We also analyze the causes of different difficulties for different tasks. We start from evaluating the accuracy gap between AR and NAR models for NMT, ASR and TTS, and then measure the token dependency based on our proposed CoMMA.
3.1 The Accuracy Gap
We first train the AR and NAR models in each task and check the accuracy gap between AR and NAR models to measure the difficulty of NAR generation in each task.
Configuration of AR and NAR Model The AR and NAR models we considered are shown in Table 1, where we use Transformer as the AR models while the representative NAR models in each task. For a fair comparison, we make some modifications on the NAR models: 1) For ASR, we train a Transformer ASR first as teacher model and then constrain the attention distributions of NAR-ASR with the alignments converted by teacher attention weights, which will be introduced and discussed in Section 5. 2) For NMT, we constrain the KL-divergence of the encoder-to-decoder attention distributions between the AR and NAR models following Li et al. (2019b). We also list the hyperparameters of AR and NAR models for each task in Section A.
Datasets and Evaluations for NMT, ASR and TTS We conduct experiments on IWSLT 2014 German-English (De-En) translation dataset² for NMT, LibriTTS dataset (Zen et al., 2019) for ASR and LJSpeech dataset (Ito) for TTS. For
speech data, we transform the raw audio into melspectrograms following Shen et al. (2018) with  $50~\mathrm{ms}$  frame size and  $12.5~\mathrm{ms}$  hop size. For text data, we tokenize sentences with moses tokenizer<sup>3</sup> and then segment into subword symbols using Byte Pair Encoding (BPE) (Sennrich et al., 2015) for subword-level analysis, and convert the text sequence into phoneme sequence with grapheme-to-phoneme conversion (Sun et al., 2019) for phoneme-level analysis. We use BPE for NMT and ASR, while phoneme for TTS by default unless otherwise stated. We train all models on 2 NVIDIA 2080Ti GPUs using Adam optimizer with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.98$ ,  $\varepsilon = 10^{-9}$  and following the same learning rate schedule in (Vaswani et al., 2017).
For ASR, we evaluate word error rate (WER) on test-clean set in LibriTTS dataset. For NMT, we evaluate the BLEU score on IWSLT 2014 De-En test set. For TTS, we randomly split the LJSpeech dataset into 3 sets: 12500 samples for training, 300 samples for validation and 300 samples for testing, and then evaluate the mean opinion score (MOS) on the test set to measure the audio quality. The output mel-spectrograms of TTS model are transformed into audio samples using the pretrained WaveGlow (Prenger et al., 2019). Each audio is listened by at least 20 testers, who are all native English speakers.
Table 1: The AR and NAR model we consider in each task. "AC" means attention constraint we mentioned in Section 5.
Task	Model	Accuracy
NMT (BLEU/WER)	Transformer	33.90/47.18
NAT	27.12/54.90
ASR (BLEU/WER)	Transformer ASR	66.60/20.10
NAR-ASR	39.23/36.20
TTS (MOS)	Transformer TTS	3.82 ± 0.08
FastSpeech	3.79 ± 0.12
Table 2: The accuracy gap between NAR and AR models.
Results of Accuracy Gap The accuracies of the AR and NAR models in each task are shown in
<sup>3</sup>https://github.com/moses-smt/mosesdecoder/blob/master/scripts/tokenizertokenizer.perl
$^{2}$ https://wit3.fbk.eu/mt.php?release=2014-01
Table 2. It can be seen that NAR model can match the accuracy of AR model gap in TTS, while the gap still exists in ASR and NMT. We calculate both the WER and BLEU metrics in ASR and NMT for better comparison. It can be seen that ASR has a larger gap than NMT. Larger accuracy gap may indicate more difficult for NAR generation in this task. Next, we try to understand what factors influence difficulties among different tasks.
3.2 The Token Dependency
In the last subsection, we analyze the difficulty of NAR models from the perspective of the accuracy gap. In this subsection, we try to find evidence from the target token dependency, which is supposed to be consistent with the accuracy gap to measure the task difficulty.
Configuration of CoMMA We train CoMMA with the same configuration on NMT, ASR and TTS: the hidden size and the feed-forward hidden size and the number of layers are set to 512, 1024 and 6 respectively. We list other hyperparameters of CoMMA in Section B. We also use the same datasets for each task as described in Section 3.1 to train CoMMA.
Results of Token Dependency We use the attention density ratio calculated from CoMMA (as described in Section 2.2) to measure the target token dependency and show the results in Figure 2. It can be seen that the rank of attention density ratio  $R(p)$  is  $\mathrm{ASR} > \mathrm{NMT} > \mathrm{TTS}$  for all  $p$ . Considering that  $R(p)$  measures how much context information from target side is needed to generate a target token, we can see that ASR has more dependency on the target context and less on the source context, while TTS is the opposite, which is consistent with the accuracy gap between AR and NAR models as we described in Section 3.1.
As we vary  $p$  from 0.1 to 0.5,  $R(p)$  decreases for all tasks since more tokens in the target side are masked. We also find that  $R(p)$  in NMT decreases quicker than the other two tasks, which indicates that NMT is good at learning from source context when less context information can be leveraged from the target side while  $R(p)$  in ASR decreases little. This can also explain why NAR in NMT achieves less gap than ASR.
Figure 2: Attention density ratio  $R(p)$  under different  $p$  in different tasks for performance gap analysis.
4 Study on Knowledge Distillation
In the current and next sections, we investigate why some techniques can help NAR generation from the aspect of target token dependency. We only analyze knowledge distillation and attention alignment techniques which are widely used in NAR, but we believe our analysis method can be applied to other NAR techniques, such as iterative refinement (Lee et al., 2018), fine-tuning from an AR model (Guo et al., 2019b) and so on.
Most existing NAR models (Oord et al., 2017; Gu et al., 2017; Wang et al., 2019; Guo et al., 2019a,b; Ren et al., 2019) rely on the technique of knowledge distillation, which generates the new target sequence given original source sequence from a pre-trained AR model and trains the NAR model for better accuracy. In this section, we first conduct experiments to verify the accuracy improvements of knowledge distillation. Next, based on our proposed CoMMA, we analyze why knowledge distillation could help NAR models.
4.1 The Effectiveness of Knowledge Distillation
Knowledge Distillation for NAR Models
Given a well-trained AR model  $\theta_T$  and source sequence  $x \in \mathcal{X}$  from the original training data, a new target sequence can be generated through
$$
y ^ {\prime} \sim P (y | x; \theta_ {T}). \tag {4}
$$
We can use beam search for NMT and ASR and greedy search for TTS to generate  $y'$ . Given the set of generated sequence pairs  $(\mathcal{X},\mathcal{Y}')$ , we train the NAR models with negative log-likelihood loss
$$
\mathcal {L} \left(\left(\mathcal {X}, \mathcal {Y} ^ {\prime}\right); \theta\right) = - \sum_ {\left(x, y ^ {\prime}\right) \in \left(\mathcal {X}, \mathcal {Y} ^ {\prime}\right)} \log P \left(y ^ {\prime} \mid x; \theta\right), \tag {5}
$$
where  $\theta$  is the parameters set of the NAR model.
Task	Model	Accuracy
NMT (BLEU)	Transformer	33.90
NAT	27.12
NAT w/o KD	21.79
TTS (MOS)	Transformer TTS	3.82 ± 0.08
FastSpeech	3.79 ± 0.12
FastSpeech w/o KD	3.58 ± 0.13
Table 3: The comparison between NAR models with and without knowledge distillation.
Experimental Results We only conducted knowledge distillation on NMT and TTS since there is no previous works on ASR yet. We train the NAR models in NMT and TTS with raw target token sequence instead of teacher outputs and compare the results with that in Table 2. The accuracy improvements of knowledge distillation are shown in Table 3. It can be seen that knowledge distillation can boost the accuracy of NAR in NMT and TTS, which is consistent with the previous works.
4.2 Why Knowledge Distillation Works
Recently, Zhou et al. (2019) find that knowledge distillation can reduce the complexity of data sets and help NAT to better model the variations in the output data. However, this explanation is reasonable on its own, but mainly from the perspective of data level and is not easy to understand. In this subsection, we analyze knowledge distillation from a more understandable and intuitive perspective, by observing the change of the token dependency based on our proposed CoMMA.
We measure the target token dependency by training CoMMA with the original training data and new data generated through knowledge distillation, respectively. The results are shown in Figure 3. It can be seen that knowledge distillation can decrease the attention density ratio  $R(p)$  on both tasks, indicating that knowledge distillation can reduce the dependency on the target-side context when predicting a target token, which can be helpful for NAT model training.
5 Study on Alignment Constraint
Without the help of target context, NAR models usually suffer from ambiguous attention to the source context, which affects the accuracy. Re
Figure 3: Attention density ratio  $R(p)$  for NMT and TTS tasks under different  $p$  with and without knowledge distillation, where "KD" means knowledge distillation.
cently, many works have proposed a variety of approaches to help with the source-target alignment of NAR models, which can improve the estimation of the soft alignment in attention mechanism model. For example, Li et al. (2019b) constrain the KL-divergence of the encoder-to-decoder attention distributions between the AR and NAR models. Gu et al. (2017) predict the fertility of the source tokens to approximate the alignments between target sequence and source sequence. Guo et al. (2019a) convert the source token to target token with phrase table or embedding mapping for alignments. Ren et al. (2019) predict the duration (the number of mel-spectrograms) of each phoneme.
In this section, we first study the effectiveness of alignment constraint for NAR models, and then analyze why alignment constraint can help the NAR models by observing the changes of token dependency based on our proposed CoMMA.
5.1 The Effectiveness of Alignment Constraint
Alignment Constraint for NAR Models We choose the attention constraint mechanism which is commonly used based on previous works for each task.
For NMT, we follow Li et al. (2019b) to minimize the KL-divergence between the attention distributions of AR and NAR model as follow:
$$
\mathcal {L} _ {a c} = \frac {1}{N} \sum_ {i = 1} ^ {N} D _ {K L} \left(A _ {i} ^ {\prime} \mid \mid A _ {i}\right), \tag {6}
$$
where  $A_{i}^{\prime}$  and  $A_{i}$  denote the source-target attention weights from the AR teacher model and NAR student model respectively.  $A^{\prime}, A \subset \mathbb{R}^{N \times M}$  where  $N$
and  $M$  are the number of tokens in the target and source sequence.
For TTS, we follow Ren et al. (2019) to extract the encoder-to-decoder attention alignments from the well-trained AR teacher model and convert them to phoneme duration sequence, and then train the duration predictor to expand the hidden of the source sequence to match the length of target sequence.
For ASR, since there is no previous work proposing alignment constraint for NAR, we design a new alignment constraint method and explore its effectiveness. We first calculate the expectation position of teacher's attention distributions for  $i$ -th target token:  $E_{i} = \sum_{j=1}^{M} j * A_{i,j}'$  and cast it to the nearest integer. Then we constrain the attention weights of  $i$ -th target token for NAR model so that it can only attend to the source position between  $E_{i-1}$  and  $E_{i+1}$ . Specially, the first target token can only attend to the source position between 1 and  $E_{2}$  while the last target token can only attend to the position between  $E_{N-1}$  and  $M$ . We apply this alignment constraint for ASR only in the training stage.
Task	Model	Accuracy
NMT (BLEU)	Transformer	33.90
NAT	27.12
NAT w/o AC	25.03
ASR (WER)	Transformer ASR	20.1
NAR-ASR	33.1
NAR-ASR w/o AC	39.23
TTS (MOS)	Transformer TTS	3.82 ± 0.08
FastSpeech	3.79 ± 0.12
FastSpeech w/o AC	1.97 ± 0.16