SlowGuess/NAACL_2021_OCR huggingface dataset:
Yu Bao†\* Shujian Huang†\* Tong Xiao‡ Dongqi Wang†\* Xinyu Dai†\* Jiajun CHEN†\*
†National Key Laboratory for Novel Software Technology, Nanjing University  
*Collaborative Innovation Center of Novel Software Technology and Industrialization  
‡NiuTrans Co., Ltd., Shenyang, China
{baoy, wangdq}  $@$  smail.nju.edu.cn, xiaotong@mail.neu.edu.cn
{huangsj,daixinyu,chenjj}@nju.edu.cn
Non-autoregressive Transformer is a promising text generation model. However, current non-autoregressive models still fall behind their autoregressive counterparts in translation quality. We attribute this accuracy gap to the lack of dependency modeling among decoder inputs. In this paper, we propose CNAT, which learns implicitly categorical codes as latent variables into the non-autoregressive decoding. The interaction among these categorical codes remedies the missing dependencies and improves the model capacity. Experiment results show that our model achieves comparable or better performance in machine translation tasks than several strong baselines.
Non-autoregressive Transformer (NAT, Gu et al., 2018; Wang et al., 2019; Lee et al., 2018; Ghazvininejad et al., 2019) is a promising text generation model for machine translation. It introduces the conditional independent assumption among the target language outputs and simultaneously generates the whole sentence, bringing in a remarkable efficiency improvement (more than  $10 \times$  speed-up) versus the autoregressive model. However, the NAT models still lay behind the autoregressive models in terms of BLEU (Papineni et al., 2002) for machine translation. We attribute the low-quality of NAT models to the lack of dependencies modeling for the target outputs, making it harder to model the generation of the target side translation.
A promising way is to model the dependencies of the target language by the latent variables. A line of research works (Kaiser et al., 2018; Roy et al., 2018; Shu et al., 2019; Ma et al., 2019) introduce latent variable modeling to the non-autoregressive Transformer and improves translation quality. The latent variables could be regarded as the springboard to bridge the modeling gap, introducing more informative decoder inputs than the previ
ously copied inputs. More specifically, the latent-variable based model first predicts a latent variable sequence conditioned on the source representation, where each variable represents a chunk of words. The model then simultaneously could generate all the target tokens conditioning on the latent sequence and the source representation since the target dependencies have been modeled into the latent sequence.
However, due to the modeling complexity of the chunks, the above approaches always rely on a large number (more than  $2^{15}$ , Kaiser et al., 2018; Roy et al., 2018) of latent codes for discrete latent spaces, which may hurt the translation efficiency—the essential goal of non-autoregressive decoding.
Akoury et al. (2019) introduce syntactic labels as a proxy to the learned discrete latent space and improve the NATs' performance. The syntactic label greatly reduces the search space of latent codes, leading to a better performance in both quality and speed. However, it needs an external syntactic parser to produce the reference syntactic tree, which may only be effective in limited scenarios. Thus, it is still challenging to model the dependency between latent variables for non-autoregressive decoding efficiently.
In this paper, we propose to learn a set of latent codes that can act like the syntactic label, which is learned without using the explicit syntactic trees. To learn these codes in an unsupervised way, we use each latent code to represent a fuzzy target category instead of a chunk as the previous research (Akoury et al., 2019). More specifically, we first employ vector quantization (Roy et al., 2018) to discretize the target language to the latent space with a smaller number (less than 128) of latent variables, which can serve as the fuzzy word-class information each target language word. We then model the latent variables with conditional random fields (CRF, Lafferty et al., 2001; Sun et al., 2019). To avoid the mismatch of the training and
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5749-5759 June 6-11, 2021. ©2021 Association for Computational Linguistics
inference for latent variable modeling, we propose using a gated neural network to form the decoder inputs. Equipping it with scheduled sampling (Bengio et al., 2015), the model works more robustly.
Experiment results on WMT14 and IWSLT14 show that CNAT achieves the new state-of-the-art performance without knowledge distillation. With the sequence-level knowledge distillation and reranking techniques, the CNAT is comparable to the current state-of-the-art iterative-based model while keeping a competitive decoding speedup.
Neural machine translation (NMT) is formulated as a conditional probability model  $p(\pmb{y}|\pmb{x})$ , which models a sentence  $\pmb{y} = \{y_1, y_2, \dots, y_m\}$  in the target language given the input  $\pmb{x} = \{x_1, x_2, \dots, x_n\}$  from the source language.
2.1 Non-Autoregressive Neural Machine Translation
Gu et al. (2018) proposes Non-Autoregressive Transformer (NAT) for machine translation, breaking the dependency among target tokens, thus achieving simultaneous decoding for all tokens. For a source sentence, a non-autoregressive decoder factorizes the probability of its target sentence as:
$$
p (\boldsymbol {y} | \boldsymbol {x}) = \prod_ {t = 1} ^ {m} p \left(y _ {t} | \boldsymbol {x}; \theta\right), \tag {1}
$$
where  $\theta$  is the set of model parameters.
NAT has a similar architecture to the autoregressive Transformer (AT, Vaswani et al., 2017), which consists of a multi-head attention based encoder and decoder. The model first encodes the source sentence  $x_{1:n}$  as the contextual representation  $e_{1:n}$ , then employs an extra module to predict the target length and form the decoder inputs.
- Length Prediction: Specifically, the length predictor in the bridge module predicts the target sequence length  $m$  by:
$$
m = n + \underset {\Delta L} {\arg \max } p (\Delta_ {L} | \text {m e a n} (e); \phi), \tag {2}
$$
where  $\Delta_L$  is the length difference between the target and source sentence,  $\phi$  is the parameter of length predictor.
(a) AT
(b) NAT
Figure 1: Different inference process of different Transformer models.
(c) LT
- Inputs Initialization: With the target sequence length  $m$ , we can compute the decoder inputs  $h = h_{1:m}$  with Softcopy (Li et al., 2019; Wei et al., 2019) as:
$$
h _ {j} = \sum_ {i} ^ {n} w _ {i j} \cdot e _ {i} \tag {3}
$$
$$
\mathrm {a n d} w _ {i j} = \mathrm {s o f t m a x} (- | j - i | / \tau),
$$
where  $\tau$  is a hyper-parameter to control the sharpness of the softmax function.
With the computed decoder inputs  $h$ , NAT generates target sequences simultaneously by  $\arg \max_{y_t} p(y_t | x; \theta)$  for each timestep  $t$ , effectively reduce computational overhead in decoding (see Figure 1b).
Though NAT achieves around  $10 \times$  speedup in machine translation than autoregressive models, it still suffers from potential performance degradation (Gu et al., 2018). The results degrade since the removal of target dependencies prevents the decoder from leveraging the inherent sentence structure in prediction. Moreover, taking the copied source representation as decoder inputs implicitly assume that the source and target language share a similar order, which may not always be the case (Bao et al., 2019).
2.2 Latent Transformer
To bridge the gap between non-autoregressive and autoregressive decoding, Kaiser et al. (2018) introduce the Latent Transformer (LT). It incorporates non-autoregressive decoding with conditional dependency as the latent variable to alleviate the degradation resulted from the absence of dependency:
$$
p (\boldsymbol {y} | \boldsymbol {x}) = p (\boldsymbol {z} | \boldsymbol {x}; \phi) \prod_ {t = 1} ^ {m} p \left(y _ {t} | \boldsymbol {z}, \boldsymbol {x}; \theta\right), \tag {4}
$$
where  $z = \{z_{1},\dots ,z_{L}\}$  is the latent variable sequence and the  $L$  is the length of the latent sequence,  $\phi$  and  $\theta$  are the parameter of latent predictor and translation model, respectively.
The LT architecture stays unchanged from the origin NAT models, except for the latent predictor and decoder inputs. During inference, the Latent Transformer first autoregressively predicts the latent variables  $z$ , then non-autoregressively produces the entire target sentence  $y$  conditioned on the latent sequence  $z$  (see Figure 1c). Ma et al. (2019); Shu et al. (2019) extend this idea and model  $z$  as the continuous latent variables, achieving a promising result, which replaces the autoregressive predictor with the iterative transformation layer.
In this section, we present our proposed CNAT, an extension to the Transformer incorporated with non-autoregressive decoding for target tokens and autoregressive decoding for latent sequences.
In brief, CNAT follows the architecture of Latent Transformer (Kaiser et al., 2018), except for the latent variable modeling (in § 3.1 and § 3.2) and inputs initialization (in § 3.3).
3.1 Modeling Target Categorical Information by Vector Quantization
Categorical information has achieved great success in neural machine translation, such as part-of-speech (POS) tag in autoregressive translation (Yang et al., 2019) and syntactic label in non-autoregressive translation (Akoury et al., 2019).
Inspired by the broad application of categorical information, we propose to model the implicit categorical information of target words in a nonautoregressive Transformer. Each target sequence  $\mathbf{y} = y_{1:m}$  will be assigned to a discrete latent variable sequence  $\mathbf{z} = z_{1:m}$ . We assume that each  $z_{i}$  will capture the fuzzy category of its token  $y_{i}$ . Then, the conditional probability  $p(\mathbf{y}|\mathbf{x})$  is factorized with respect to the categorical latent variable:
$$
p (\boldsymbol {y} | \boldsymbol {x}) = \sum_ {\boldsymbol {z}} p (\boldsymbol {z} | \boldsymbol {x}) \cdot p (\boldsymbol {y} | \boldsymbol {z}, \boldsymbol {x}). \tag {5}
$$
However, it is computationally intractable to sum all configurations of latent variables. Following the spirit of the latent based model (Kaiser et al., 2018; Roy et al., 2018), we employ a vector quantized technique to maintain differentiability through the categorical modeling and learn the latent variables straightforward.
Vector Quantization. The vector quantization based methods have a long history of being successfully in machine learning models. In vector
quantization, each target representation  $\mathrm{repr}(y_i)\in$ $\mathbb{R}^{d_{\mathrm{model}}}$  is passed through a discretization bottleneck using a nearest-neighbor lookup on embedding matrix  $\mathbf{Q}\in \mathbb{R}^{K\times d_{\mathrm{model}}}$ , where  $K$  is the number of categorical codes.
For each  $y_{i}$  in the target sequence, we define its categorical variable  $z_{i}$  and latent code  $q_{i}$  as:
$$
z _ {i} = k, q _ {i} = \mathbf {Q} _ {k},
$$
$$
\text {a n d} k = \underset {j \in [ K ]} {\arg \min } | | \operatorname {r e p r} \left(y _ {i}\right) - \mathbf {Q} _ {j} | | _ {2}, \tag {6}
$$
where  $||\cdot ||_2$  is the  $l_{2}$  distance,  $[K]$  denote the set  $\{1,2,\dots ,K\}$ . Intuitively, we adopt the embedding of  $\pmb{y}$  as the target representation:
$$
\operatorname {r e p r} \left(y _ {i}\right) = \operatorname {e m b e d d i n g} \left(y _ {i}\right)
$$
where the embedding matrix of the target language is shared with the softmax layer of the decoder.
Exponential Moving Average. Following the common practice of vector quantization, we also employ the exponential moving average (EMA) technique to regularize the categorical codes.
Put simply, the EMA technique could be understood as basically the k-means clustering of the hidden states with a sort of momentum. We maintain an EMA over the following two quantities for each  $j \in [K]$ : 1) the count  $c_{j}$  measuring the number of target representations that have  $\mathbf{Q}_j$  as its nearest neighbor, and 2)  $\mathbf{Q}_j$ . The counts are updated over a mini-batch of targets  $\{y_1, y_2, \dots, y_{m \times B}\}$  with:
$$
c _ {j} = \lambda c _ {j} + (1 - \lambda) \sum_ {i} ^ {m \times B} 1 \left[ z _ {i} = j \right], \tag {7}
$$
then, the latent code  $\mathbf{Q}_j$  being updated with:
$$
\mathbf {Q} _ {j} = \lambda \mathbf {Q} _ {j} + (1 - \lambda) \sum_ {i} ^ {m \times B} \frac {1 \left[ z _ {i} = j \right] \operatorname {r e p r} \left(y _ {i}\right)}{c _ {j}}, \tag {8}
$$
where  $1[\cdot ]$  is the indicator function and  $\lambda$  is a decay parameter,  $B$  is the size of the batch.
3.2 Modeling Categorical Sequence with Conditional Random Fields
Our next insight is transferring the dependencies among the target outputs into the latent spaces. Since the categorical variable captures the fuzzy target class information, it can be a proxy of the target outputs. We further employ a structural prediction module instead of the standard autoregressive
Transformer to model the latent sequence. The former can explicitly model the dependencies among the latent variables and performs exact decoding during inference.
Conditional Random Fields. We employ a linear-chain conditional random fields (CRF, Lafferty et al., 2001) to model the categorical latent variables, which is the most common structural prediction model.
Given the source input  $\pmb{x} = (x_{1},\dots ,x_{n})$  and its corresponding latent variable sequence  $z = (z_{1},\dots ,z_{m})$ , the CRF model defines the probability of  $\pmb{z}$  as:
$$
\begin{array}{l} p (\boldsymbol {z} | \boldsymbol {x}) = \frac {1}{\mathbb {Z} (\boldsymbol {x})} \exp \left(\sum_ {i = 1} ^ {m} s \left(z _ {i}, \boldsymbol {x}, i\right) \right. \tag {9} \\ \left. + \sum_ {i = 2} ^ {m} t (z _ {i - 1}, z _ {i}, \boldsymbol {x}, i)\right), \\ \end{array}
$$
where  $\mathbb{Z}(\pmb {x})$  is the normalize factor,  $s(z_{i},\pmb {x},i)$  is the emit score of  $z_{i}$  at the position  $i$  , and the  $t(z_{i - 1},z_i,\pmb {x},i)$  is the transition score from  $z_{i - 1}$  to  $z_{i}$
Before computing the emit score and transition score in Eq. 9, we first take  $h = h_{1:m}$  as the inputs and compute the representation  $f = \mathrm{Transfer}(h)$ , where  $\mathrm{Transfer}(\cdot)$  denotes a two-layer vanilla Transformer decoding function including a self-attention block, an encoder-decoder block followed by a feed-forward neural network block (Vaswani et al., 2017).
We then compute the emit score and the transition score. For each position  $i$ , we compute the emit score with a linear transformation:  $s(z_{i}, \boldsymbol{x}, i) = (W^{T}f_{i} + b)_{z_{i}}$  where  $W \in \mathbb{R}^{d_{\mathrm{model}} \times K}$  and  $b \in \mathbb{R}^{K}$  are the parameters. We incorporate the positional context and compute its transition score with:
$$
M _ {\mathrm {d}} ^ {i} = \operatorname {B i a f f i n e} ([ f _ {i - 1}; f _ {i} ]),
$$
$$
\boldsymbol {M} ^ {i} = \boldsymbol {E} _ {1} ^ {T} \boldsymbol {M} _ {\mathrm {d}} ^ {i} \boldsymbol {E} _ {2}, \tag {10}
$$
$$
t (z _ {i - 1}, z _ {i}, \pmb {x}, i) = M _ {z _ {i - 1}, z _ {i}} ^ {i},
$$
where  $\mathrm{Biaffine}(\cdot):\mathbb{R}^{2d_{\mathrm{model}}}\to \mathbb{R}^{d_{\mathrm{t}}\times d_{\mathrm{t}}}$  is a biaffine neural network (Dozat and Manning, 2017),  $E_{1}$  and  $E_{2}\in \mathbb{R}^{d_{\mathrm{t}}\times K}$  are the transition matrix.
3.3 Fusing Source Inputs and Latent Codes via Gated Function
One potential issue is that the mismatch of the training and inference stage for the used categorical
variables. Suppose we train the decoder with the quantized categorical variables  $\mathbf{z}$ , which is inferred from the target reference. In that case, we may fail to achieve satisfactory performance with the predicted categorical variables during inference.
We intuitively apply the gated neural network (denote as GateNet) to form the decoder inputs by fusing the copied decoder inputs  $h = h_{1:m}$  and the latent codes  $q = q_{1:m}$ , since the copied decoder inputs  $h$  is still informative to non-autoregressive decoding:
$$
g _ {i} = \sigma (\text {F F N} ([ h _ {i}; q _ {i} ])), \tag {11}
$$
$$
o _ {i} = h _ {i} * g _ {i} + q (z _ {i}) * (1 - g _ {i}),
$$
where the  $\mathrm{FFN}(\cdot):\mathbb{R}^{2d_{\mathrm{model}}}\to \mathbb{R}^{d_{\mathrm{model}}}$  is a two-layer feed-forward neural networks and  $\sigma (\cdot)$  is the sigmoid function.
While training, we first compute the reference  $z^{\mathrm{ref}}$  by the vector quantization and employ the EMA to update the quantized codes. The loss of the CRF-based predictor is computed with:
$$
\mathcal {L} _ {\mathrm {c r f}} = - \log p \left(\boldsymbol {z} ^ {\text {r e f}} \mid \boldsymbol {x}\right). \tag {12}
$$
To equip with the GateNet, we randomly mix the  $z^{\mathrm{ref}}$  and the predicted  $z^{\mathrm{pred}}$  as:
$$
\boldsymbol {z} _ {i} ^ {\text {m i x}} = \left\{ \begin{array}{l l} \boldsymbol {z} _ {i} ^ {\text {p r e d}} & \text {i f} p \geq \tau \\ \boldsymbol {z} _ {i} ^ {\text {r e f}} & \text {i f} p <   \tau \end{array} , \right. \tag {13}
$$
where  $p\sim \mathbb{U}[0,1]$  and  $\tau$  is the threshold we set 0.5 in our experiments. Grounding on the  $z_{\mathrm{mix}}$  the non-autoregressive translation loss is computed with:
$$
\mathcal {L} _ {\mathrm {N A T}} = - \log p (\boldsymbol {y} | \boldsymbol {z} ^ {\text {m i x}}, \boldsymbol {x}; \theta). \tag {14}
$$
With the hyper-parameter  $\alpha$ , the overall training loss is:
$$
\mathcal {L} = \mathcal {L} _ {\mathrm {N A T}} + \alpha \mathcal {L} _ {\mathrm {c r f}}. \tag {15}
$$
CNAT selects the best sequence by choosing the highest-probability latent sequence  $z$  with Viterbi decoding (Viterbi, 1967), then generate the tokens with:
$$
\boldsymbol {z} ^ {*} = \underset {\boldsymbol {z}} {\arg \max} p (\boldsymbol {z} | \boldsymbol {x}; \theta),
$$
$$
\text {a n d} \boldsymbol {y} ^ {*} = \underset {\boldsymbol {y}} {\arg \max } p (\boldsymbol {y} | \boldsymbol {z} ^ {*}, \boldsymbol {x}; \theta),
$$
where identifying  $\mathbf{y}^*$  only requires independently maximizing the local probability for each output position.
Datasets. We conduct the experiments on the most widely used machine translation benchmarks: WMT14 English-German (WMT14 EN-DE, 4.5M pairs)<sup>1</sup> and IWSLT14 German-English (IWSLT14, 160K pairs)<sup>2</sup>. The datasets are processed with the Moses script (Koehn et al., 2007), and the words are segmented into subword units using byte-pair encoding (Sennrich et al., 2016, BPE). We use the shared subword embeddings between the source language and target language for the WMT datasets and the separated subword embeddings for the IWSLT14 dataset.
Model Setting. In the case of IWSLT14 task, we use a small setting ( $d_{\mathrm{model}} = 256$ ,  $d_{\mathrm{hidden}} = 512$ ,  $p_{\mathrm{dropout}} = 0.1$ ,  $n_{\mathrm{layer}} = 5$  and  $n_{\mathrm{head}} = 4$ ) for Transformer and NAT models. For the WMT tasks, we use the Transformer-base setting ( $d_{\mathrm{model}} = 512$ ,  $d_{\mathrm{hidden}} = 512$ ,  $p_{\mathrm{dropout}} = 0.3$ ,  $n_{\mathrm{head}} = 8$  and  $n_{\mathrm{layer}} = 6$ ) of the Vaswani et al. (2017). We set the hyperparameter  $\alpha$  used in Eq. 15 and  $\lambda$  in Eq. 7-8 to 1.0 and 0.999, respectively. The categorical number  $K$  is set to 64 in our experiments. We implement our model based on the open-source framework of fairseq (Ott et al., 2019).
Optimization. We optimize the parameter with the Adam (Kingma and Ba, 2015) with  $\beta = (0.9, 0.98)$ . We use inverse square root learning rate scheduling (Vaswani et al., 2017) for the WMT tasks and linear annealing schedule (Lee et al., 2018) from  $3 \times 10^{-4}$  to  $1 \times 10^{-5}$  for the IWSLT14 task. Each mini-batch consists of 2048 tokens for IWSLT14 and 32K tokens for WMT tasks.
Distillation. Sequence-level knowledge distillation (Hinton et al., 2015) is applied to alleviate the multi-modality problem (Gu et al., 2018) while training. We follow previous studies on NAT (Gu et al., 2018; Lee et al., 2018; Wei et al., 2019) and use translations produced by a pre-trained autoregressive Transformer (Vaswani et al., 2017) as the training data.
Reranking. We also include the results that come at reranked parallel decoding (Gu et al., 2018; Guo et al., 2019; Wang et al., 2019; Wei et al., 2019), which generates several decoding candidates in parallel and selects the best via re-scoring using a
Model	WMT14	IWSLT14
EN-DE	DE-EN	DE-EN
LV-NAR	11.80	/	/
AXE CMLM	20.40	24.90	/
SynST	20.74	25.50	23.82
Flowseq	20.85	25.40	24.75
NAT (ours)	9.80	11.02	17.77
CNAT (ours)	21.30	25.73	29.81
Table 1: Results of the NAT models with argmax decoding on test set of WMT14 and IWSLT14.
pre-trained autoregressive model. Specifically, we first predict the target length  $\hat{m}$  and generate output sequence with arg max decoding for each length candidate  $m\in [\hat{m} -\Delta m,\hat{m} +\Delta m]$  ( $\Delta m = 4$  in our experiments, means there are  $N = 9$  candidates), which was called length parallel decoding (LPD). Then, we use the pre-trained teacher to rank these sequences and identify the best overall output as the final output.
Baselines. We compare the CNAT with several strong NAT baselines, including:
- The NAT builds upon latent variables: NATFT (Gu et al., 2018), LT (Kaiser et al., 2018), Syn-ST (Akoury et al., 2019), LV-NAR (Shu et al., 2019) and Flowseq (Ma et al., 2019).
- The NAT with extra autoregressive decoding or iterative refinement: NAT-DCRF (Sun et al., 2019), IR-NAT (Lee et al., 2018), and CMLM (Ghazvininejad et al., 2019).
- The NAT with auxiliary training objectives: NAT-REG (Wang et al., 2019), imitate-NAT (Wei et al., 2019).
We compare the proposed CNAT against baselines both in terms of generating quality and inference speedup. For all our tasks, we obtain the performance of baselines by either directly using the performance figures reported in the previous works if they are available or producing them by using the open-source implementation of baseline algorithms on our datasets.
Metrics. We evaluate using the tokenized and cased BLEU scores (Papineni et al., 2002). We highlight the best NAT result with bold text.
Translation Quality. First, we compare CNAT with the NAT models without using advanced techniques, such as knowledge distillation, reranking,
1https://drive.google.com/uc?export  $\equiv$  download&id=0B_bZck-ksdkpM25jRUN2X2UxMm8
Model	WMT14	IWSLT14
EN-DE	DE-EN	DE-EN
NAT-FT	17.69	21.47	/
LT	19.80	/	/
NAT-REG	20.65	24.77	23.89
imitate-NAT	22.44	25.67	/
Flowseq	23.72	28.39	27.55
NAT-DCRF	23.44	27.22	27.44
Transformer (ours)	27.33	31.69	34.29
NAT (ours)	17.69	18.93	23.78
CNAT (ours)	25.56	29.36	31.15
Table 2: Results of NAT models trained with knowledge distillation on test set of WMT14 and IWSLT14.
Model	N	WMT14
EN-DE	DE-EN
NAT-FT	10	18.66	22.42
NAT-FT	100	19.17	23.20
LT	10	21.00	/
LT	100	22.50	/
NAT-REG	9	24.61	28.90
imitate-NAT	9	24.15	27.28
Flowseq	15	24.70	29.44
Flowseq	30	25.31	30.68
NAT-DCRF	9	26.07	29.68
NAT-DCRF	19	26.80	30.04
Transformer (ours)	-	27.33	31.69
CNAT (ours)	9	26.60	30.75
or iterative refinements. The results are listed in Table 1. The CNAT achieves significant improvements (around 11.5 BLEU in EN-DE, more than 14.5 BLEU in DE-EN) over the vanilla NAT, which indicates that modeling categorical information could improve the modeling capability of the NAT model. Also, the CNAT achieves better results than Flowseq and SynST, which demonstrates the effectiveness of CNAT in modeling dependencies between the target outputs.
The performance of the NAT models with advance techniques (sequence-level knowledge distillation or reranking) is listed in Table 2 and Table 3. Coupling with the knowledge distillation techniques, all NAT models achieve remarkable improvements.