Self-Improvement of Non-autoregressive Model via Sequence-Level Distillation
Authors: Yusheng Liao
Shuyang Jiang
Yiqi Li
Yu Wang
Yanfeng Wang
Venue: EMNLP 2023 Main (2023)
Keywords: non-autoregressive transformers
self-improvement
distillation
TL;DR: A simple yet effective method to generate the distilled data which is more suitable for non-autoregressive model to learn.

Abstract: Although Non-autoregressive Transformer (NAT) models have achieved great success in terms of fast inference speed, this speedup comes with a performance drop due to the inherent \emph{multi-modality} problem of the NAT model. Previous works commonly alleviate this problem by replacing the target side of the raw data with distilled data generated by Autoregressive Transformer (AT) models. However, the multi-modality problem in the distilled data is still significant and thus limits further improvement of the NAT models. In this paper, we propose a method called Sequence-Level Self-Distillation (SLSD), which aims to generate distilled data by the NAT model itself, eliminating the need for additional teacher networks. Furthermore, SLSD can adapt to different NAT models without precise adjustments since the self-distilled data is generated from the same types of NAT models. We conduct extensive experiments on WMT14 EN$\leftrightarrow$DE and WMT16 EN$\leftrightarrow$RO and choose four classic NAT models as the backbones to validate the generality and effectiveness of SLSD. The results show that our approach can consistently improve all models on both raw data and distilled data without sacrificing the inference speed.

Seff-Improvement of Non-autoregressie Model via Sequence-Level Distillation
Yusheng Liao', Shuyang Jiang', Yiqi Li', Yanfeng Wang 1.2 Yu Wang
'Cooperative Medianet Innovation Center, Shanghai Jiao Tong University *Shanghai Artificial Intelligence Laboratory
Fuudan University
(liao20160907 77-addmant, 1 yuwangsjtuJesjtu.cdm cn shuyangjiang23ems fudan. edu. cn
Abstract
on the previous ones, NAT models lack contextual dependency and are therefore more likely to gener- ate sequences with mixed modalities, leading to a decline in performance.
Although Non-autoregressive Transformer (NAT) models have achieved great success in terms of fast inference speed, this speedup comes with a performance drop due to the inher- ent muuli--mddalin problem of the NAT model. Previous works commonly alleviate this proh- lem by replacing the target side of the data with distilled data generated by Autoregressive Transformer (AT) models. However, the multi- modality problem in the distilled data is still significant and thus limits further improvement the NAT models. In this paper, propose method called Sequence-Lev (SLSD), which aims to generate distilled data by the NAT model itself, eliminating the need for additional teacher networks. Furthermore, SLSD can adapt to different NAT models with- out precise adjustments since the self-distilled data is generated from the same types NAT models. We conduct extensive experiments on WMT14 DE and WMT16 EN RO and choose five classic NAT models as the back- bones validate the generality and effective- ness of SLSD. The results show that our ap- proach can consistently improve models on both raw data and distilled data without sacri- ficing the inference speed.
One way to improve the ability of NAT mod- els to handle complex data is by enhancing their capacity (Ghazvininejad et al., 2020; Qian et al., 2021; Du et al., 2021; Shao et al., 2022). Latent alignment models (Libovický and Helcl, 2018; Sa- haria al., 2020) relax the alignment restriction by marginalizing out all monotonic latent align- ments using the connectionist temporal classifica- tion (CTC) loss (Graves et al., 2006). Recently, Huang et al. (2022c) proposed the Directed Acyclic Transformer (DAT) that can capture multiple trans- lation modalities simultaneously by modeling sev- eral decoding paths. Despite the success of these two models, they struggle to handle non-monotoiic alignments in machine translation (Shao and Feng. 2022; Ma al., 2023), which are also common multi-modaliit problems.
Self- Distiillatln
Another common approach to alleviate the multi-modality problem is modifying the target se- quence (Huang et al., 2022b). One standard method is to simplify raw data with sequence-level knowl- edge distillation (Kim and Rush, 2016), which re- places the target side of training data with the out- put of AT models. However, AT distilled data still has two issues. First, the multi-modality issue in distilled data still remains (Zhou al., 2020). Sec- ond, Zhou et al. (2020) find that higher-quality distilled data does not necessarily improve the per- formance of NAT models generally. Weaker NAT models achieve the best performance on distilled data generated by smaller AT, while stronger NAT models prefer the data generated by stronger AT. This phenomenon suggests that, when simplifying data for training specific NAT model, high qual- ity may not be the first consideration, but rather making the data more suitable for NAT to learn Considering the aforementioned problems, some
1 Introduction
Non-autoregressive Transformer (NAT) mod- els (Gu et al., 2018) have significantly improved inference speed compared to autoregressive Trans- former (AT) models (Vaswani et al., 2017). NAT models make the conditional independence assump- tion in the output space and generate the entire se- quence in parallel. However, this speedup comes at the cost translation quality due to the multi- modality problem (Gu et al., 2018), where the data distribution typically includes numerous potential translations of the same sequence. Unlike AT mod- els, which generate subsequent tokens conditioned
"Corresponding author

---

recent works focus on modifying the distillation strategy to generale more adaptive distilled data for NAT models. For instance, Guo et al. (2021) con- struct the distilled data in five steps via re-ranking and filtering the data using an AT model, and Shao et al. (2022) generate multi-reference distilled data with multiple AT models. However, these works introduce redundant pipeline and only achieve limited performance improvements.
We propose simple yet effective method, SLSD, to generate the distilled data by NAT models itself, which can significantly alleviate the multi mulii-mmddlity problem in the data and be more adaptive for NAT models to learn.
We further explore the application of SLSD on various NAT models and find that the pro- posed framework can be directly applied to raw data without sacrificing inference speed or relying additional teacher networks.
Therefore, most existing works only consider knowledge distillation as a necessary data process- ing technique, rather than producing adaptive data for NAT models to learn better. In this paper, we propose a simple yer effective approach to generate distilled data that is more adaptive for NAT models learn, named Sequence-Leve Level Seff-Distillation (SLSD). Inspired by Shao al. (2022) and Sun and Yang (2020), which suggest that targets with a higher likelihood of the NAT models have fewer problems with multi-modality, we search for self- distilled targets in the space of the output distri- butions produced by the NAT models. We define a score function to select the self-distilled targets to ensure the quality of the self-distilled data. In the SLSD framework, the NAT model itself pro- duces the distilled data, eliminating the need for additional teacher networks. Furthermore, it can adapt to the training processes of different NAT models without requiring precise adjustments.
2 Preliminary
In this section, we first briefly describe the task formulation and then introduce three types of NAT models. The machine translation task can be for- mally defined as a sequence-to-sequee tion problem. Given the target language sequence y= 12 ..., yr} and source language sequence x=(x1, 22,, .5}. the non-autooregresive models assume conditional independence between the out- put tokens and factorize the output probabilities as Po yyl x: > p(y [z), where 0 represents the parameters of the NAT models.
genera-
2.1 Vanilla Non-autoregressive Models
Vanilla NAT models are typically trained with cross-entropy loss to maximize the likelihood of the training data:
To validate the effectiveness and generality of the proposed SLSD approach, we conduct tensive experiments on four machine translation benchmarks: WMT14 EN+DE and W EN+RO. We chose five classic NAT methods as the bascline methods, including VNAT (Gu et al., 2018). CMLM (Ghazvininejad et al., 2019), GLAT (Qian et al., 2021), CTC, and DAT. The results show that the SLSD approach can con- sistently improve all models for different trans- lation directions both raw and distilled data. Additionally, under the same training strategy, the model fine-tuned on self-distilled data can achieve better performance than the raw data and distilled data generated by AT. The exper- iments indicate that self-distileed data can miti- gate the multi-modality problem in raw and dis- tilled data, thereby improving the performance of the NAT models. Our code is released at _AT.
CCE = log pe (v(
(1)
However, the conditional independence assumption makes it difficult for vanilla NAT models to learn directly from raw data. So some works attempt to improve the modeling ability of NAT models by adding contextual information to the inputs:
T
LCE = log pe (ye
(2)
where ?(y) is a function to generate the input con- text. For example, ?(y) randomly samples words from y and masks these sampled words in the in- puts in CMLM and GLAT further adaptively con- trols the sampling number according the distance between the output and target sequences.
https://gthub.ccmm//he//eeeoss/
2.2 Connectionist Temporal Classification
The major contributions of our paper are sum-
Instead of the strict position-tor to-posttinn calculation in the cross-entropy loss, CTC models a flexible
marized as follows:

---

monotonic alignment between the output sequence and the target sequence. There are two differences between CTC models and NAT models: 1) The input length of the CTC model is typically times the length of the source sentence. 2) CTC mod- els are allowed to output a "blank" token. With these unique features, the output of the CTC mod- els can align with the target sequence by removing all consecutive repeated tokens and the "blank" to- kens. Assume that (y) is the set of all possible alignments between the output sequence and target sequence, the training object of the CTC models is calculated by marginalizing the likelihoods of all possible alignments:
steps: the sampling of the self-distilled targets and the selection of the self-distilled targets. Finally, we will discuss the training details of the SLSD framework.
3.1 Motivation
Previous studies have shown that the data distilled by AT models may not be simple enough for NAT models to learn from, and that high-quality dis- tilled data does not necessarily lead to improved performance of the NAT models in general. This can be attributed to two main reasons: 1) There is mismatch between the modeling types of NAT models and AT models. The distilled data gener- ated by AT models in autoregressive manner may be the most suitable for the learning pro- cess of NAT models (Guo et al., 2021). 2) is challenging to balance the quality and suitability of the distilled data. While higher data quality can prevent information loss, it also indicates that the distilled data is more complex and has more seri- ous multi-modality problem (Zhou et al., 2020). To address these issues, propose using NAT models as distillation teachers instead of AT models, which can alleviate the mismatch problem. Additionally, our proposed method allows for the control of the quality and suitability of the self-distilled data by adjusting the size of the candidate set, as discussed later in Section 5.1. To prevent confusion, in this paper, we refer to the data generated by the AT models distilled data and the data generated by our method as self-distilled data.
LOTc - -log
Σ (b(()))
(3)
bC3(2)
2.3 Directed Acyclic Graph
Previous NAT models hardly handle the multi- modality problem in the raw data. Directed Acyclic Transformers (DAT) attempt to address this issue by stacking a directed acyclic graph (DAG) on the top of the NAT decoder, where the vertices and edges in DAG correspond to hidden states of the de- coder and the transitions between the hidden states respectively. The transitions between the connect- ing vertices constitute multiple possible decoding paths, allowing DAT to capture multiple translation modalities simultaneously. The path probability pe (e(a) is factorized based on the Markov hypoth- esis:
aa
(a(a))) = Po (a: (l:; x))
I
(4)
3.2 Sampling process of the SLSD
Vanilla NAT Sampling all possible combinations of the whole vocabulary is computationally forbid- den there are total of | samples, where | is the vocabulary size. Instead, we sample N candi- dates from the output distributions of NAT models to form the candidate set for the selection of the self-distilled targets. A candidate h the candi- dates set H(x) is sampled from the distribution as below:
where |a] is the DAT output length and typically À times the length of the source sequence. Once path a is determined, token Yi can be generated conditioned on the decoder hidden state with index o;, And the DAT can be trained by minimizing the negative log-likelihood loss as below:
CDAT = 1 loo Σ pe (aylaa x))o(aa)))
(5)
aeT
where I' is all possible output paths with the same length of target sequence y.
h nat
II
pe (hpat m))
(6)
Unlike vanilla NAT models. the tokens in the outputs of CTC models are not totally condi- tionally independent, which makes it difficult calculate the probability of the output sequences. Specifically, the probability of the candidate is the
CTC
3 Method
In this section, we first explain the motivation of the proposed SLSD framework. Then we describe the process of generating self-distilled targets in two

---

marginalization of all possible corresponding align- ment sequences:
target sequence as (3)). and the number of times each n-gram g (z) appears y as (y))). Therefore, the number of n-gram matches between the candidate and reference can be defined as:
h cl
(b(() (C
(7)
Mn(y,h) h)=
Σ gEG.(v)
min (Cg( Cy(h) (11)
However, the corresponding alignment set 3(h") is exponentially large, making Equation in- tractable. Alternatively, we sample the alignment sequence from the output distribution of CTC models to approximate the candidates sampling process:
and the total number of n-gram in the reference and the self-distilled data can be defined as:
Mn(y)
Σ
Co(y)
(12)
b ~ (blac
(8)
gEGm(8)
Then we can get the candidates with the collaps- ing function h = 3 ((b) (Saharia et al., 2020). Note that multiple different alignments may corre- spond to the same candidate during sampling, so the candidate may contain duplicate samples.
Mn(h) = Σ Cg(h)
(13)
gEG,(h)
Based on the denotation above, the similarity function is defined as the minimum value of n- gram precision and recall of the candidate against the reference
DAT Similar to CTC models, it is intractable to marginalize all possible output paths for the can- didates. So we factorize the probabilities into the production of the output paths and the output to- kens and thus sample the output sequence from the DAT models following two-step sampling pro- cess. The decoding paths are sampled from the transition distributions:
E Mn.y.. h)
sim h)
(14)
min (M) M.(h))
where is the maximum size of n-gram. Consider- ing candidates sampled from the output distribution of CTC and DAT models have different lengths, we further add a length penalty to constrain the lengths of the candidates:
a
Po (0/ 1a;, a)
BP h) -1-1hh//lbbl
(15)
Then the output tokens are sampled based on the decoding path to get the output sequence f:
where |h| and |y| are the length of the candi- date and reference, respectively. Note that when adopted on the vanilla NAT models, the length penalty is equal to 1. Finally, the score function can be formulated as below:
f (pe(fll,
(10)
Finally, can get the candidates with the collaps-
ing function h hddg = 3 (f)).
score(y. h) BP h) sim h))
(16)
3.3 Selection Process of the SLSD
With the score function to measure the quality of the candidates, we choose the sample with the high- est score as the self-distille targets r:
It is intuitive that samples in the candidate set sam- pled from the output distribution of the NAT mod- els are the ones that NAT models prefer and are easy to learn. this section, we mainly focus on selecting the high-quality sample in the candidate set.
arg max score h) hcH(#)
(17)
We use score function score(y. (yo h) to measure how similar the candidate h is to the reference y. If sample in the candidate set is close to the reference, we can assume is high-quality. Our im- plementation uses n-gram overlap as the function to measure the distance between two sequences. We define the set of non-repeating n-grams in the
3,4 Training Schedule
To make sure the candidates sampled from the out- put distribution of NAT models are meaningful and close enough to the reference, NAT models that have been pretrained on the data are chosen as the initialization of the model in SLSD framework. Then we adopt the self- distilldd targets sampling

---

WMT14
WMT16
Categories
Models
EN-DE
DE-EN
EN-RO
RO-EN
Speedup
AT
Transformer (Vaswani et al., 2017)
27.74
31.09
34.16
34.46
x1.0
VNAT (Gu and Kong, 2021)
11.40
16.47
24.52
24.79
x15.3
VNAT
VNAT (Ours)
11.42
15.85
24.79
24.44
x15.3
SNSD
23.69
28.20
31.30
31.76
x15.3
CMLM, (Ghazvininejad et al., 2019)
10.64
21.22
x15.3
AXE (Ghazvininejad et al., 2020)
20.40
24.90
30.47
31.42
x
OaXE (Du al., 2021)
22.40
26.80
xl4.2
CMLM
ngram-OaXE (Du al., 2022)
23.60
27.90
x15.3
CMLM, (Ours)
11.67
15.32
21.17
22.28
x15.3
SLSDCMLM
24.81
28.35
31.30
32.31
x15.3
GLAT (Qian et al., 2021)
19.42
26.51
x15.3
LatemtGLAT (Bao al., 2022)
24.71
29.16
x11.3
GLAT
GLAT (Ours)
19.56
26.03
30.58
31.76
x15.3
SLSDGLAT
24.66
28.77
31.21
32.27
x15.3
CTC (Libovický and Helel, 2018)
18.42
23.65
x14.6
CTC+GLAT (Qian et al., 2021)
25,02
29.14
x14.2
CTC
CTC+DSLP (Huang et al., 2022a)
24.81
28.33
x14.0
CTC+GLAT (Ours)
24.73
28.79
31.03
32.06
x14.2
SLSDerc
26.17
29.77
32.27
32.96
x14.2
DAT () (Huang al., 2022c)
26.16
x14.2
DAT (X 8) (Huang et al., 2022c)
26.57
30.68
x13.9
DAT
DAT (A 4) (Ours)
26.06
30.38
32.67
33.02
x14.2
SLSDDAT
26.76
31.41
33.04
33.42
x14.2
Table 1: The performance of the baseline models and SLSD models on WMT14 EN+DE and WMT16 EN+RO raw data. The baseline model with (Ours) indicates our re- re-implementation. CMLM, refers to the CMLM model that inferences with iterations. The best performance of each type of NAT model is bold. SLSD represents our method with different backbone.
and selection pipeline described above to generate the self-distilled data.
tion benchmarks: WMT14 English++German (EN+DE, 4.5M sentence pairs) and WMT16 English<>Romanian Romanian (EN+>RO, 0.6M sentence pairs). For WMT14 EN+DE, we preprocessed the datasets with a joint BPE with 40K merge opera- tions following the pipelines provided in the fairseq toolkit (Ott et al., 2019). For WMT16 EN++RO use the pre-processed data provided by Lee al. (2018). 8). Besides, choose two additional datasets, Quora and ROCStory (Mostafazadeh al., 2016), to validate the generalization and effectiveness of the SLSD method. this paper, we evaluate the performance of models using the BLEU metric (Pa- pineni et al., 2002) for all datasets.
For each step in the sclf-distillation process, we sample from the current output distribution of the model to generate self-distileed targets, which can ensure that the quality of the targets and the per- formance of the model are synchronously updated, For the decoder input of the NAT models, previ- ous works found that giving some context in the input helps the learning process of NAT models. contrast, we adopt full masked sequences as input of the NAT decoders for the targets in the self-distillation process is easy to learn and can reduce the context mismatch between the training and inference process.
Hy perparaaames of the initialization Follow- ing previous works, we adopt the basic implemen- tation of Transformer-buse for machine translation tasks. Each model consists of a 6-layer encoder and
4 Experiments
4.1 Settings
Datasets We conduct experiments on both directions two standard machine transla-
htwp:/o/wdddatttu.coooooooo/ooee Quiersst-Quoo-r-eeee Dataset- Paluuse-Quesstion-Pairrs

---

WMT14
Quora BLEU
ROCStory
Models
EN-DE
DE-EN
Model
BLEU-1
BLEU-2
Transformer (Vaswani ct al., 2017)
27.61
31,48
VNAT
21.37
29.40
4.70
Vanilla NAT (Gu et al., 2018)
17.69
21.47
SLSD
24.69
35.70
8.30
CTC (Libovický and Helel, 2018)
25.52
28.73
AXE (Ghazvininejad al., 2020)
23.53
27.90
CTC
25.72
38.40
7.60
OaXB (Du et al., 2021)
26.10
30.20
SLSDCTC 27.19
41.60
10.30
Seq-NAT (Shao al.. 2021)
25.52
29.91
CTC+GLAT (Qian et al., 2021)
26.39
29.54
Table 3:
The performance of the baseline models and SLSD on other generation tasks. Note that for the CTC methods we adopt À 2 on Quora and = 8 on ROC- Story due to the length ratio the source sequences and the target sequences. seq
n-gram-OaXF (Du al., 2022)
26.50
30.50
CTC+DSLP (Huang ct al., 2022d)
27.02
31.61
DAG (A=8) (Huang et al., 2022c)
27.49
31.37
CMLM; (Ours)
21.06
25.36
SLSDCMLM
26.50
30.36
GLAT (Ours)
26.11
30.51
SLSDGLAT
26.36
30.74
Inference For CMLM and GLAT, we use Length Parallel Decoding (LPD) (Wei et al., 2019) with
STC+GLAT((Oun
26.51
30.08
SLSDeTe
27.15
31.35
DAT (A=4)
26.78
31.42
== 5 length candidates during inference. For CTC models, we directly decode the output sequence. For decoding steps in DAT, coding algorithm. We select the 5 best checkpoints on the validation sets and average them as the eval- uation checkpoint.
SLSDDAT
27.22
31.75
lookahead de-
Table 2: The performance of the baseline models and
the SLSD models on WMT14 EN+DE distilled data.
4.2 Main Results
a 6-layer decoder with 8 attention heads. The hid- den dimension and feed-forward layer dimension are 512 and 2048, respectively. For optimization, we use the Adam optimizer (Kingma and Ba, 2015) with ,- (0.9, 0.98). The weight decay is set to 0.01 and the label smoothing (Szegedy al., 2016) is set to 0.1. The learning rate warms up for 10k steps to 5e - 4 and decays with an inverse square schedule. For WMT16 Ent+ Ro, we use dropout rate of 0.3 and a batch size of 32K tokens, while for WMT14 En switch to 0.1 and 64K accord- ingly. All models are trained with 300k steps both datasets, For Quora and ROCStory datesets, we train Transformer-smon with dropout rate of 0.3 for 100k steps. Code implementation based on fairseq.
Table shows the BLEU scores for each baseline and our methods on the WMT14 EN+>DE and WMT16 EN raw datasets. We validated the efficacy of our method on five types of models: VNAT, CMLM, GLAT, CTC, and DAG. The results indicate that our method considerably improves all models on all datasets. Self-distillation boost CMLM models exceed previous CMLM- based methods and even achieve higher perfor- mance compared to CTC-based models. Specif- ically, our framework can improve CTC+GLAT models by an average of more than BLEU score, which is considerable improvement for strong NAT baseline raw data. Moreover, the CTC+GLAT model can achieve a BLEU score of 26.17 on WMT14 EN+DD raw data, which even surpasses DAT (A=4) with only x=2. Since train- ing DAT (Q=8) requires significant computing re- sources and time, we only implement our method on DAT (A=4), which still outperforms DAT (A=8) by up to 1.03 BLEU score. The effectiveness of our method verifies that the self-training method can significantly mitigate the multi multi-modality problem in the raw data.
Finetuning Hyperparameters for SLSD We se- lect the last checkpoint of pretrained stage as the initialization of the self-training stage. We finetune models on machine translation dataset and other generation task dataset for 100k steps and 30k steps, respectively. The learning rate is set to 2e-6 6 for all models. The batch size of tokens and the dropout rate are following the corresponding pretrained stage configuration. The size of the candidate set 100 for DAT models and 40 for the other models. The n- n-gram size of the score function is set to 4.
Besides, the improvement of our methods can be seen not only in the raw data but also in the distilled data, which is already simple for NAT models to learn. The performance of our methods and base-

---

WMT14
learning.
Models (Steps)
EN-DE
DE-EN
5 Analysis
GLAT (300k)
19.56
26.03
w/ raw data (100k)
19.45
25.62
distilled data (100k)
23.92
28.81
H(())
score(y,
NCMx(R)
BLEU
w/ self-distilled data (100k)
24.66
28.77
0
18.20
CTC+GLAT (300k)
24.73
28.79
raw data (100k)
23.67
27.43
2
32.2
0.79
22.81
w/ distilled data (100k)
25.69
29.60
5
33.8
0.83
23.02
w/ self-distilled data (100k)
26.17
29.84
10
35.2
0.93
23.25
40
37.4
1.09
23.32
Table 4: The performance of continually training with
different data on WMT14 EN++DE raw data.
100
38.7
1.18
23.25
Table 5: Ablation study the size of the candidates set during self-distillation process. The experiments are conducted the WMT14 EN-DE validation set.
line models WMT14 EN++DE distilled data shown in Table 2. All baseline models can bene- fit from SLSD methods with an average of more than BLEU score gain. The improvements reflect that self-training data generated by the same NAT models is more adaptive for the learning process of various NAT models. Furthermore, we also con- duct the experiments on other generation tasks. As the results shown in Table 3, the SLSD also improve VNAT and CTC methods Quora and ROCStory dataset, which again validate the gener- alization and effectiveness of the SLSD method.
30
GLAT
25
15
30
ETCHGLAT
CTCHGLATYST
20
4.3 Effectiveness of Self-Distilled Targets
10
101
140
In order to eliminate the gain effect brought by more training steps and fairly demonstrate the ef- fectiveness of the proposed SLSD method, continually train each baseline for another 100k steps. Since the models trained on raw data and distilled data without the help of the GLAT train- ing strategy will result in performance degrada- tion (Qian et al., 2021), we adopt the GLAT ra- tio fratio as the minimum one in the pretrained stage for the self-distillation process. Specifically, adopt 00.3 for GLAT and =02 for CTC+GLAT. The results are shown in Table 4. can be seen that continually training pretrained models on raw data worsen the models' perfor- mance, which shows that simply increasing the number of training steps cannot improve the per- formance of the model. In contrast, distilled data and self-distilled data can simplify the learning process of the NAT models and thus help NAT models achieve higher scores. Specifically, the per- formance of models finetuned self-trained data exceeds the models finetuned on distilled data in most language pairs, which again shows that the self-distileed data is more suitable for NAT model
Length Target
Figure I: The performance of the generated translations with respect to the lengths of the target sequences. Re- sults are reported on the WMT14 EN-DE
5.1 Size of Candidates Set
We conduct the ablation study about the size of the candidates set and show its impact on the self- distillation process. The self-distilled targets set and the self-distileed corpus are denoted as R and (X' R), respectively. We measure the complexity of the self-distilled targets set by the Normalized Corpus-level Multi-modality (NCM) for NAT mod- els (Sun and Yang. 2020), which can be calculated as:
E (xxr)) log (r(æ))
NCMx
E E
(18) For data with more complex and serious multi- modal problems, NAT models tend to capture more possible translations simultaneously and conse- quently give rise to reduce probabilities in the mod- els and an increase NCM.

---

GLAT CTC+GLAT
GLAT
CTCYGLAT
Rolative Score
Abso
20
20
1
50
100
500
1000
5000
1
100
500
1000
5000
Training Steps
Training Steps
(b)
Figure 2: The distribution of (a) the Absolute Score and (b) the Relative Score of candidates on two baselines, GLAT and CTC+GLAT, the first 5k steps of the self-distilled process. The three dashed lines in each violin diagram from top to bottom represent the upper, middle, and lower quartiles, respectively.
GLAT
1.0
1.0
GLAT WN SLSD
Max Token Probabilss
0.8
Max Token Probsbiltty
0.8
0.6
0.6
0.4
0.4
CTC GLAT SLSD
0.0
0.2
0.4
0.6
0.8
1.0
0.4 0.s o'a Passing Probablitty
1.0
0.2
0.4 o's o'a 1.0
Max Taken Probability
Passing Probability
Figure 3: Output distribution of three GLAT-based NAT models. (a) shows the effect of the SLSD on GLAT and CTC+GLAT models. The difference between the joint distribution of passing probability and max token probabilities can be seen in the output distribution of (b) DAT and SLSDDAT. Darker areas have higher probability values.
The results are shown Table 5. As the size of the candidate set increases, the score of the self- distilled target and the NCM of the self-distilled targets set are both increased. This shows that larger candidate set can generate self-distilled targets with higher quality but more complexity. We achieve balance between the quality and complexity of the self-distileed data by controlling the size of the candidate set, thereby adapting to the training of the NAR model while ensuring data quality. After tuning on the development set, we can select the best number of candidates set as 40,
based on the reference sentence lengths in a range of 10. Figure illustrates the results of adopting SLSD on GLAT and CTC+GLAT backbones in the WMT14 EN-DE test set. It is found that GLAT can only handle short sequences, while SLSD bridges the performance gaps between different lengths and achieve consistent performances across all target lengths. For CTC+GLAT, SLSD can improve its performance in all cases. These results show that SLSD can adapt well to the training of different types of NAT models in different target sentence lengths.
5.2 Impact on Sequence Length
5.3 Candidates Score Distribution
We also investigate the effectiveness brought by delf-distillation on distinct target sequence lengths. To this end, we split the test set into six buckets
We calculate the score distribution of the candi- dates set the first 5k steps to better understand the dett-distillation process. We use two metrics,

---

i.e., Absolute Score and Relative Score, to observe the distribution of scores of the candidates set from two orthogonal aspects. The Absolute Score is the maximum score of the candidate sets and re- flects the distribution of the whole self-distilled data. The Relative Score is calculated as the ratio of each score to the maximum score in the can- didate set, which can measure the distribution of the scores in each candidate set. As shown in Fig- ure 2, both the absolute score and the relative score rapidly increase and stabilize at the beginning of the 500 steps. After the stabilization of the distri- bution, the absolute score of the GLAT is lower than the CTC+GLAT on average, showing that the self-distilled targets of the CTC+GLAT are better. In contrast, the relative score of the GLAT is higher than CTC+GLAT, which shows that the GLAT is more prone to converge on self-distilled data and CTC has more diverse candidates.
NAT models, well-initialized model is necessary. Moreover, selecting self-distilled targets from the candidate set involves computing the score of each candidate, which requires additional training time and computational cost.
A Acknowledgements
This work is supported by the National Key R&D Program of China (No. 20222D0162101). Shanghai Science and Technology Committee (No. 21511 I 101 100), STCSM (22D72229005), 111 plan (No. BP0719010), We would like to thank all of the anonymous reviewers for their helpful comments.
References
Yu Bao, Hao Zhou, Shujian Huang. Donggi Wang, Li- hua Qian, Xinyu Dai, Jiajun Chen, and Lei Li. 2022. latent-gla glat: Glancing at latent variables for parallel text generation.
Proceedings of the 60th Aunual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Popers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8398 8409. Associ- ation for Computational Linguistics.
5.4 Output Distributions of NAT models
To verify that the SLSD suits the NAT models, we calculate the output distributions of three GLAT- based NAT models and present the result in Fig- ure 3. After training on the self-distileed data, the max token probabilities of GLAT and CTC+GLAT are improved. Besides, for DAT, the passing proba- bilities are close to 0 or 1, and token probabilities are close to 1. This indicated that SLSD can reduce the multi- modtitymddlity problem in the data, and thus improve the probabilities of the output path in the DAT.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order- agnostic cross entropy for non -on-auutoreressvvee ma- chine translation. In Proceedings of the 38th Inter- national Conference on Machine Learning, ICMI, 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedingy of Machine Learning Research, pages 2859. PMLR.
Cunxiao Du Zhaopeng Tu, Longyue Wang, and Jing Jiang. 2022. agram-oaxe: basede order- agnostic cross entropy for non-autoregreesive ma- chine translation. In Proceedings of the 29th Inter- national Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, Oc- tober 12-17, 2022, pages 5035-5045. International Committee on Computational Linguistics.
6 Conclusion
In this paper, we introduced simple yet effec- tive method, SLSD, generate distilled data us- ing NAT models themselves. This upproach can greatly reduce the multi-modality problem in the data, and consistently improves the performance of four types of NAT models across all datasets. Ablation experiments verify that self-distilled data is better suited for NAT models to learn from, com- pared to the distilled data generated by AT models.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for attreeeoreeresivvee machine translation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3515-3523 PMLR
Limitations
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models, In Proceedings of the 2019 Conference Empiri- cal Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 6111-6120. Association for Computational Linguistics.
The proposed SLSD method produce self- distilled data that is better suited for learning by NAT models than the distilled data generated by AT models, which does not need additional teacher models. However, to obtain relevant and high- quality candidates from the output distribution of

---

Alex Graves, Santiago Fernández, Faustino J. Gomez, and Jürgen Schmidhuber. 2006. Connectionist tem- poral classification: labelling unsegmented sequence data with recurrent neural networks. In Machine Learning, Proceedings of the Twenty-Third Interna- tional Conference (ICMI. 2006), Pittshurgh, Pennsyl- vania, USA, June 25-29, 2006, volume 148 of ACM International Conference Proceeding Series, pages 1699376..0 ACM
2016 Conference on Empirical Methods Natural Language Processing, EMNLP 2016, pages 1317- 1327. The Association for Computational I Linguisties.
Diederik P. Kingma and Jimmy Ba. 2015, Adam: A method for stochastic optimization. In 3rd Inter- national Conference on Learning Representations, ICLR 2015.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural se- quence modeling by iterative refinement. In Proceed- ings of the 2018 Conference on Empirieal Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pages 1173-1182. Association for Computational Linguisties.
Jiatao Gu, James Bradbury, Caiming Xiong, Victor O. K. Li, and Richard Socher. 2018. von-auvoreeressivve neural machine translation. In 6th International Con- ference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net.
Jiatao Gu and Xiang Kong. 2021. Fully non- autoregressive neural machine translation: Tricks of the trade. In Findings of the Association for Com- putational Linguisties; ACL-IJCNLP 2021, pages 123-133, Online. Association for Computational Lin- guistics.
Jindrich Libovický and Jindrich Helcl. 2018. End-to- end non- autoregressive neural machine translation with connectionist temporal classification. Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, Brussels, Bel- gium, October 31 -ovember 4, 2018, pages 3016- 3021. Association for Computational Linguisties.
Jiaxin Guo. Minghan Wang, Daimeng Wei, Hengchao Shang. Yuxia Wang. Zongyao Li, Zhengzhe Yu, Zhanglin Wu, Yimeng Chen, Chang Su, Min Zhang, Lizhi Lei, Shimin Tao, and Hao Yang. 2021. Self- distillation mixup training for non- autoreresseeevee neural machine translation. CoRR, abs/21 1640.
Zhengrui Ma. Chenze Shao. Shangtong Gui. Min Zhang. and Yang Feng. 2023. Fuzzy alignments in directed acyelic graph for non-autoregressive machine trans- lation. CoRR, abs/2303. 06662..
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James F, Allen. 2016. A cor- pus and cloze evaluation for deeper understanding of commonsense stories. In NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, San Diego California, USA, June 12-17, 2016, pages 839-849. The Associ- ation for Computational Linguistics.
Chenyang Huang, Hao Zhou, Osmar R. Zaïane, Lilii Mou, and Lei Li. 2022a. Non-autoregresssive trans- lation with layer-wise prediction and deep supervi- sion. In Thirty-Sixth AAAI Conference on Artificial Intelligence, AAAI 2022, Thirty-Fourty Conference on Innovative Applications of Artificial Intelligence. IAAI 2022, The Twelveth Symposium on Educational Advances in Artificial Intelligence, EAAI 2022 Virtual Event, February 22- March 1, 2022, pages 10776- 10784. AAAI Press.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier. and Michael Auli. 2019. fairseq: A Fast, Extensible Toolkit for Sequence modeling. In Proceedings of NAACL- HLT 2019 Demonstrations.
Fei Huang, Tianhua Tao, Hao Zhou, Lei Li, and Minlie Huang. 2022b. On the learning of non-autoregressive transformers. In International Conference on Ma- chine Learning, ICML 2022, 17-23 July 2022, Balti- more, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 9356-9376. PMLR.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia, PA. USA, pages 311-318. ACL.
Fei Huang, Hao Zhou, Yang Liu, Hang Li. and Minlie Huang. 2022c. Directed acyclic transformer for non- autoregressive machine translation. In International Conference on Machine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA. volume 162 of Proceedings of Machine Learning Research, pages 9410-9428. PMLR.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for non-autoregressive neural machine translation. In Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Longuage Processing, ACI/I/CNLP 2021, (Volume I: Long Papers), Virtual Event, Au- gust 1-6, 2021, pages 1993-2003. Association for Computational Linguisties.
Xiao Shi Huang, Felipe Perez, and Maksims Volkovs. 2022d. Improving non-autoregressive translation models without distillation. In International Con- ference on Learning Representations, ICLR 2022.
Yoon Kim and Alexander M. Rush. 2016. Sequence- level knowledge distillation. In Proceedings of the

---

Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive ma- chine translation with latent alignments. In Proceed- ings of the 2020 Conference Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 1098-1 108. Associa- tion for Computational Linguisties.
Chenze Shao and Yang Feng. 2022. Non-monotonic latent alignments for etc-based non-autoregressive machine translation. NeurIPS.
Chenze Shao, Yang Feng, Jinchao Zhang, Fandong Meng, and Jie Zhou. 2021. Sequence-level training for non-autoregressive neural machine translation. Comput. Linguistics, -794)9991
Chenze Shao, Xuanfu Wu, and Yang Feng. 2022. One reference is not enough: Diverse distillation with ref- erence selection for non-autoregressive translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Compula- tional Linguistics: Human Language Technologies, NAACL 2022, Seattle, WA. United States, July 10-15, 2022, pages 3779 7779137991..... Association for Computa- tional Linguisties.
Zhiqing Sun and Yiming Yang. 2020. An EM approach non-autoreeresssvve conditional sequence genera- tion. Proceedings of the 37th International Con- ference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 9249-9258. PMLR.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe. Jonathon Shlens, and Zhigniew Wojna. 2016. Re- thinking the inception architecture for computer vi- sion. 2016 IEEE Conference on Computer Vision and Puttern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 2818 2826. IEEE Computer Society.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention all you need. In Advances in Neural Information Pro- cessing Systems 30: Anmual Conference Neural Information Processing Systems 2017, pages 5998- 6008.
Bingzhen Wei, Mingxuan Wang, Hao Zhou, Junyang Lin, and Xu Sun. 2019, Imitation learning for non- autoregressive neural machine translation. In Pro- ceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Pa- pers, pages 1304-1312, Association for Computa- tional Linguistics.
Chunting Zhou, Jiatao Gu, and Graham Neubig. 2020. Understanding knowledge distillation in non- autoregressive machine translation. In 8th Inter- national Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRReiew


Reviews:
[{"review_id": "C8JloUo0G1", "reviewer": "Reviewer_nVwZ", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1：The improvements are relatively small for sophisticated NAT methods (GLAT, CTC, DAT). This means that the method in this study is not so effective for the state-of-the-art NAT methods.**\n\nA1: Thank you for your comments. Firstly, We must declare that the proposed method can improve GLAT with 4.9 BLEU scores and CTC with 1.44 BLEU scores on WMT EN-DE data, which is already a significant improvement in machine translation. Besides, considering the proposed method mainly improves the performance of NAT models by alleviating multimodal problems, it is reasonable for some state-of-the-art models that can already handle multimodal problems to have relatively small improvements (but still can improve DAT models with up to 1.03 BLEU score). Our method also enables the model to perform better with fewer memory requirements. For example, CTC (λ=2) with our approach can achieve better performance than DAT (λ=4) and DAT (λ=4) with our method can achieve better performance than DAT (λ=8).\n\n**Q2：The idea is similar to Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translation in that both studies use distilled data from NAT models and use reranking and filtering. But the method in this study is simpler and more effective.** \n\nA2：Thank you for your comments. This paper has been cited in our paper. There are two differences in total between our method and SDMRT (Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translation):\n* SDMRT still uses the distilled data generated from AT models to improve the NAT models. In our method, the distilled data is entirely generated by the NAT model itself, which indicates that NAR models can achieve good performance without relying on the AR model and thus become more efficient and convenient.\n* SDMRT needs to process the training data with a sophisticated framework before the training process, and the data remains unchanged during the training process. In contrast, our method resamples at every step of training to generate new data, which ensures synchronous growth of data quality and model performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "It is a clever idea to select good samples from self-distilled data using the reference target, because it is more suitable for fitting NAT models and it is better samples to follow than the self-distilled data itself. This simple method shows good results in several NAT methods.", "reasons_to_reject": "* The improvements are relatively small for sophisticated NAT methods (GLAT, CTC, DAT). This means that the method in this study is not so effective for the state-of-the-art NAT methods.\n* The idea is similar to \"Self-Distillation Mixup Training for Non-autoregressive Neural Machine Translation\" (https://arxiv.org/abs/2112.11640) in that the both studies use distilled data from NAT models and use reranking and filtering. (But the method in this study is simpler and more effective).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This study solves the problem that high quality distilled data from Autoregressive Transformer Models does not necessarily lead to improved performance of the Non-Autoregressive Transformer Models. The high quality data from AT models is complex and has a multi-modal problem so that it is not suitable for NAT models to fit it easily. \n\nThe main contributions of this study are :\n1. Use of sequence-level self-distilled data from the NAT models so that it is more suitable for NAT models to fit it. \n2. Selection of the higher quality samples in the candidate set from the self-distilled data via a similarity measure with the reference target sample. ", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Gk1IidEzl8", "reviewer": "Reviewer_orC1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1：It is not clear how the proposed method solves the multi-modality issue 2**\n\nA1: Thank you for your comments.  As mentioned in Section 3.1 of the paper, the multimodal problem in the NAT model is caused by the NAT model's inability to model the distribution of complex text data. The proposed method mainly solves multi-modality in two aspects:\n* The first is generating the distilled data with NAT models. It is easier for the models to learn because it is generated by the same type of models.\n* The second is is easy to balance data quality and complexity by adjusting the size of candidate sets.\nThese two points make our method alleviate the multi-modality problem and can easily adapt to different NAT models.\n\nThe experiment results shown in Sections 4.3 and 5.1 indicate that the proposed method mainly improves the NAT models with these two aspects.\n\n**Q2: The methods need to be evaluated in various tasks**\n\nA2：Thank you for your comments. We supplemented the results of the proposed method on two additional tasks, including Quora (Paraphrase Generation) and ROCStory (Story Generation) [1]. Considering the size of the dataset, we adopt transformer-small as the backbone. All the models are training on the date with 100k steps and our method adopts an extra 30k steps self-training process. The batch size is equal to 8k, and the dropout rate is 0.3. The remaining hyperparameters are the same as in other situations.  Specifically, due to the length ratio of the source sequences and target sequences, we adopt λ=2 for Quara and λ=8 for ROCStory, where λ is the upsampling ratio of the CTC. The results are shown below:\n| **Model** | **Quora** | **ROCStory (BLEU1/BLEU2)** |\n|:---------:|:---------:|:--------------------------:|\n| CTC       | 25.72     | 38.40/7.60                 |\n| SLSD_CTC  | **27.19**     | **41.60/10.30**                |\n| VNAT      | 21.37     | 29.40/4.70                 |\n| SLSD_VNAT | **24.69**     | **35.70/8.30**                 |\n\n[1] Huang F, Ke P, Huang M. Directed Acyclic Transformer Pre-training for High-quality Non-autoregressive Text Generation[J]. arXiv preprint arXiv:2304.11791, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1, The method is simple but useful to improve the NAT models, without using external models\n", "reasons_to_reject": "1, It is not clear how the proposed method solve the multi-modality issue\n2, The methods need to be evaluated in various tasks", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method to tackle the issue multi-modality of non-autoregressive transformer models (NAT). From methods of replacing the target side of the raw data with distilled data generated by Autoregressive Transformer (AT) models, this paper proposed generating distilled data from NAT model itself, eliminating external models to generate the data. Results on several translation tasks show the improvement.\n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "unl6ky8eEt", "reviewer": "Reviewer_Gn82", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1：The proposed method requires a well-initialized NAT model to generate good candidates. For example, it might not work for Vanilla NAT, which is not evaluated in the paper. But this is already acknowledged in the Limitations section.**\n\nA1：Thank you for your comments. We have conducted the experiments of Vanilla NAT on WMT14 EN-DE raw data, which has similar performance compared with CMLM.\n| **Model** | **WMT14 EN-DE** |\n|:---------:|:---------------:|\n| VNAT      | 10.34           |\n| SLSD_VNAT | **23.69**           |\n\nBesides, our method does not have strict requirements for model initialization. For example, CMLM models only achieve 10.64 BLEU Score on WMT14 ENDE data, which is similar to the performance of Vanilla NAT. However, our method can still improve CMLM to 24.81 BLEU score that even surpasses GLAT. The shortcoming of our model is that it must undergo two-stage training, which causes inconvenience. We will avoid such confusion in the revised paper. Sorry for the confusion and we will modify this statement. \n\n**Q2：The design of the selection function does not seem very principled (n-gram-based). It is well-known that the n-gram-based metric (e.g., BLEU and ROUGE) has a high variance for sequence-level evaluation.**\n\nA2：Thank you for your great comments. We have carefully considered this issue during the developments. We attempted various methods to select the target for the self-distillation, like using AR models or language models. However, due to the fact that our selection function needs to be called at each training step, the selection function based on neural networks can significantly increase the training time. In addition, since the candidate set is generated by the NAR model, the better selection function for self-distillation targets is still limited by the performance of the model itself, which means that a better selection function will not have too much gain on the final performance. Using n-gram based metrics can also be consistent with the testing metrics, reducing the gap between training and testing. Therefore, we adopted the simplest method to select the target of self distillation to strike a balance between performance and efficiency.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes a simple and effective method that works for multiple NAT models without much tuning.\n2. The proposed method helps mitigate the multi-modality problem better than distillation with autoregressive models.\n3. The paper provided detailed analysis and ablation studies to understand the effect of self-distillation.", "reasons_to_reject": "1. The proposed method requires a well-initialized NAT model to generate good candidates. For example, it might not work for Vanilla NAT, which is not evaluated in the paper. But this is already acknowledged in the Limitations section.\n2. The design of the selection function does not seem very principled (n-gram-based). It is well-known that the n-gram-based metric (e.g., BLEU and ROUGE) has a high variance for sequence-level evaluation.\n\nBut overall, this is a nice extension over prior work that shows how self-distillation can further simplify the training data and improve NAT models. The consistent gains across models and datasets are promising.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method called Sequence-Level Self-Distillation (SLSD) to improve non-autoregressive translation models. The key idea is to generate distilled training data using the NAT model itself and then select good ones with an n-gram-based score function. This helps create training data better suited to the NAT model and alleviates the multi-modality problem. The method is evaluated on 4 popular NAT models - CMLM, GLAT, CTC, and DAT on WMT14 En-De and WMT16 En-Ro, and shows consistent and significant improvements over all models on both raw and distilled data.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a self-distillation method for training non-autoregressive models. The idea is simple, effective and well-evaluated with extensive experiments.
Decision: Accept-Main