Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation
Authors: Chenze Shao
Yang Feng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Non-Monotonic
Latent Alignments
CTC
Non-Autoregressive
machine translation

Abstract: Non-autoregressive translation (NAT) models are typically trained with the cross-entropy loss, which forces the model outputs to be aligned verbatim with the target sentence and will highly penalize small shifts in word positions. Latent alignment models relax the explicit alignment by marginalizing out all monotonic latent alignments with the CTC loss. However, they cannot handle non-monotonic alignments, which is non-negligible as there is typically global word reordering in machine translation. In this work, we explore non-monotonic latent alignments for NAT. We extend the alignment space to non-monotonic alignments to allow for the global word reordering and further consider all alignments that overlap with the target sentence. We non-monotonically match the alignments to the target sentence and train the latent alignment model to maximize the F1 score of non-monotonic matching. Extensive experiments on major WMT benchmarks show that our method substantially improves the translation performance of CTC-based models. Our best model achieves 30.06 BLEU on WMT14 En-De with only one-iteration decoding, closing the gap between non-autoregressive and autoregressive models.


Non-Monotonic Latent Alignments for CTC-Based Non-Autoregressive Machine Translation
Chenze Shao 1.2, Yang Fengl.2+
KKy Laboratory of Intelligent Information Processing Institute of Computing Technology. Chinese Academy of Sciences University of Chinese Academy of Sciences shaochenzee8240cct ac. cn, fengyang@ict.ac..
Abstract
Non-autoregressive translation (NAT) models are typically trained with the cross- entropy loss, which forces the model outputs be aligned verbatim with the target sentence and will highly penalize small shifts in word positions. Latent alignment models relax the explicit alignment by marginalizing out all monotonic latent alignments with the CTC loss. However, they cannot handle monooonto alignments, which is non-negligible as there is typically global word reordering in machine translation. In this work, we explore non-monotonic latent alignments for NAT. We extend the alignment space to non-mon mon-monotonic alignments allow for the global word reordering and further consider all alignments that overlap with the target sentence. We non-monotonically match the alignments to the target sentence and train the latent alignment model to maximize the Fl score of non- monotonic matching. Extensive experiments on major WMT benchmarks show that our method substantially improves the translation performance of CTC-based models. Our best model achieves 30.06 BLEU on WMT14 14 En-De with only one- iteration decoding, closing the gap between non-autoregressive and autoregressive models
Introduction
Non-autoregressive translation (NAT) models achieve significant decoding speedup in neural machine translation [NMT, 11 47 by generating target words simultaneously [17]. This advantage usually comes at the cost of translation quality due to the mismatch of training objectives. NAT models are typically trained with the cross-entropy loss, which forces the model outputs to aligned verbatim with the target sentence and will highly penalize small shifts in word positions. The explicit alignment required by the cross-entropy loss cannot be guaranteed due to the multi-modalityy problem that there exist many possible translations for the same sentence making the cross-entropy loss intrinsically mismatched with NAT.
As the cross- cross-entropy loss can not evaluate NAT outputs properly, many efforts have been devoted 38
to designing better training objectives for NAT alignment models (29, 36 relax the alignment restriction by marginalizing out all monotonic latent alignments with the connectionist temporal classification loss [CTC, which receive much attention for the ability to generate variable length translation.
29 40
13
Among them, latent
Latent alignment models generally make strong monotonic assumption on the mapping between model output and the target sentence. As illustrated in Figure a, the monotonic assumption holds in
Conresponding author: Yang Feng Source code:
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

target
ate
pizza
this
afternoon
target
I ate
pizza
this
afternoon
/
/
t
output
atc
pizza
g this
aftemoon
output
this
afternoon
ate
pizza
input
imput
F
F
#
7
(a) CTC for ASR
(b) CTC for NAT
Figure 1: Illustration of the monotonic alignment assumption of CTC: (a) CTC for ASR where there a natural monotonic mapping between the speech input and ASR target, (b) CTC for NAT where there exists global word reordering that induces non-monotonic alignment. is the blank token that means "output nothing".
classic application scenarios of CTC like automatic speech recognition (ASR) there is natural monotonic mapping between the speech input and ASR target. However, non-monotonic alignments there is typically global word reordering, which is
negleglii in machine translation common source of the multi-modalit problem. As Figure 1b shows, when the target sentence is "I ate pizza this afternoon" but the model produces translation with a different but correct word ordering "this afternoon I ate pizza", the CTC loss cannot handle this non-monotonic alignment between output and target and wrongly penalizes the model.
In this paper, we propose model non-monotonic latent alignments for non-autoregressive machine translation. We first extend the alignment space from monotonic alignments to non-monotonic alignments to allow for the global word reordering in machine translation. Without the monotonic structure, we have to optimize the best alignment found by the Hungarian algorithm 46 28 since becomes difficult to marginalize out all non-monotonic alignments with dynamic programming, This difficulty be overcome by not requiring exact match between alignments and the target sentence. In practice, it not necessary to have the translation include exact words as contained in the target sentence, but it would be favorable to have large overlap between them. Therefore, further extend the alignment space by considering all alignments that overlap with the target sentence. Specifically, we are interested in the overlap of n-grams, which is the core of some evaluation metrics (e.g.. BLEU). We accumulate n-grams from all alignments regardless of their positions and non-monotonically match them target n-grams. The latent alignment model is trained to maximize the Fl score of n-gram matching, which reflects the translation quality to a certain extent 3
We conduct experiments major WMT benchmarks for NAT (WMT14 En<>De, WMT16 En)+Ro), which shows that our method substantially improves the translation performance and achieves comparable performance autoregressive Transformer with only one-iteration parallel decoding.
2 Background
2.1 Non-Autoregressive Machine Translation
1i proposes non-autoregresssvv neural machine translation, which achieves significant decoding speedup by generating target words simultaneously. NAT breaks the dependency among target tokens and factorizes the joint probability of target words in the following form:
T
plylæ, x, 0) = II (c
(1)
where is the source sentence belonging to the input space X, = (v1..., ..., yr is the target sentence belonging to the output space ), and p. (yt [x; 0) indicates the translation probability in position t.
The NaAi has length predictor that takes encoder states as input to predict the target length. During the training, the target length set to the golden length, and the vanilla-NAT is trained with the cross-entropy loss, which explicitly aligns model outputs to target words:
(CE(0) = log(p((ve(x,))
(2)

---

C.C.A.AAA..BBBBBB...C.
A, A, CTC. (a)
SCTC: (a)
Figure 2: An example of target sentences obtained from collapsing functions of CTC and SCTC. The collapsing function of SCTC 3 which only removes blanks in the alignment a, where the collapsing function of CTC removes repetitions first and then removes blanks.
2.2 NAT with Latent Alignments
The vanilla-NAT suffers from two major limitations. The first limitation the explicit alignment required by the cross-entropy loss, which cannot be guaranteed due to the multi-modality problem and therefore leads to the inaccuracy of the loss. The second limitation is the requirement of target length prediction. The predicted length may not be optimal and cannot be changed dynamically, often required to use multiple length candidates and re-score them to produce the final translation.
The two limitations can be addressed by using CTC-based latent alignment models 15 which extend the output space y with a blank token that means 'output nothing'. We define the extended output space as )" Following prior works [14 36 we refer to the elements a E )" as alignments, since the location of the blank tokens determines an alignment between the extended output and target sentence. Assume the generated alignments have length Ta. we define a function ((y) which returns a subset of y including
29 36]
possible alignments of length T, for y, where 3-1 ) + y is the collapsing function that first collapses all consecutive repeated words in a and then removes all blank tokens to obtain the target sentence. As illustrated in Figure 2 the alignment (e, A, A. A, B, B, C, e, D) is collapsed to the target sentence y (A, A, B, C, D) with a monotonic mapping from target positions to alignment positions. During the training, latent alignment models marginalize all alignments with the CTC loss [I5]
logplylæ, 0) log Σ
p(alæ,0), aEB(M)
where the alignment probability p(a|z, 0) is modeled by non-uutoregressivv Transformer:
p(a,z, 0) II Pe ())
NAT models with latent alignments achieve superior performance by overcoming the two major limi-
tations of NAT 29
66
16
25
33,
49 The weakness in handling non-monee non-monotonic latent alignments
noticed in previous work
36
16 but remains unsolved.
Approach
In this section, explore non-monotonic latent alignments for NAT. We introduce simplified CTC loss section 3.T which has simpler structure that helps further analysis and derivation under regular CTC loss. We explore non-monotonic alignments under the simplified CTC loss in section and utilize the results to derive non-monotonic aligmments under the regular CTC loss in section Finally, introduce the training strategy to combine monotonic and monooonto latent
3.2
3.3
alignments in section 3.4
3.1
Simplified Connectionist Temporal Classification
In CTC, the collapsing function 8-1 maps an alignment to a target sentence in two steps: (1) collapses all consecutive repeated words, (2) removes all blank tokens. The two operations make the alignment space B(y) complex. Therefore, we first introduce the simplified connectionist temporal classification loss (SCTC), which has a simpler structure that helps the derivation of oo-monotonic alignments, and the results SCTC are helpful for further analysis under the regular CTC loss.

---

To simplify the alignment structure, we consider only using one operation in the collapsing function. One option only collapsing all consecutive repeated words in the alignment. The concern is that will limit the expressive power of the model. For example, the target sentence y (A, A, B, C, D) has probability since has repeated words that cannot be mapped from any alignment. Therefore, we favor another option that only removes all blank tokens. In this way, the expressive power stays the same and the alignment space becomes much simpler, where alignments of y simply contain T larget words and T, - 7'' blank tokens.
We denote this simplified loss as SCTC, and illustrate the difference between CTC and SCTC in Figure 2 The alignment space for the target sentence is defined as (y). and the collapsing function defined as B. We can still obtain the translation probability p(ylæ,, 0) with dynamic programming. We define the forward variable a, (s) to be the total probability of yi:s considering all possible alignments 01:t, which can be calculated recursively from 0; 0;-1(8) and 1 1):
a2 = 0:11 (s () (x, 0) + (:=1(s)),(((zz,))))
(5)
Finally, the total translation probability pay|ær, 0) is given by the forward variable or, (T), so we can train the model with the cross-entropy loss (0) log ar. (7).
3.2
Non-Monotonic Alignments under SCTC
3.2.1 Bipartite Matching
In this section, we explore non-monotonic alignments under the SCTC loss, which are helpful for further analysis under the regular CTC loss. We first extend the alignment space from monotonic alignments (y) to non-monotonic alignments Y (v) to allow for the global word reordering in machine translation, where (v) is defined as:
Yo(y) E'EP(w) 3.(3').
In the above definition, P(y) represents all permutations of y. For example, P((A, B)) f(A, B), (B, By enumerating P(y), we consider all possible word reorderings of the tar- get sentence. Ideally, we want to traverse all alignments in (v( to calculate the likeliiid loss:
sum (0) log
Σ
p(a|a, 0).
(ET,(1)
However, without the monotonic structure, it becomes difficult to marginalize out all latent alignments with dynamic programming. Alternatively, we can minimize the loss of the best alignment, which is an upper bound of Equation 7
Linas (0) - log max p(a/æ, 0).
avr,(y)
D we formulize finding the best aligmment as a maximum bipartite matching problem and solve it with the Hungarian algorithm (28). Specifically, we observe that the aligmment space (v) is simply permutations of T target words and T. - T blank tokens. Therefore, finding the best alignment is equivalent to finding the best bipartite matching between the Ta model predictions and the T target words plus T. T blank tokens, where the two sets of nodes are connected by edges with the prediction log-probability as weights.
Following prior work 46
3.2.2 N-Gram Matching
Without the monotonic structure desired for dynamic programming, calculating Equation becomes difficult. This difficulty is also caused by the strict requirement of exact match for alignments, which makes it intractable to simplify the summation of probabilities. In practice, is not necessary to force the exact match between alignments and the target sentence since large overlap is also favorable. Therefore, we further extend the alignment space by considering all alignments that overlap with the target sentence. Specifically, we interested in the overlap of n n-grams, which is the core of some evaluation metrics (e.g., BLEU).
We propose a non-monotonic and non-exclusive n-gram matching objective based on SCTC to encourage the overlap of n-grams. Following the underlying idea of probabilistic matching [39) we introduce the probabilistic variant of the n-gram count to make the objective differentiable.

---

Specifically, we use Cy( to denote the occurrence count of n-gram g (g1,..., ..... 9r.) in the target sentence y, and use ()) denote the probabilistic count of n-gram g for the model with imput and parameter 0, where the superscript s indicates SCTC. For simplicity, we omit the source sentence xr in (0) and the following notations. The probabilistie n-gram count obtained by accumulating n-grams from all possible target sentences:
()
Σ OC, vEY
Ep(aala (1, BE
The calculation of C² (0) is the core of this method and will be described in detail later. We use (0) to denote the match count of n n-gram between the latent alignment model and the target sentence y (we omit y in Mÿ (0) for simplicity), which is defined as follows:
()) (v), C)),
(10)
Our objective is to maximize the n-gram overlap between the target sentence and model output, so we can train the model to maximize the precision or recall of n-gram matching:
(0) Egec. C2(0) (0) = Egea. Mg
Eger. ()
(1I)
EOEc, C₂(y)
where we use G, to denote the set of all n n-grams. However, since latent alignment models have the ability to control the translation length, both precision and recall cannot accurately reflect the translation quality. The precision will encourage short translations to reduce the denominator, while the recall prefers long translations. Therefore, consider both of them and maximize the F1 score:
2. EYEG. (0)
(0) = (0) = EgEG,
(12)
where we use Gn to denote the set of all n-grams. In the numerator of Equation 12 Mg(() is non-
zero only if (v) is non-zero, so we only need to calculate probabilistic n gram counts (0) for n-grams in the target sentence. In the denominator, Egeg,, Cg(y) equals to the constant 7 - n + 1. Therefore, there are only two tasks left: calculating (0) and the summation Egea,, (0).
For the first task of calculating (0), we first consider the simple case n = 1. The main difference between 1 I-gram matching and bipartite matching is that I-gram matching is moclexccusivve which allows us to marginalize out all alignments using the property of non non-autoregressive generation. Due to the space limit, we refer readers to Appendix.A for the derivation and directly give the probabilistic 1 1-gram count as follows:
To
C2(0) = Σ Po (oll, 0).9 E G1. t=1
(13)
where 11 denotes the set of all 1-grams g = (g1). Both 1-gram matching and bipartite matching are completely non- mo-oooooonic, which aim at generating correct target words regardless of the word order. When the target sentence is "I ate pizza this afternoon", they will think that "pizza ate I this aftermoon" " is also good translation. We consider 2-gram matching to overcome this limitation, which is also non-monotonic but takes target dependency into consideration. We first introduce a transition matrix A of size Ta x Ta, where Aij is the probability that all positions between and j blank tokens:
0
i<i+1
Arj
j=i+1.
(14)
IT (c(c 0) j j> i+1
Then we give the probabilistic 2-gram count for g (91, 922 (see Appendix.A for derivation):
Ta
To
())
Σ
(1), 0).9 E G2.
(15)
For a larger n, the probabilistic n- n-gram count is:
Te
C)
Pt () x
II
A in Pi, (g g G..
(16)
5

---

Equation 15 can be efficiently calculated in matrix form with O time complexity. For Equation 16 we need (nTT") operations to calculate by brute force, which is impractical when n is large. We refer readers to Appendix.B B for the ( algorithm, which is similar to calculating the n-step transition probability of Markov chain.
For the second task of efficiently calculating the summation of probabilistic n-gram counts EgEa, (0), we also consider the simple case 1 first, where the summation can be cal- culated by directly summing all probabilities:
To
-0
- -0==>>mmaee 0), e EG [=1
(17)
g
For n. >> 1, the search space is too big for the direct summation. Fortunately, the definition of C7 in Equation alllows us to safely extend the result of to larger since n-grams count in sentence is always n less than 1 1-grams count.
E gEG.
Σ ( n gEG,
(18)
In summary, with the above results (Equation 13-18), we are able to efficiently calculate the loss defined in Equation 12 to maximize the Fl score of non-monotonic n-gram matching, which reflects the translation quality certain extent.
3.3 Non-Monotonic Alignments under CTC
In this section, we explore mon-monotonic latent alignments under the regular CTC loss. Under SCTC, we can use both the bipartite matching and n-gram matching to train the latent alignment model. However, bipartite matching is infeasible under the regular CTC loss. Recall that the search space (v() is simply permutations of target words and Ta T blank tokens, which allows us to find the best alignment with the Hungarian algorithm. However, as consecutive repeated words will be collapsed in the regular CTC loss, alignments in (v) may contain different words, so the best alignment cannot be found by the Hungarian algorithm.
Fortunately, for n-gram matching, we can utilize previous results of SCTC to help the analysis and derivation under the regular CTC loss. As the output of the collapsing function (a) simply contains more repeated words than & (a), we can first identify how much influence these repeated words have, and then remove this influence from the results of SCTC. We first formally define the probabilistic n-gram count under the regular CTC:
Cg( = Σ p(aax, (),,)()) aE)
(19)
We define (g(()) to be the gap bewteen (0) and (0) when n which represents the number of repeated words 9I removed by the collapsing function. (0) can be efficiently calculated as follows (see Appendix.C C for derivation):
Tw-1 (2(0) = -,,0) Σ 0100
0), G1.
(20)
The above equation enables us to efficiently calculate (0) and E, ()) when E G For 2- Orgamms (91.92), we divide it into two cases 91 = 92 and 91 * 92. The collapsing of consecutive repeated words will only remove 2 2-grams of the first case, where each removed word corresponds to removed 2-gram. Taking Figure 2 as an example, the output of SCTC 3-1(a) have two more 2-grams ((A, A), (B, B)} of the 9i f2 case than the output of CTC 3 (a), and they contain exactly the same 2-grams of the 91 * case. Therefore, we can directly reuse previous results of SCTC and the number repeated words R2 to calculate the probabilistic 2-gram count:
(,( { (0) (0), 9I 92 ()),
g G2.
(21)
g1 + g2
Unfortunately, for > 2, there is no such clear relationship between (0) and (O), where the CTC output contain n- n-grams (for example, (A, B, C) in Figure that the SCTC output does not

---

have. If we directly formulize (0) like in Equation 16 it will be more complex and the O
algorithm is no longer applicable. Therefore, we do not use n- n-gram matching with n under the
regular CTC loss. Finally, we define the match count Mg (0) similarly as in Equation 10 and train the
CTC-based latent alignment model to maximize the F1 score:
(n) 2. 2. M9(
(22)
(1E6,((((0))000000)
3.4 Training
In the above, we propose non-monotonic training objectives for the latent alignment model. They need to be combined with monotonic training objectives, otherwise, they may generate translations like "pizza ate this afternoon". Therefore, we first pretrain the latent alignment model with monotonic training objectives like SCTC or CTC. The model pretrained by SCTC can be finetuned with the bipartite matching objective (Equation pretrained by CTC can be finetuned with the n-gram matching objective (Equation 22 and the choices of are limited to {1,
S or the n-gram matching objective (Equation 12 The model
4 Experiments
4.1 Experimental Setup
Datasets We evaluate our methods on the most widely used public benchmarks in previous NAT stud- ies: WMT14 English++German (Ent+De, 4.5M sentence pairs) [5] and WMT16 Romania (En++Ro, 0.6M sentence pairs) [6. For WMT14 En++De, the validation set is newstest2013 and the test set is newstes12014. For WMT16 En the validation set is newsdev-2016 and the test set is newstest-2016. Following prior works, we use tokenized BLEU 3. to evaluate the translation quality. Considering that BLEU might be biased report METEOR 2 which is not directly based on n n-gram overlap. We learn a joint BPE model [7 with 32K operations to process the data and share the vocabulary for source and target languages.
method optimizes the n- I-gram overlap, also
Knowledge Distillation Following prior works 17 we apply sequence-level knowledge distillation 26 to reduce the complexity of training data. We use Transformer- base setting for the teacher model and train NAT on the distilled data.
as our autoregressive baseline as well as the teacher model. We uniformly copy encoder outputs to construct decoder inputs. For CTC-based NAT models, the length for decoder inputs is 3x as long as the source length. On WMT14 En we use a dropout rate of 0.2 to train NAT models and use dropout rate of 0. for finetuning. On WMT16 En++Ro, the dropout rate is 0.3 for both the pretraining and finetuning. We use the batch size 64K and train NAT models for 300K steps on WMT14 En and 150K steps on WMT16 En During the finetuning, we train NAT models for 6K steps with the batch size 256K. All models are optimized with Adam 27 with 3 = (0.9, 0.98) and c 10-8 The learning rate warms up to 5 10-1 within 10K steps in the pretraining and warms up to 10- within 500 steps in the finetuning. We use the GeForce RTX 3090 GPU to train models and measure the translation latency. We implement our models based on the open-source framework of fairseq [MIT License, 31I.
Implementation Details We adopt Transformer-base
Decoding For autoregressive models, we use beam search with beam size 5 for the decoding. For Vanilla-AAT we use argmax decoding to generate the translation. For CTC-based models, we can also use argmax decoding to generate the alignment and then collapse it to obtain the translation. Besides, following (29) [o], we can use beam search decoding combined with a 4-gram language model 20 to find the translation that maximizes:
p(yl x. 0) + a PLM + 3 log(lv/),
(23)
where and 3 are hyperparameters for the language model score and length bonus. We use fixed beam size 20 with grid-search a, B. The beam search does not contain any neural network computations and can be implemented efficiently in C++
7

---

Table 1: Performance comparison between baselines and methods on WMT14 En-De test set.
Model
BLEU
METEOR
Speed
Transformer
27.54
54.38
1.0x
Base
Vanilla-NAT
19.32
45.79
15.5x
SCTC
25.26
52.06
+bipartite
26.13
53.17
+1-gram
26.22
53.20
SCTC
+2-gram
26.79
53.54
14.7x
+3-gram
26.66
53.48
+4-gram
26.62
53.43
CTC
26.34
53.15
CTC
+1-gram
27.16
53.95
14.7x
+2-gram
27.57
54.50
Table 2: Performance comparison between models and existing methods. The speedup is
measured on WMT14 En-De test set with batch size AT means autoregressive. Iter. denotes the number of iterations inference time. "_' indicates that the result is not reported. '12-1' means the Transformer with 12 encoder layers and decoder layer 2
WMT14
WMT16
Models
Iter.
Speed
EN-DE
DE-EN
EN-RO
RO-EN
Transformer (teacher)
1.0x
27.54
31.57
34.26
33.87
AT
Transformer (12-1)
N
2.6x
26.09
30.30
32.76
32.39
KD
N
2.7x
27.61
31.48
33.43
33.50
NAT-FT
17
15.6x
17.69
21.47
27.29
29.06
NAT-REG
48
27.6x
20.65
24.77
AXE II
23.53
27.90
30.75
31.54
GLAT 3
15.3x
25.21
29.84
31.19
32.04
Non-CTC
Seq-NAT
42
15.6x
25.54
29.91
31.69
31.78
NAT
AligNART
455
13.4x
26.40
30.40
32.50
33.10
CMLM 12
10
27.03
30.53
33.08
33.31
LevT IE
2.05
4.0X
27.27
33.26
JM-NAT 19
10
27.69
32.24
33.52
33.72
RewniteNAT
2.70
27.83
31.52
33.63
34,09
CMLMC 2A
10
28.37
31.41
34.57
34.13
CTC 2D
16.56
18.64
19.54
24.67
Imputer
25.80
28.40
32.30
31.70
GLAT+CTC
14.6x
26.39
29.54
32.79
33.84
CTC-based NAT
REDER 40
15.5x
26.70
30.68
33.10
33.23
+ beam&rreanking
5.5x
27.36
31.10
33.60
34.03
DSLP 21
14.8x
27.02
31.61
34.17
34,60
Fully-NAT 16
16.8x
27.20
31.39
33.71
34.16
Imputer
28.20
31.80
34.40
34.10
Bag-of-ngrams
15.5x
25.28
29.66
31.37
31.51
Our
rescore candidates
9.5x
25.95
30.43
32.78
32.92
Implementation
OAXE 9
15.3x
24.70
29.30
31.06
31.37
rescore 5 candidates
14.0 x
25.78
30.29
32.45
32.63
CTC w/o finetune
14.7x
26.34
29.58
33.45
33.32
CTC w/ NMLA
14.7x
27.57
31.28
33.86
33.94
beam&lm
5.0%
28.35
32.27
34.72
34.95
Our work
DDRS w/o finetune
14.7x
27.18
30.91
34.42
34.31
DDRS w/ NMLA
14.7x
28.02
31.80
34.73
34.76
beam&lm
5.0%
28.63
32.65
35.51
35.85

---

4.2
Main Results
We first compare the performance of different non-monotonic training objectives for latent alignment models. Baseline models include the Transformer, vanilla-NAT, SCTC, and CTC, and we finetune latent alignment models with bipartite matching and n-gram matching objectives. In Table I we report BLEU and METEOR scores along with the speedup on WMT14 En-De test set.
From Table D we have the following observations: (1) Non-monotonic training objectives can effectively improve the performance of latent alignment models, which improves SCTC by 1.53 BLEU and 1.48 METEOR and improves CTC by 1 1.23 BLEU and 1.35 METEOR, illustrating the importance of non-monotonic alignments. (2) Despite having the same expressive power, SCTC underperforms CTC in the task of non non-autoregressive translation. We speculate that it is because of the difference in the distribution of the alignment space. Most target sentences do not contain repeated words, in which case the alignment space of SCTC 5 (1) is subset of (-1()), making the ability of SCTC relatively weak. (3) SCTC achieves the best performance with 2-gram finetuning, and then the performance decreases with the increase of n due to the sparsity of higher rank n-grams. suggests that we do not need a large n and relieves the concern on CTC that we cannot calculate n-gram matching for > 2. (4) The correlation between the reported BLEU and METEOR scores is very strong, suggesting that BLEU is not biased when evaluating our methods. Therefore, we can safely use BLEU to evaluate the translation quality in the following experiments.
We call the 2-gram matching objective NMLA since represents our best non monotonno training method for latent alignment models. Besides CTC, we also apply NMLA finctuming on a stronger baseline DDRS 43 which trains the CTC model with multiple references. To extend our method to multiple references, we calculate the n-gram counts for each reference and use their average as the target n-gram count. We compare their performance with existing approaches in Table 2 Notably, CTC with NMLA achieves comparable performance to autoregressive Transformer on all benchmarks, demonstrating the effectiveness of our method. On the strong baseline DDRS, NMLA finetuning also brings remarkable improvements and outperforms autoregressive baselines. With the help of beam search and 4-gram language model, NMLA even outperforms other iterative NAT models and the autoregressive Transformer with only one-iteration parallel decoding. Compared to our implementations of prior works that explore non-monotonic matching for NAT [4 [. we can see that they underperform the CTC baseline even when rescoring 5 candidates, and their speedup after rescoring also lays behind CTC. It demonstrates the importance of cxploring non-monotoni matching on latent alignment models, which is more desirable for its flexibility of generating predictions with variable lengths.
As NMLA finetuning brings remarkable improvements over the strong baseline DDRS, we further conduct experiments to verify whether the NMLA orthogonal to other methods. In Table 3 we reimplement GLAT+CTC (Fully-NAT) [6 33 based on their open-source code finetune it with the NMLA objective. Our implementation replaces the SoftCopy mechanism with UniformCopy. which surprisingly improves 0.7 BLEU and achieves 27.07 BLEU WMT14 En-De test set. Finetuning with NMLA can further improve its performance to 27.75 BLEU, demonstrating the extensibility of NMLA. In Table 4 apply NMLA on the big version of DDRS to explore its capability on large models. The results show that the performance of DDRS-big can be greatly boosted by NMLA finetuning. Combined with beam search, it achieves 30.06 BLEU WMT14 14 En-De, which narrows the performance gap between non-autoregressive and autoregressive models to new level.
Table 3: BLEU scores of GLAT+CTC and NMLA finetuning on WMT14 En-De test set.
Table 4: BLEU scores of DDRS-big and
NMLA finetuning on WMT14 En-De test set.
Model
BLEU
Speed
Model
BLEU
Speed
GLAT+CTC w/o finetune
27.07
14.7x
DDRS-big w/o finetune
28.24
14.1 x
GLAT+CTC NMLA
27.75
14.7x
DDRS-big w/ NMLA
29.11
14.1
beam&lm
28.46
5.0x
+ beam&Im
30.06
4.8x
bunps///ittnb.com///LC77777L

---

4.3 Training Cost
In this section, we will analyze the training cost and show that our method is cost-effective. The additional training cost of our method only comes from the NMLA finetuning. In Table we compare the training cost of CTC and NMLA on WMT14 En-De. We measure the training speed by Words Per Second (WPS) and find that the training speed of NMLA is slower than CTC, which can be attributed to the large calculation cost of the NMLAA objective. However, since we only finetune the model for 6K steps, the overall cost of the NMLA finetuning is much smaller than the CTC pretraining. On the whole, the training cost of CTC with NMLA is less than 1.2x compared to the CTC baseline.
Table 5:
The comparison of CTC pretraining and NMLA finetuning. 'WPS' means words per second. "Step' means the number of training steps. 'Batch' means batch size. 'Time' means the training time measured on GeForce RTX 3090 GPUs.
WPS
Step
Batch
Time
CTC pretraining
141K
300K
64K
26.4h
NMLA finetuning
60K
6K
256K
5.0h
5
Related Work
first proposes non-autoregressive machine translation accelerate the decoding process of NMT. The cross-entropy loss is applied to train the Vanilla-NAT, which requires the explicit alignment between the model output and target sentence. The explicit alignment cannot be guaranteed due to the multi-modality problem, and many efforts have been devoted to mitigating this issue.
17
One research direction enable explicit alignment by leaking part of the target-side information during the training. [17] uses fertilities as a latent variable to specify the number of output words corresponding each input word.
23 35 E apply vector quantization train NAT with discrete latent variables, and 30 [6] apply variational inference to train NAT with continuous latent also introduce latent variables that carry the reordering information. Besides using latent variables, 12 directly feeds the partially masked target sentence as the decoder input, and applies glancing sampling strategy to control the mask ratio.
variables. 34
Another research direction is to propose better training objectives for NAT. 44 introduces two auxiliary regularization terms to reduce errors of repeated translations and incomplete translations. 40 directly optimizes sequence-leve level objectives with reinforcement learning. 22 trains NAT with deep supervision and feeds additional layer-wise predictions. 43 creates a dataset with multiple references and dynamically selects an appropriate reference for the training. Prior to this work, have explored ways to establish alignment between the model output and target sentence, which can promote the training of NAT. Their limitation is that they are basically built on the framework of Van, which requires a target length predictor and cannot dynamically adjust the output sequence length. CTC-based latent alignment models 29 36 [6] are more desirable due to their superior performance and the flexibility of generating predictions with variable lengths. However, latent alignment models have weakness in handling non-monotonic alignments, which is noticed in previous work [36 [6 and solved in this work.
41 13 D
Conclusion
Latent alignment models cannot handle non- mon-ooootonic alignments during the training, which is nonnegligible in machine translation. In this paper, we explore non-monotonic latent alignments for NAT and propose two matching objectives named bipartite matching and n-gram matching. Experiments show that our method achieves strong improvements on latent alignment models.
7 Acknowledgement
We thank the anonymous reviewers for their insightful comments. This work was supported by National Key R&D Program of China (N0.2018AAA0102500),
10

---

References
[1] D. Bahdanau, K. Cho, and Y. Bengio. Neural machine translation by jointly learning to align and translate. In Y. Bengio and LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015. URL htthttp:///e arxiv..xvvvvv. org/ 1409. 0473
[2] S. Banerjee and A. Lavie. METEOR:
An automatic metric for MT evaluation with improved correlation with human judgments. Proceedings of the ACL. Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https //aclanthology org/ 00rg//0550005
[3] Y. Bao, H. Zhou, J. Feng, M. Wang. S. Huang, J. Chen, and L. Lei. Non-autoreerressive transformer by position learning, arXiv preprint arXiv:1911. 2019.
[4] Y. Bao, S. Huang, T. Xiao, D. Wang. X. Dai, and J. Chen. Non-autoreerressive translation by learning target categorical codes. In Proceedings of the 2021 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, pages 5749-5759, Online, June 2021. Association for Computational Linguistics. doi: 10. 18658/ v1/2021 11/2021.naacl-ne .na.cc-maii..555.
URL attpss://cclanttoooogg wrg/o022.12222 http//2.....aammii.... naacl-main. 458
[5] O. Bojar, C. Buck, C. Federmann, B. Haddow, Kochn, J. Leveling, C. Monz, P. Pecina, M. Post, H. Saint-Amand, R. Soricut, L. Specia, and A. Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 12-58, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. StLhtt:://
aclweb. org/ hogg/anthology///e /ttpoo//////////14133/30
[6] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham, B. Haddow, M. Huck, A. Jimeno Yepes, Koehn, V. Logacheva, C. Monz, M. Negri, A. Neveol, M. Neves, M. Popel, M. Post, R. Rubino, C. Scarton, L. Specia, M. Turchi, K. Verspoor, and M. Zampieri. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation, pages 131-198, Berlin, Germany, August 2016. Association for Computational Linguistics. URL http://www wcwweb............... http://nnhoooogo///o/o///////
2301
[7] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pages 213-229. Springer, 2020.
[8] S. Castilho, J. Moorkens, F. Gaspari, I. Calixto, J. Tinsley, and A. Way. Is neural machine translation the new state of the art? Prague Bull. Math. Lin- guisties, 108:109-120. 2017. URL http://ufal. mff cuni wwtpc///pppp///////ee art cea-castille tho hooeillloomoookesssooooooeo daspari-tinslee calixto- way. pdf
[9] C. Du, Z. Tu, and J. Jiang. Order-agnostic
entropy for non-autoregressive machine translation. In M. Meila and T. Zhang, editors, Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 2849-2859, PMLR, 2021. URL mlr. hatss/vv39/dullccom
[10] M. Fomicheva and L. Specia. Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments. Computational Linguistics, 45(3):515 55555 09 2019. ISSN 0891-2017. doi: 10.1/62/coli a 003360 URL lnttps://doiii orgo 10. 11w2/cooi...... a a_00356)
[11] X. Geng, X. Feng, and B. Qin. Learning to rewrite for non-autoregressive neural machine translation. In Proceedings of the 202/ Conference on Empiricul Methods in Natural Lan- guage Processing, pages 3297-3308, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguistics. doi: 10. 18653/11/2022 hetp22..emnpp-ppee malpp.mman..22 265. URL
https://aclanthologyy org 2021 emnlp-main.28 265
11

---

[12] M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer. Mask-predich Parallel decoding of conditional masked language models. In Proceedings of the 2019 Conference on Em- pirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLP), pages 6112-6121, 2019. URL https://www, wwwwww.www....... org. orgganthologryD19---00
[13] M. Ghazvininejad, V. Karpukhin. Zeulemoyer, and O. Levy. Aligned cross entropy for non oon-autoegrresivve machine translation. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 3515 3523.. PMLR, 2020. URL
mlr. pptpss/el1/gaaaereeeeee hta.1////ggaaiiiiieeeeea000 htmml
[14] A. Graves. Sequence transduction with recurrent neural networks. arXiv:/211.377 2012.
arXiv preprint
[15] A. Graves, S. Fernández, F. Gomez, and J. Schmidhuber. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23rd International Conference on Machine Learning, ICML '06, page 369-376, New
York, NY, USA, 2006. Association for Computing Machinery. ISBN 1595933832. doi: 10.1145/1 1448444114389999 143891. URL battps://boi.org. 1ee.e11111111111 1143891
[16] J. Gu and X. Kong. Fully non-autoregressive neural machine translation: Tricks of the trade, 2020.
[17] J. Gu, J. Bradbury, C. Xiong, V. K. Li, and R. Socher. Non-autoregressive neural machine translation. In 6th International Conference on Learning Representations, ICLR 2018, Van- couver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018. URL
https::/openneviis..ettt/
[18] J. Gu, C. Wang, and J. Zhao. Levenshtein transformer. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' B'uehééBue, E. B. Fox, and R. Garnett, editors, Advances in Neu- ral Information Processing Systems 32: Annual Conference on Neural Information Pro- cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, neur apeurpp..o wweurips.ccppppeeeeeeeeeee cc/ppaper//01//aaeeeeeeeeeeeeeee
pages 11 11119-111111 1189, 2019. URL http:ttttp/://ttttttttst 775556826626665cc0c2444777667773 http:/00oeeoee---baaatt...... Abstract.
[19] J. Guo, L. Xu, and E. Chen. Jointly masked
model for non-autoregresssvv
neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 376-385, Online, July 2020. Association for Computational Linguistics. doi: 10. 1125 acl.mmmm..... URL. hattps://aclaatthoogg. org/ 2220
acl mellmmiin.33 36
[20] K. Heafield. KenLM: Faster and smaller language model queries. In Proceedingy of the Sixth Workshop on Statistical Machine Translation, pages 187-197, Edinburgh, Scotland, July 2011 1. Association for Computational Linguistics. URL http:httccccaaattaaeee
org/ //////11222111
[21] C. Huang, H. Zhou, O. R. Zaiane, Mou, and L. Li. Non-autooreressive translation with layer-wise prediction and deep supervision. ArXiv, abs/21 2021.
[22] X. S. Huang, F. Perez, and M. Volkovs. Improving non-autoregressive translation models without distillation. In International Conference on Learning Representations, 2022. URL nttps://openrevits
[23] L. Kaiser, S. Bengio. A. Roy, A. Vaswani, Parmar, Uszkoreit, and N. Shazeer. Fast decoding in sequence models using discrete latent variables. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning, volume 80 of Pro- ceedings of Machine Learning Research, pages
PMLR, 10-15 Jul 2018. URL
mlr. www.:rreessse.eeee /at.ppee///////aaaeellaaaaa
[24] J. Kasai, N. Pappas, H. Peng, J. Cross, and N. Smith. Deep encoder, shallow decoder: Reevalu- ating non-autoreereessvve machine translation. In International Conference on Learning Repre- sentations. 2021. htw.l//ppp.ppppeeeeeeeeee/////es
12

---

[25] Z. Kasner, J. Libovický, and J. Helel. Improving fluency of non -on-auorrerresive machine translation, 2020.
[26] Y. Kim and A. M. Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 1317-1327, Austin, Texas, Nov. 2016. Association for Computational Linguistics. doi: 10.18653/w1/Di6-111
URL BRLLhttps:///
aclweb. orgg/ anthology/D16 D16-1199
[27] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprini arXiv:1412.6988) 2014.
[28] H. W. Kuhn. The hungarian method for the assignment problem. Naval Research Logistics Quarterly, 2(1-2):83-97, 1955. doi: http:///dob.ooo//1..000/////aa /onlinelibrary wwwey. wwwp//oo/abs//..11000222 100.///a...0//aa//../// 3800020009
URL https
[29] J. Libovický and J. Helcl. End-to-end non-autoregressive neural machine translation with connectionist temporal classification. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3016-3021, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10. 10.18653/1/D1181155 1336. URL https 7/www. aclweb. org/ anthology/DDs DD8- 1336
[30] X. Ma, C. Zhou, X.
Li, G. Neubig, and E. Hovy. FlowSeq: -on-autoregressive conditional sequence generation with generative flow. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNIPP, pages
Hong Kong, China, Nov. 2019. Association for Computational Linguistics. doi: 10.18653/11/D1--44555 URL https://www aclweb. owwebboorg/ntthooeoo www.or///..aooooooeooo-oe. 1437
[31] M. Ott, S. Edunov, A. Baevski, A. Fan, S. Gross, N. Ng. D. Grangier, and M. Auli. fairseq: A fast, extensible toolkit for sequence modeling. In Proceedings of NAACL-HLT 2019: Demonstrations, 2019.
[32] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evalua- tion of machine translation. In Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pages 311-318, Philadelphia. Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.1115/1073083.10733333 URL https://www aclweb. htgg/aanhoooor//P22-110
[33] L. Qian, H. Zhou, Y. Bao, M. Wang, L. Qiu, W. Zhang, Y. Yu, and L. Li. Glancing transformer for non-autoregressivve neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1993-2003, Online, Aug. 2021. Association for Computational Linguistics. doi: 10.18653/v 1/20l11211222222 w.c.laaclloooooo mell-oong.115 URL org/ 2021 2021.acl-a acl.clllllo long. 155
[34] Q. Ran, Y. Lin, P. Li, and J. Zhou. Guiding non-autoregreessvve neural machine translation decoding with reordering information. In Thirty-Fifth AAAI Conference on Artificial Intelligence, AAAI 2021, Virtual Event, February 2-9, 2021, pages 13727-13735. AAAI Press, 2021. URL https://ojs. aaai. org/ indiiidexxxx. php/
[35] A. Roy, A. Vaswani, A. Neelakantan, and N. Parmar. Theory and experiments on vector quantized autoencoders. arXiv preprint arXiv: 1000,, 2018.
[36] C. Saharia, W. Chan, S. Saxena, and M. Norouzi. Non-autoregressive machine translation with latent alignments. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1098-1 1088,110,,000 Online, Nov. 2020. Association for Computational Linguistics. doi: 10.18653/v h///22.ee02n/2p.../ 1alip-mmii...... URL https:://www aacwebb. org/ anthology/2020. emn 1p main. 88
[37] R. Sennrich, Haddow, and A. Birch. Neural machine translation of rare words with subword units. In ACL, August 2016. URL http::////et http:://ww.accwebb aclweb. org/ http://wwww...ooorro.aaaoo//////
antholoyy/PP16-116. 1162
13

---

[38] Y. Shan, Y. Feng, and C. Shao. Modeling coverage for non-autoregreesive neural machine translation. Xrviv preprint arXiv:2104.1189,, 2021.
[39] C. Shao, X. Chen, and Y. Feng. Greedy search with probabilistic n-gram matching for neural machine translation, In Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages 4778-4784, Brussels, Belgium, Oct.-Nov. 2018. Association for Computational Linguistics. doi: 10D11/663//111D11//1 1510. URL https: wwww://wwwwwwwwwbbbbb//.... aclweb org/ anthology/D18-1510
[40]
C. Shao, Y. Feng. J. Zhang, Meng, X. Chen, and J. Zhou. Retrieving sequential information for non-autoorerresiive neural machine translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. pages 3013-3024, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v 1281999-1188--- 1288.
URL https
//www wwwweb..wb..... org/ anthology/PPS: 1288
[41] C. Shao, J. Zhang, Y. Feng, F. Meng, and J. Zhou. Minimizing the bag-of-ma off-nrrams difference for non-autoregressiive neural machine translation. In The Thirly-Fourth AAAI Conference on Artificial Intelligence, AAA/ 2020, New York, NY, USA, February 7-12, 2020, pages 198-
205. AAAI Press, 2020. URL https://aaaai:: http://aaa..orr. org/oo////.oo.ooooooooo htt///jssinnnn /inddxddxx..// wwnex.ppp//iiei...emmeppp Aahp/AAAT/rrea /ttp/////aaiileeeiiiireeee 5351
[42] C. Shao, Y. Feng, J. Zhang, F. Meng, and J. Zhou. Sequence-Level Training for Non- Autoregressive Neural Machine Translation. Computational Linguistics, pages 1-35, 10 2021. ISSN 0891-2017. doi: 10.1 00421. URL http::tts/// :/dp::://doo.. org 10. 11l2/co11.....
00421
[43] C. Shao, X. Wu, and Y. Feng. One reference is not enough: Diverse distillation with reference selection for non-autoregressive translation. Proceedings of the 2022 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3779-3791, Seattle, United States, July 2022. As- sociation for Computational Linguistics. doi:
URL
https: https:://aaaaathooooeoee org/ 222//2022..2 2ac...aacc.an.....mm.mm...aa maacc-main........aaa..m mma.mmaii....... 277
[44] R. Shu, J. Lee, H. Nakayama, and K. Cho. Latent- non-autoregressivee neural machine translation with deterministic inference using a delta posterior. In The Thirty-Fourth AAAI Conference Artificial Intelligence, AAAI 2020, New York, NY, USA. February 7-12, 2020. pages 8853, AAAI Press, 2020. URL https://aaaa org/ hts////s/inndess php/ AAAI/ article/view/647
[45] J. Song, S. Kim, and S. Yoon. AligNART: Non-autoregressive neural machine translation by jointly learning to estimate alignment and translate. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1-14, Online and Punta Cana, Dominican Republic, Nov. 2021. Association for Computational Linguisties. doi: 18.18613/v1/ 2021 1021.emllp-min.. 1. URL htttps://claanhholgg.. org/ 2021. wmw.pmeamlppm..........mmmmmm...
[46] R. Stewart, M. Andriluka, and Ng. End-to-end people detection in crowded scenes. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2325-2333, 2016.
[47] A. Vaswani, N. Shazeer, N. Parmar. J. Uszkoreit, L. Jones. A. N. Gomez, u. Kaiser, and I. Polosukhin. Attention is all you need. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS' 17, pages 6000-6010, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964.
[48] Y. Wang. F. Tian, D. He, T. Qin, C. Zhai, and T. Liu. Non-autoregressive machine translation with auxiliary regularization, In The Thirty-Thiin AAAI Conference on Artificial Intelligence, AAAI 2019, Honolulu, Hawaii, USA, 2019, pages 5377-5384. AAAI Press, 2019. doi: 16.1609/ v33101.33015377. URL httpttttt//oo..../ org/10, 1ww0//////../////.......// 1609/aaai.... wettaaa.vv3.100 v33101.330155777
[49] Z. Zheng, H. Zhou, S. Huang. J. Chen, J. Xu, and L. Li. Duplex sequence-to- for reversible machine translation. In M. Ranzato, A. Beygelzimer, Dauphin, Liang, and J. W. Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 21070 21070-21084, Curran Associates, Inc., 2021. URL https: httpc::/p/rrooiieeeeeeeeo www.iiins..uurii......o
learning
pdf
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits. hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g.. with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "nvrx8FXIgP6", "reviewer": "Reviewer_u9nE", "summary": "The multi-modality problem in Non-Autoregressive Machine Translation (NAT) makes non-monotonic alignment an important research direction. This paper explores the non-monotonic matching on latent alignment models. The authors first introduce a Simpliﬁed Connectionist Temporal Classiﬁcation (SCTC) where only blank tokens are removed in the collapsing function. Then they derive bipartite matching and n-gram matching as two non-monotonic alignments under SCTC. Finally, they generalize the n-gram matching method under SCTC to regular CTC. The experimental results show the superiority of non-monotonic training objectives, and the best method achieves comparable performance to AT models.", "questions": "What best method does NMLA stand for specifically? CTC+2-gram?", "limitations": "No majot limitations.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n* The motivation is clearly given and the paper is well-written.\n* The proposed non-monotonic alignment methods may facilitate the development of this field.\n\n**Weaknesses**\n\nThe relationship of this paper to related work is not very clear. What specific problems exist in the previous works are not sufficiently described.\n\nTypo (line 108): the s translation ->  the translation ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "dMpygcwWqpz", "reviewer": "Reviewer_NovV", "summary": "This paper proposed methods to train non-autoregressive NMT models by relaxing the monotonic alignment restriction in the CTC loss. The paper first proposed a simplified CTC (SCTC) which only takes the blank symbol into consideration and does not allow token repetitions. In order to taking non-monotonic alignments into consideration during training, two solutions, i.e., bipartite matching and n-gram matching, are introduced. The n-gram matching for the regular CTC is further explored.\n\nExperiments are conducted on WMT En—De and En—Ro tasks. The results show the proposed bipartite matching and n-gram matching can significantly improve over the SCTC and CTC baseline, where the combination of n-gram and CTC achieves the best performance. Compared with related works, the proposed systems achieves comparable results.\n", "questions": "See the above weaknesses.", "limitations": "NAN", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe motivation of this paper is reasonable and clear as the regular CTC loss would penalize predictions which are correct but have different word orders.\n\nExperiments show the proposed solutions for calculating non-monotonic alignment losses are effective. \n\n\nWeaknesses: \n\nExperimental results are not convincing enough. Compared with the existing Fully-NAT, the proposed NMLA systems achieves comparable results but slower speed. Although the proposed method show improvements over naïve baselines, whether the proposed method could improve Fully-NAT further as an addition is unknown which uses the CTC loss as well during training.\n\nThe paper writing should be improved with more clear descriptions of the proposed methods. For example, Line 109 introduces a forward variable which is not mentioned anywhere else. More detailed introduction on CTC may be needed as a comparison so that readers would know the differences between CTC and SCTC formally if any. In addition, I did not get the role of the bipartite matching in this paper which seemingly neither shows better results nor help the derivation of n-gram matching for CTC.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "_VdgF5ewO06", "reviewer": "Reviewer_bcDR", "summary": "This paper proposes relaxing CTC-based models, which train the model with the objectives that allow non-monotonic alignment between the output and the reference.  Furthermore, the authors propose to finetune a CTC with n-gram matching (n=1, 2). Experiments show that such approaches consistently improve performance, showing the effectiveness of relaxing objectives and finetuning strategy.", "questions": "n/a", "limitations": "n/a", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I think the paper is publishable, which has:\n- The motivation for allowing non-monotonic alignment of the CTC model is clear.\n- There is a systematic study on invoking non-monotonic alignment, and the authors show consistent improvements over receptive baselines.\n- Their best variant achieves strong performance.\n- The proposed SCTC provides a bridge to n-gram matching for the general CTC model, which is novel.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept