Non-Autoregressive Document-Level Machine Translation
Authors: Guangsheng Bao
Zhiyang Teng
Hao Zhou
Jianhao Yan
Yue Zhang
Venue: EMNLP 2023 Findings (2023)
Keywords: NAT
Document-level MT
Machine Translation

Abstract: Non-autoregressive translation (NAT) models achieve comparable performance and superior speed compared to auto-regressive translation (AT) models in the context of sentence-level machine translation (MT). However, their abilities are unexplored in document-level MT, hindering their usage in real scenarios. 
In this paper, we conduct a comprehensive examination of typical NAT models in the context of document-level MT and further propose a simple but effective design of sentence alignment between source and target.
Experiments show that NAT models achieve high acceleration on documents, and sentence alignment significantly enhances their performance. 

However, current NAT models still have a significant performance gap compared to their AT counterparts. Further investigation reveals that NAT models suffer more from the multi-modality and misalignment issues in the context of document-level MT, and current NAT models struggle with exploiting document context and handling discourse phenomena.
We delve into these challenges and provide our code at \url{https://github.com/baoguangsheng/nat-on-doc}.

Non-Autoregressive Document-Level Machine Translation
Guangsheng Baol.:; Zhiyang Teng", Hao Zhou', Jianhao Yan'?, and Yue Zhang 2.5.1 Zhejiang University School of Engineering, Westlake University Nanyang Technological University 4 *sinnguaa University
Institute of Advanced Technology, Westlake Institute for Advanced Study
2 (baoguangsheng, yanjianhao, thangyue>Jewsttee edu. cn
3 zhiyang. tengêntu. edw...t.eedde.... sg zhouhao@air. tsinghua, edu. cn
Abstract
from sentence-level MT, document-level MT con- siders much larger inter-sentential context and models discourse dependence, such as anaphora, ellipsis, and lexical cohesion (Voita et al., 2019) 9). The sentence-by-sentenne translation done by NAT models cannot possibly produce contextual coher- ent translations, hindering the usage of NAT models in real scenarios.
aoneautoreerrssive translation (NAT) models achieve comparable performance and superior speed compared to auto-regressive translation (AT) models in the context of sentence-level machine translation (MT). However, their abili- ties are unexplored in document-level MT, hin- dering their usage in real scenarios. In this paper, we conduct a comprehensive exami- nation of typical NAT models in the context of document-level MT and further propose simple but effective design of sentence align- ment between source and target. Experiments show that NAT models achieve high acceler- ation on documents, and sentence alignment significantly enhances their performance.
Document-level context enhances the translation quality of AT models significantly (Werlen et al., 2018; Maruf al., 2019; Liu al., 2020; Bao al., 2021; Sun et al., 2022; Bao al., 2023). However, no precedent research explores the possibility of ap- plying NAT models to document- level MT, which leaves open research questions including: 1) Can NAT models leverage document context to improve translation quality? 2) Can NAT models gener- translations with cross- moss--sentental cohesion and coherence? and 3) Can NAT models achieve the same performance as AT models? ? To address these questions, we investigate the challenges and oppor- tunities of NAT models on document-level MT.
However, current NAT models still have a sig- nificant performance gap compared to their AT counterparts. Further investigation re- veals that NAT models suffer more from the modaityodaliiy and misalignment issues in the context of document-level MT. and current NAT models struggle with exploiting docu- ment context and handling discourse phenom- ena. We delve into these challenges and provide our code at https://g" comb baoguangsheng/nate taaireennnnttt--dddeeo
NAT models primarily face two challenges: multi-modality (Gu et al., 2018), which causes failure when one source has multiple possible trans- lations, and misalignment (Saharia et al., 2020; Shao and Feng, 2022), which causes repetition and disorder in translations. Previous studies reduce modalities using knowledge distillation (Zhou al., 2019) and improve alignment using new loss (Sa- haria et al., 2020; Shao and Feng. 2022: Huang et al., 2022). These methods have proven effective on the sentence level, while their efficacy is uncer- tain on the document level. First, document-level MT necessitates the modeling of cross-sententia coherence and discourse structure. Second, the extended length of input and output sequences in- tensifies the potential modalities. Last, the align- ment between the source and target becomes harder because the enlarged input and output sequences increase the aligmment space exponentially.
1 Introduction
Von-autoregressive neural machine translation (NAT) models achieve significant acceleration on the inference, with even better translation perfor- mance, when compared to auto-regressive transla- tion (AT) models on sentence-level MT (Gu al., 2018, 2019; Stern et al., 2019; Ma al., 2019; Lee et al., 2020; Huang et al., 2022; Shao and Feng, 2022). However, real applications (e.g., Google Translation and ChatGPT) typically need to under- stand and respond in discourse, which requires a quick process of document-level content. Different
"Contributed equally. *Corresponding author.

---

In this paper, we first assess recent NAT models on document-level MT tasks to understand their abilities and limitations, where we observe model failures and unstable performance. According to these observations, we further introduce sentence alignment to NAT models, which equips the en- coder and decoder of the NAT models with the group-attention (Bao et al., 2021), restricting the target-to-source attention inside each target and source sentence pairs. Since each target sentence is aligned with source sentence, we do not need to consider the possibility of aligning the target sentence with other source sentences. Therefore, the alignment space between the input and output sequences is significantly reduced.
which cannot be determined without condition on the first token. The multi-modality issue causes vanilla NAT models to fail on complex datasets. Previous research leverages knowledge distillation to reduce modalities of a dataset, so that the con- ditional independence assumption is more likely satisfied (Gu et al., 2018). They generate a trans- lation for each training sample using AT model and then train NAT models this generated train- ing set. In this paper, we compare NAT models trained on both raw and knowledge-distilled data.
The misalignment issue elicits various NAT tech- niques. Vanilla NAT (Gu et al., 2018) uses an im- plicit token-alignment approach, enforcing mono- tonic alignment between source and target by copy encoder outputs to decoder inputs. NAT+CTC (Li- bovický and Helel, 2018) introduces connectionist temporal classification (CTC) (Graves et al., 2006) to model the monotonic alignment explicitly. Oth- ers introduce non-monotonic alignment between source and target, such as n-gram matching (Shao et al., 2020; Shao and Feng, 2022), aligned cross- entropy (AXE) (Ghazvininejad et al., 2020), and order-agnostic cross-entropy (OAXE) (Du et al., 2021). In this paper, we investigate NAT models using the first two techniques.
Experiments on three benchmark datasets TED, News, and Europarl show that NAT models achieve a translation speed of more than 30 times faster than their AT counterparts on document-level MT, but still lag behind on the translation performance. Sen- tence alignment significantly enhances the transla- tion performance for NAT models, closing the gap from 2.59 to 1.46 d-BLEU (document-level BLEU score) compared to the best AT results. To the best of our knowledge, we are the first to discuss the challenges and opportunities of NAT models in the context of document-level MT. We release our code and data to stimulate future research.
Document-Level Machine Translation. Previ- ous methods on document-level MT are dominated by auto-regressive models, which can be catego- rized into two approaches. The first approach splits document into sentences and translates each sen- tence using its context as additional inputs (Zhang et al., 2018: 8; Maruf et al., 2019; Zheng et al., 2021). The second approach takes a document as a whole translation unit and translates the document in one beam search (Liu et al., 2020; Bao et al., 2021; Sun et al., 2022; Bao et al., 2023). In this paper, we follow the second approach for our investigation of NAT models.
2 Related Work
Non-Autoregressive Machine Translation. Ex- isting NAT models include three types: fully NAT models (Gu et al., 2018: Libovický and Helcl, 2018; Qian al., 2021; Huang et al., 2022), iterative NAT models (Ghazvininejad et al., 2019; Stern et al., 2019; Gu et al., 2019; Chan et al., 2020), and semi- autoregressive model (Ran et al., 2020). All these models generate tokens in parallel at some level. The fully NAT models generate tokens in single forward pass, while the other two types generate tokens in multiple iterations or steps. In this pa- per, we take the fully NAT models to investigate document-leves MT.
Efficient Model for Long Sequence. Re- cent advances in efficient models such as Long- former (Beltagy et al., 2020), Reformer (Kitaev et al., 2020), Linformer (Wang et al., 2020), and FlashAttention (Dao et al., 2022) improve the time and space complexity of the attention mechanism, which may affect the inference speed of both the auto-regressive and non-autoregressive MT models. In this paper, we focus on the relative inference acceleration of NAT compared AT models with standard multi- attention, leaving advanced attention mechanisms for the future.
NAT models face two fundamental challenges: the multi-modality issue and the misalignment is- suc. The multi multi-modality issue happens when one source has multiple possible translations, breaking the conditional independence assumption. For ex- ample, "thank you " in English can be translated into "Vielen Dank." " or "Danke." in German. The second output token could either be "Dank" or "

---

3 Methods
the length, and denotes the source input. For simplicity, we take the model as whole and use the to denote all the parameters, which includes the parameters of the length model and the translation model.
3.1 AT Baselines for Document-Level MT
Document-level MT can be formulated as a seq2seq problem, where the input document and output document y are represented in token sequence. The auto outo-segressive factorization is
An implicit token alignment is used between the source and target by copying the outputs of the encoder to the inputs of the decoder, where if the source and target have different lengths, they interpolate the decoder inputs uniformly. During training, initializes the decoder with T token po- sitions, where T denotes the real number of target tokens. During inference, it first predicts a number T using pe [T)) and then initializes the decoder with T token positions to predict target tokens. GLAT (Qian et al., 2021) improves vanilla NAT with glancing mechanism, which adopts adap- tive glancing sampling strategy, exposing some tar- get fragments to the decoder inputs so that the de- coder can fit the training target easier. The glancing mechanism reduces the difficulty of independent training of target tokens and improves the perfor- mance on sentence-level MT significantly.
T
po(yla)= - II Po (xmlitee))
(1)
where 7 denotes the number of target tokens and Y/ conditions on all previous tokens V<(-
We choose two AT baselines.
Transformer (Vaswani et al., 2017) the stan- dard encoder-decoder model, which is widely used as a baseline. We represent each document as a sequence of tokens and train the model to do the seq2seq mapping.
G-Transformer (Bao et al., 2021) extends Trans- former by separating the self-attention and cross- attention into sentence-level group-attention and document-level global-attention, Figure illus- trates. The group-attentin enables sentence align- ment between input and output sequences.
Latent-GLAT (Bao et al., 2022) further im- proves GLAT using latent variables, where the la- tent variables are supposed alleviate the multi- modality problem. It represents the modality and alignment the latent variables, dividing the learn- ing of the target tokens into the modeling of the latent variables and the reconstruction the target sequence.
Knowledge Distillation (KD). In addition to the raw data, we also experiment with KD data for training NAT models (Zhou et al., 2019), following previous NAT studies. We use G- G-Transformer fine- tuned sentence Transformer as the teacher to generate distilled document translations.
3.2 Existing NAT Models
NAT+CTC (Libovický and Helcl, 2018) ap- plies connectionist temporal classification (CTC) (Graves et al., 2006) alignment to decoder outputs
Due to the large amount of NAT models (a non- exhaustive search shows more than 60 recent NAT models proposed between 2020 and 2023), we follow the common practice (Huang et al., 2022: Qian et al., 2021) to choose representative NAT models for our investigation, leaving more com- plex iterative methods, autreervorerrssivvee meth- ods, and pre-training settings for future. Specifi- cally, select representative fully NAT models for document-leved experiments, where all the models are transformer-bast based and implemented in Fairseq (Ott et al., 2019).
M
po(y| = Σ
II
po(a))x),
(3)
(EB(V) 1=1
where denotes alignment between y and the reserved M token positions in the decoder while B(y) denotes all possible alignments. The align- ment a is assumed conditional independent given the source input x. The reserved token positions are usually set as times of the length of the source input.
Vanilla NAT (Gu et al., 2018) is the earliest NAT model, which factorizes translation into two parts
CTC alignment in Eq. 3 requires summation over all possible alignments, which is generally in- tractable. However, with conditional independence assumption for a, the summation can be achieved using dynamic programming.
po(y|=) pe (T) II po(v)))),
where the first part predicts the length T of target y. the second part predicts target tokens y given
GLAT+CTC (Qian et al., 2021) combines GLAT with CTC alignment.

---

Sentence1
Sentence2
Target
SentenceI
Sentence2
Target
Decoder
Decoder
Group-Attention
Global-Attention
Group-Attention
Global-Atteation
Encoder
Encoder
Sentence1
Sentence2
Source
Sentence1
Sentence2
Source
G-Transformer with sentence alignment be- tween source and target documents, where the low lay- ers the group-attention and only the top 2 layers use the global-attention.
Figure I:
Figure 2: NAT model with sentence alignment, where we replace NAT encoder and decoder layers with G- Transformer layers and we remove the causal mask from G- Transformer decoder layers.
DA-Transformer (Huang al., 2022) repre- sents the output alignment a directed acyclic graph (DAG), which contains vertices and edges. A path in DAG represents a possible alignment. models translation as
adopt in our NAT models to reduce the alignment space.
Specifically, we adopt the G-Transformer en- coder and decoder layers by removing the causal mask from self-attention in the decoder layers. We replace the encoder and decoder layers in NAT models with G-Transformer layers and redesign the initial output to include the special token of begin- of-sentence "
" and end-of-sentence "
" for each target sentence.
po(1)z)
Σ p) aE8(y)
(4)
where a denotes path represented in sequence of vertex indexes and B(y) here denotes all paths having the same length as the target y. The right expression in Eq. 4 can further be expanded as
Substituting the causal masking layer in G- Transformer is only necessary part at the attention level to facilitate sentence alignment. Its effective- ness hinges on both model design and training loss. Specifically, we introduce new length prediction per sentence in G-Trans+GLAT, improve CTC loss for sentence alignment in G-Trans+GLAT+CTCC and improve DAG design to restrict the transition in each sentence in G- G-Trans+DAT as follows.
M-1
pa(a|x)=
II
po po(4:1la;; i=1
(5)
M
(o(yla, a, x)) = (o (ulla,,,, x),
I
(6)
G- TeTTTranGGLAA predicts the target length per sentence instead of the whole sequence. It factor- izes the translation as
where are dependent on each other in a linear chain and yı are independent with each other given 0; and z.
Similar to CTC alignment, DA-Transformer also adopts dynamic programming to marginalize all possible paths. The model provides three decoding strategies: greedy, lookahead, and beam search. In this paper, we choose lookahead decoding for balanced performance and inference speed.
K (o())>) II
T3 po(T) II no (vs. 1-1
where K denotes the number of sentences. We predict the length T₁ of the j-th target sentence and generate tokens Uy.t accordingly. In addition to Eq. 2, where the corresponding source sentence of Y/ is unknown, Vj. also conditions on the source sentence Z-j-
3.3
NAT Models with Sentence Alignment
We propose novel NAT models with sentence align- ment, as shown in Figure 2. The sentence align- ment is key feature in the G-Transformer (Bao al., 2021), which is implemented through group-attention module. The group-attention stabi- lizes self-attention and cross-attention the model over long sequences. Inspired by the success, we
For each source sentence Iy, predict the length T, of the target sentence using linear clas- sifier based on the mean pool of output features of the source sentence tokens. Consequently, we calculate the length the target sequence by aggregating sentence lengths T E/ Tj.

---

TED
News
Europari ())B
Average (d-BLEU)
Method
(d-Bl (L-BLEE)
(d- (d-BLEU)
Speed
Raw
KD
Raw
KD
Raw
KD
Raw
KD
-u
AT Baselines
G-Transformer (Bao et al., 2021)
27.23
26.42
27.22
26.38
34,09
32.87
29.51
28.56
Transformer (Vaswani et al., 2017)
0.69
26.04
0.23
0.43
33.41
26.43
11.4
17.63
1.0x
Existing NAT Models
Vanilla NAT (Gu et al., 2018)
0.45
0.34
0.12
0.61
1.30
2.43
0.62
1.13
41.2x
GLAT (Qian et al., 2021)
1.77
0.03
0.01
2.56
2.13
8.17
1.30
3.59
40.0x
Latent-GLAT (Bao et al., 2022)
2.10
1.87
0.30
2.30
5.26
4.25
2.55
2.81
29 2x
NAT+CTC (Libovický and Helll, 2018)
21.34
24.98
13,95
24.03
0.00
31.58
12.50
26.86
27.7x
GLAT+CTC (Qian al., 2021)
18.49
25.31
10.23
20.01
0.00
29.47
9.57
24.93
26.3x
DA Transformer (Huang al., 2022)
20.47
25.02
13,99
23.37
30.34
32.14
21.60
26.84
30.8x
NAT Models with Sentence
Alignment
G- G-Trans+GLAA (ours)
19.96*
24.23"
15.14*
23.04
25.67*
31.45*
20,26
26.24
30.0x
Crans+GLAT+CCTE (ours)
24.09*
26.31
21.68*
25.61
30.35*
32.24*
25.37
28.05
20.0x
G-Trans+DA DA--rans (ours)
23.45*
25.73"
21.43*
24.70*
30.36
32.29
25.08
27.57
25.1x
Table 1: Main results
data (Raw) and knowledge distilled data (KD), where the failures marked and investigated in section 5.1. We use the official code from the respective papers of NAT models, except NAT+CTC, which is implemented by ourselves based on the GLAT code. "*" denotes a statistical significance the level of p < 0.01 using t-test, compared to the corresponding baseline NAT model without sentence alignment.
G-Trans+GLAT+CTC integrates the sentence alignment with the CTC alignment. The default CTC algorithm aggregates all possible latent align- ments across the entire sequence, which may align a source sentence to wrong target sentence (e.g., the first source sentence to the second target sen- tence). Such global alignment not only slows down the training process but also causes unstable model performance. Different from the global alignment Eq. 3. we apply CTC alignment on each sentence
Formally, we use the same factorization as Eq. 4 but with a different collection (3y) of paths. At the implementation level, we simply mask the transi- tion matrix to disable transitions from one sentence to another sentence, so that the dynamic program- ming algorithm keeps unchanged.
4
Experiments
4.1 Experimental Settings
We evaluate the models using document-level MT benchmark (Maruf et al., 2019), which includes three datasets TED, News, and Europarl, repre- senting three domains and various data scales for English- German translation. More details about each dataset and the preprocessing are in Appendix A. We follow Liu et al. (2020), evaluating the model performance in sentence-level BLEU score (s- (s-BLEU) and document-level BLEU score (d- BLEU), which are explained in detail as Appendix A.2. We experiment on Base model (where Big model does not provide a stronger baseline) and evaluate the speedup using 1 GPU and 4 CPUs of Tesla V100 environment as Appendix A.3.
M; pa po(osalaz.a)) [
II
Σ
II
(8)
where M, denotes the reserved token positions for the j-th target sentence. Since CTC alignment is restricted inside each target sentence, the alignment space B(y;) is enormously reduced compared to the B(y) in Eq. 3. G-Trans+DA-Trann
introduces the sentence alignment into the output DAG of the DA- Transformer. The default DAG models the whole sequence, which enables the transition from a ver- tex in one sentence making the transition space linearly increase with the enlargement of the sequence length.
vertex in another sentence,
4.2 Overall Results
As shown in Table 1, NAT models achieve high speedup but still suffer from significant perfor- mance gap with their AT counterparts.
To address the issue, we enforce a constraint to isolate the vertex transitions of cach sentence, forcing the path on each sentence starting with special vertex "
" and ending with special ver- tex "
" The transition between sentences only happens from the vertex "
" previous sen- tence to the vertex "
" the current sentence.
Speedup. The inference accelerations of NAT models on the document level are between 25x and 41x, which surpasses the accelerations on the sen- tence level (between 2x to 15x) by a big margin. Specifically, NAT and GLAT provide the biggest

---

acceleration by around 40x. More complex mod- cls, such as Latent-GLAT, DA-Transformer, and GLTT, accelerate inference by approxi- mately 30x. CTC-based models have lower acceler- ations between 20x and 27x. On average, the accel- eration on the document level is about 30x, which means that document-level MT systems could po- tentially save 96% computational resources and energy consumption by using NAT models.
raw data and 0.73 points on KD data. These re- sults suggest that sentence alignment mitigates the challenge brought by more modalities in raw data.
Token-level alignment. NAT models with im- plicit alignment such as vanilla NAT, GLAT, and Latent-GLAT fail on document-level MT, resulting in messy translations (repetitions, wrong words, and meaningless phrases). Conversely, NAT mod- els with explicit aligmments, such as NAT+CTC, +LAT+CTC, and DA- PA-Transfommer, produce supe- rior document-level performance. DA-Transformer delivers the best overall performance among exist- ing NAT models, even outperforming Transformer on Europarl (KD). These results suggest that token- level alignment plays an important role in NAT models to achieve good performance.
Performance. Though NAT models under- perform G- G-Transformer, some outpace the Trans- former. The lengthy text sequence challenges both the AT models and the NAT models, resulting in exceptionally low d-BLEU scores (<10) in some settings. We treat these low scores as model fail- ures and investigate them in section 5.1.
With the help of sentence alignment, the per- formance gap between NAT models and AT base- lines is largely reduced, especially when trained on KD data. For example, G- G-Trans+GLAT+CT achieves an average d-BLEU 28.05 on KD data, which is only 0.51 points lower than the d-BLEU of 28.56 of G-Transformer. However, its perfor- mance on Raw data is 4.14 points lower than G- Transformer, suggesting that NAT models experi- ence severe challenges with raw data. These results demonstrate that even though knowledge distilla- tion and sentence alignment enhance NAT models largely, there is still a gap between the NAT models and the strongest AT baseline.
However, we also observe that training with CTC loss on documents is not always stable, leading to occasional failure and a high variance in their per- formance (e.g., +/-2.66 d-BLEU on raw data and +/-0.84 KD data for GLAT+CTC). The failure happens more frequently data. We speculate that this instability is caused by the increased alignment space on long sequences, which can be mitigated by sentence alignment. Ex- periments on G- G-Trans+GLAT+CCC show stable training and lower variance in performance (+/- 0.19 d-BLEU). These results suggest that token- level alignment alone is not enough to achieve sta- ble training and good performance.
data than KD
Knowledge distillation. Intuitively, documents have more modalities because they are much longer than sentences. If it is the case, knowledge distil- lation will become more critical for training NAT models on document-level MT than on sentence- level MT. We compare NAT models trained with and without knowledge distillation.
4.3 Breakdown Results
Sentence-level alignment. Sentence alignment substantially enhances the performance of NAT models. Specifically, for GLAT, sentence align- ment enhances the performance on KD data from an average of 3.59 to 26.24 d-BLEU of G- Trans+GLAT. For GLAT+CTC, sentence alignment elevates the performance on KD data from 24.93 to 28.05 d-BL.UU on average. This score is compara- ble to G- G-Transformer KD data, leaving minor gap of 0.51 d-BLEU average. The better results of G-Trans+GLAT+CTC than G-Trans+GLAT sug- gest that sentence alignment complements output alignment techniques such as CTC.
As Table shows, NAT models trained on KD data generally outperform the same model trained on raw data. The improvement is especially signifi- cant for NAT models with explicit alignment. For example, DA-Transformer obtains an average of 26.84 d-BLEU KD data, which is 5.24 points higher than 21.60 d- d-BLEU -A--raasformer on sentence-level MT achieves similar results on raw data and on KD data (Huang et al., 2022). The results suggest that compared to sentences, documents have more severe multi- modality issues.
data. In contrast,
The sentence alignment brings more benefits models trained on raw data. As the average scores in Table show, G-Trans+GLAT+CIC out- performs GLAT+CTC by 15.80 points on raw data and 3.12 points on KD data. G-Trans+DA- Araans outperforms DA-Transe Transfoffrrerr by 3.48 points on
Although knowledge distillation enhances the performance of NAT models, it also sets a ceiling

---

GLAT GLATlex
GLAT-CTE GLAT-CTI/e
Speedup
40
Speedup
100
30
50
10
0
sent
64
128
256
512
1
Segment Length
Batch Size
Figure 3: Speedup rises as long the sequence length grows, evaluated on TED and News.
Figure 4: Speedup with various batch sizes evaluated on News, where the "/ex" conducts a strict calculation of speedup by excluding the time cost model initial- ization from both the AT baseline and the NAT models.
to the performance on document-level MT. Com- paring the performance of G- G-Transformer on KD data to raw data, we see that KD data downgrades the performance by about 1 d-BLEU average (from 29.51 to 28.56). These results differ from pre- vious results on sentence-level MT (Huang al., 2022), where knowledge distillation maintains or even enhances the performance of AT models. The discrepancy calls for further research on techniques to reduce the modalities on the document level.
the inference speedup of GLAT is 125x and 44x for the batch size of and 8, respectively. These results suggest that although the speedup ratio decreases for bigger batch sizes, NAT models on document- level MT still show significant acceleration in the parallel inference settings.
5 The Challenge of Long Sequence
The long input and output in document-level MT bring unexplored challenges to NAT models. In sec- tion 4, we learn that NAT models have a significant performance gap with their AT counterparts, and various NAT models even fail in some settings. We investigate these challenges in this section, leaving other discussions in Appendix D.
Speedup on different sequence lengths. We evaluate GLAT and GLAT+CTC on various se- quence lengths, including the single sentence and segments with maximum length of 64, 128, 256, and 512. As Figure 3 shows, the speedup dis- plays consistent trends among different datasets and NAT models. The GLAT generally has higher speedup than GLAT-CTC, especially when the seg- ment length goes above 256. The speedups on TED and News are almost identical, which is expected because the time costs are supposed to be irrele- vant to the data domains. The trends suggest that we benefit more from the inference acceleration longer documents, where document-level MT tasks provide NAT models with the best scenario.
5.1 The Failure of NAT Models
Previous study (Bao al., 2021) suggests that Transformer fails on long sequences because of local minima of loss values during the training process. We first investigate this possibility for the cause of the failure on NAT models. We evaluate GLAT, GLAT+CTC, Transformer, and G- Transformer on different lengths of sequences, shown in Figure 5a. Overall, the four models pro- duce different patterns of trends. The d-BLEU score of the Transformer rises long as the se- quence length increases until the failure happens the length of 512 tokens. In contrast, the d-BLEU scores of the GLAT and GLAT+CTC descend pro- gressively as long as the sequence length increases, which suggests a performance decline instead of training failure failure on NAT models.
Speedup different batch sizes. The previous study (Helel et al., 2022) reports that NAT mod- els have limited acceleration on the sentence-level MT in parallel inference settings. We evaluate the models in the settings with different batch sizes, including 1, 2, 4, and 8 instances per batch. Given that document-level MT sequences are much longer than sentence-level MT sequences, do not eval- uate using batch size larger than 8.
As Figure 4 shows, when we increase the batch size from to 8, the overall speedup of GLAT de- creases from 40x to 9x. However, we consider more strict calculation of the speedup, excluding the time for initializing the models (which takes al- most constant time) from the total evaluation time,
Further evidence the bigger dataset Europarl confirms that the low performance of NAT models is different from the failure of Transformer. Bao et al. (2021) suggests that the local minima encoun- tered by Transformer can be escaped by using a big- ger dataset, resulting in normal scores on Europarl.

---

40
G-trensoome
SLATCTC
BLE3-1 BLEW-2
BLEAS BLEU4
80
BLEU4 BLEU22
BLEUS 0LEU4
Tranafsrmer SLAT
d-BLEU
30
d-BLEU
d-BLEU
20
40
40
10
128 Sequence Length
sent
256
Sequence Length
Sequence Length
(a) NAT and AT performance on different sequence length.
(b) GLAT detailed BLEU scores on different sequence lengths.
(c) GLAT+CTC detailed BLEU scores on different sequence lengths.
Figure 5: NAT vs. AT models on trends evaluated on News, where Transformer fails abruptly at the length of 512 while NAT models decrease progressively.
We evaluate Transformer, GLAT, and GLAT+CTC Europarl, obtaining d-BLEU scores of 33.41, 2.13, and 0.00, respectively. We could see that GLAT and GLAT+CTC still give low scores on the bigger dataset, suggesting a different cause from the local minima.
Source: companies with high percentage of flnating rate debr stand to lose the most, Goldman said. outside pure stock plays, consumers stand to benefit as well through the rising dollar.
Target: Unternehmen mit einem hohen Anteil an flexi- blen Zinsen werden am meisten verlieren, sagte Goldman. außerhalb der reinen Aktienspiele werden Verbraucher ebenfalls durch den steigenden Dollar profitieren.
We look into the d-BLEU score details, where BLEU-n (n E 2, 4)) denotes the precision n-gram. As the d-BLEU scores GLAT in Figure 5b shows, the BLEU-3/4 decreases rapidly, reaching almost 0.0 the lengths of 256 and 512. The BLEU-1 remains relatively high 27.3 on the length of 256, resulting in messy translations, where the generated tokens are related to the con- tent but are repeated, disordered, or the wrong collocation as shown in Table 2. In comparison, the d-BLEU scores on GLAT+CTC in Figure 5c decrease slowly, where the BLEU-1 even increases the length of 512. We speculate the rapid de- crease in BLEU-3/4 on GLAT is caused by the multi multi--oddlity and misalignment issues, which can be mitigated by explicit alignments, such as CTC.
GLAT: Unternehmen mit einem hohen, freien freien freien erverlieren die meisten meisten krästen profitieren profitieren profitieren profitieren profi- tieren durch durch steigenden steigenden profitieren profi- tieren.
te te dazu
GLAT+CTC: Unternehuenn mit hohen der Schulden, die verlieren außerhalb spielt Verbraucher den profitieren.
Table 2:
GLAT and GLAT+CTC produce good transla- tion at the beginning but downgrade later with repeti- tions and missing translations.
discussion in Section 4.3.
5.2 Document Context
We compare AT and NAT models with document context to the Transformer baseline without docu- ment context (trained on sentence) in Appendix B. The results suggest that document context enhances AT model (G-Transformer) by 0.70 S-BLEU. How- ever, NAT models with document context still un- derperform the Transformer baseline, indicating that current NAT models have a limited ability utilize document context.
The exceptional zero scores. Table 1 reveals some unexpected results. Both NAT+CTC and GLAT+CTC score 0.00 d-BLEU on the raw Europarl dataset, performance notably inferior that of NAT and GLAT, This unprecedented anomaly may stem from the challenges of applying CTC loss on long sequences. The table further in- dicates that both combinations, NAT/GLAT+CTC and NAT/GLAT, experience training failures on the raw Europarl dataset. These failures are likely due to multi modai-yydality and misalignment issues. These results empirically demonstrate that, while CTC loss can be effective, its application document- level training is not consistently stable. We hypoth- esize that this instability arises from the expanded alignment space between lengthy input and output sequences, as detailed in the teven- alignment
We further apply an ablation study on the document context to quantify its contribution. As Table 3 illustrates, the performance of G- Trans+GLAT+CTC without document context drops by approximately 0.28 s-BLEU on average over the three benchmarks. Specifically, the target- side context contributes merely 0.01, while the source-side context contributes 0.27. The context contributions G- TTrans+GLA++CTC are less than that in G- G-Transformer (0.23 and 0.70 B-LEEU on

---

Method (s-BLEU)
TED
News
Europari
Drop
Method
deixis
el.infi.
e..V
lexcoh
G-Transformer O
25.12
25.52
32.39
Transformer (sent)
50.0
28.4
45.9
target-side context
25.05
25.41
32.16
-0.14
G-Transformer Q
87.1
82.4
79.8
58.6
source-side context
24.56
24.58
31.39
-0.70
G- G-Trans+GAT+CTC
50.0
55.2
46.6
45.9
G- G-IransGLAT+CC
24.16
24.09
30.57
G- T-TrrmsDAA DA-Trans
56.7
33.0
21.0
45.2
target-side context
24.31
24.00
30.47
-0.01
source-side context
23.96
24.00
30.02
-0.27
Table 4:
Discourse phenomena. el.infl. ellipsis of in- flection. el. ellipsis of verb phrase. lexcoh lexical cohesion. 2 Transformer baseline trained on sentences. G- Transformer baseline trained on documents.
Table 3: Impact of document context, evaluated in
BLEU. Q the scores are from the paper report, where the model is trained on the raw datasets. The NAT models trained on the KD datasets.
cause its DAG link models the target-side depen- dence somehow. These results suggest that current NAT models have some discourse abilities but still struggle with handling discourse phenomena.
target and source contexts, respectively). The mi- nor contribution from the target-side context is ex- pected, given that NAT models predict target sen- tences independently. The relatively low contri- bution of source-side context indicates that NAT models do not fully exploit the source-side contex- tual information.
Conclusion and Future Work
We investigated NAT models document-level MT, revealing more severe affections of multi- modality and misalignment issues on documents than on sentences. We proposed NAT models with sentence alignment, reducing the possible align- ment space, and achieving the best results across three benchmarks. Our experiments show that NAT models significantly accelerate text generation in documents while their performance still lags be- hind their AT counterparts. Further analysis shows that fully NAT models underutilize document con- text, leading to loose discourse relations.
Leak of source context information on adja- cent sentences? We observe serious repetitions in the translations generated by NAT models without sentence alignment, raising the concern that the information in a source sentence may leak into its adjacent sentences during translation. We measure the repetitions on NAT models with sentence align- ment, where the sentence boundaries are assured by the model design. We find that these models do not generate obvious cross cross- sentential repetitions. For example, on the TED test set, G-Trans+GLAT generates translations with repetition ratios of 0. and 0.02 for and grams, respectively, which are almost identical to the ratios of the reference.
As the first research of NAT models on document-level MT, we hope this work could stim- ulate future research reducing the modalities, exploiting document contexts, and modeling dis- course dependence.
5.3 Discourse Phenomena
Limitations
We assess the discourse ability of NAT models using a human-annotated test suite English- Russian MT (Voita et al., 2019). We train both the AT baselines and NAT models using the 1.5M document (only 4 sentences) pairs. In contrast to previous work (Bao et al., 2021), we do not use the additional 6M sentence pairs for training for the purpose of highlighting their discourse capabilities. As Table 4 shows, the NAT models signifi- cantly underperform G- G-Tansffommrr across the four discourse phenomena. The performance of G- Trans+GLA++CTC matches the Transformer base- line (without document context) on deixis and lexical cohesion (lexcoh), but excels on the el- lipsis of inflection (el.infl.) and ellipsis of verb phrase (P). G-Trans+DA- Trann achieve rela- tively higher deixis than G- Trans-GLAT+CCC be-
We do not enumerate all recent NAT models. For the purpose of our investigation, only evaluate the pure (fully) NAT models, leaving other NAT models, such as semi-autoregressivve and iterative NAT models, out our scope.
Acknowledgements
We would like to thank the anonymous review- ers for their valuable feedback. This work is funded by the China Strategic Scientific and Tech- nological Innovation Cooperation Project (grant No. 2022YFE0204900) and the National Nat- ural Science Foundation of China (grant NSFC No. 62161 160339). Zhiyang Teng is partially sup- ported by CAAI-Huawei MindSpore Open Fund

---

References
Jiatao Gu, James Bradbury, Caiming Xiong, Victor OK Li, and Richard Socher. 2018. Non-autoreeressive neural machine translation. In International Confer- ence on Learning Representations (ICLR).
Guangsheng Bao, Zhiyang Teng. and Yue Zhang. 2023. Target-side augmentation for document-level ma- chine translation. In Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 10725- 10742.
Jiatao Gu, Changhan Wang. and Junbo Zhao. 2019. Lev- enshtein transformer. Advances in Neural Informa- lion Processing Systems, 32.
Guangsheng Bao, Yue Zhang, Zhiyang Teng, Boxing Chen, and Weihua Luo. 2021. G-transformer for seeuuennt-eevel machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3442-3455.
Jindfich Helel, Barry Haddow, and Alexandra Birch. 2022. Non- aon-autoreeresssvve machine translation: It's not as fast as it seems. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1780-1790.
Fei Huang. Hao Zhou, Yang Liu, Hang Li, and Minlie Huang. 2022. Directed acyclic transformer for non- autoregressive machine translation. In International Conference on Machine Learning, pages 9410-9428. PMLR.
Yu Bao, Hao Zhou, Shujian Huang, Donggi Wang. Li- hua Qian, Xinyu Dai, Jiajun Chen, and Lei Li. 2022. gatent-glat: Glancing at latent variables for parallel text generation. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers). pages 8398 8409.
Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. arXiv preprint arXiv:2001.04451.
Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The -ocumeettnnnt transformer. arXiv preprint 25X55:200.051150
Philipp Koehn, Hicu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In Pro- ceedings of the 45th annual meeting of the associa- tion for computational linguistics companion volume proceedings of the demo and poster sessions, pages 177-180.
William Chan, Chitwan Saharia, Geoffrey Hinton, Mo- hammad Norouzi, and Navdeep Jaitly. 2020. Imputer: Sequence modelling via imputation and dynamic pro- gramming. In International Conference on Machine Learning, pages 1403-1413. PMLR.
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. 2022. Flashattention: Fast and memory-efficient exact attention with io-awarensss. Advances in Neural Information Processing Systems, 35:16344-16359.
Jason Lee, Elman Mansimov, and Kyunghyun Cho. 2020. Deterministic non-autoregressive neural se- quence modeling by iterative refinement. 2018 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2018, pages 1173-1182. Association for Computational Linguistics.
Cunxiao Du, Zhaopeng Tu, and Jing Jiang. 2021. Order- agnostic cross entropy for non-autoregressive ma- chine translation. In International Conference on Machine Learning, pages 2849-2859 PMLR.
Jindfich Libovický and Jindfich Helel. 2018. End-to- end non aonreestreee neural machine translation with connectionist temporal classification. In Pro- ceedings of the 2018 Conference on Empirical Meth- ods in Natural Language Processing, pages 3016- 3021.
Marjan Ghazvininejad, Vladimir Karpukhin, Luke Zettlemoyer, and Omer Levy. 2020. Aligned cross entropy for non-autoregressive machine translation. In International Conference on Machine Learning, pages 3515-3523, PMLR.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel de- coding of conditional masked language models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLL-IJCNILP), pages 61 12-6121.
Xuezhe Ma. Chunting Zhou, Xian Li, Graham Neu- big, and Eduard Hovy 2019. Flowseq: Non- autoregressive conditional sequence generation with generative flow. Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4282-4292.
Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber. 2006. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In Proceedings of the 23d international conference on Machine learning, pages 369-376.

---

Sameen Maruf, André FT Martins, and Gholamreza Haf- fari. 2019. Selective attention for context-an aware neu- ral machine translation. In Proceedings of NAACI- HLT, pages 3092-3102.
Elena Voita, Rico Semnrich, and Ivan Titov, 2019. When good translation wrong in context: Context-aware machine translation improves deixis, ellipsis, and lexical cohesion. Proceedings of the 57th Annual Meeting of the Association for Computational Lin- guistics, pages 1198-1212.
Myle Out, Sergey Edunav, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. fairseq: fast, extensible toolkit for se- quence modeling. arXiv preprint arXiv: 1904, 01038.
Sinong Wang. Belinda 7.1., Madian Khahsa, Han Fang, and Hao Ma. 2020. Linformer: Self-attention with linear complexity. arXiv preprint arXiv:2006.044 04768.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguistics, pages 311-318.
Lesly Miculicich Werlen, Dhamanjay Ram, Nikolaos Pappas, and James Henderson. 2018. Document- level neural machine translation with hierarchical attention networks. In Proceedings of the 2018 Con- ference on Empirical Methods in Natural Language Processing, pages 2947-2929 29544.
Lihua Qian, Hao Zhou, Yu Bao, Mingxuan Wang, Lin
Qiu, Weinan Zhang, Yong Yu, and Lei Li. 2021. Glancing transformer for non-autoonn autorerressvee neural machine translation.
Jiacheng Zhang. Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. 2018. Improving the transformer translation model with document-level context. Proceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 533-542.
Proceedings of the 59th An- nual Meeting of the Association for Computational Linguistics and the 11th International Joint Confer- ence on Natural Language Processing (Volume Long Papers), pages 1993-200 2003.
Qiu Ran, Yankai Lin, Peng Li, Jie Zhou. 2020. Learning to recover from multi-modality errors for aoorerrtreee neural machine translation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 3059- 3069.
Zaixiang Zheng, Xiang Yue, Shujian Huang, Jiajun Chen, and Alexandra Birch. 2021. Towards making the most of context in neural machine translation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3983-3989.
Chitwan Saharia, William Chan, Saurabh Saxena, and Mohammad Norouzi. 2020. Non-autoregressive ma- chine translation with latent alignments. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pages 1098 1108.
Chunting Zhou, Graham Neubig, and Jiatao Gu. 2019. Understanding knowledge distillation in autoregressive machine translation. arXiv preprint arXiv:1911.02727.
Chenze Shao and Yang Feng. 2022. Non-monotonic la- tent alignments for ete-based -on-autoregressive ma- chine translation. arXiv preprint arXiv:2210.03953.
Chenze Shao, Jinchao Zhang, Yang Feng. Fandong Meng, and Jie Zhou. 2020. Minimizing the bag- of-ngrams difference for non- autoregressive neural machine translation. In Proceedings of the AAAI con- ference on artificial intelligence, volume 34, pages 198-205.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. 2019. Insertion transformer: Flexible se- quence generation via insertion operations. In In- ternational Conference on Machine Learning, pages $976-5985 PMLR.
Zewei Sun, Mingxuan Wang, Hao Zhou, Chengqi Zhao, Shujian Huang, Jiajun Chen, and Lei Li. 2022. Re- thinking document-leved neural machine translation. Findings of the Association for Computational Linguistics: ACL 2022, pages 3537-3548.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Fukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural processsnn processing systems, 30.

---

Sentences
Documents
Segments
Sents per Segment
Tokens per Segment
Dataset
train dev / test
train dev / test
train dev
train dev test
train dev
TED
0.21M/ 9K 2.3K
1.7K /2///2//2 22
11K 483 123
18.37 13.718.5715 18.3
436 428 429 380 / 355
News Furoparl
0.24M/ 2K
6K / 80/ 154
18.5K 172 263
12.8/ 12.6 11.3
1.67M/ 3.6K 5.1K
118K / 239 359
162K 346 1446/49 4998
10.3/ 10.4/ 10.3
320/ 323
Table 5: English- Mnglish-German datasets for evaluation, where split each document into segments with maximum of
512 tokens.
TED
News
Europari
Average
Method (s-BLEU)
Raw
KD
Raw
KD
Raw
KD
Raw
KD
AT Baseline
Transformer (sent baseline)
24.63
24.97
31.34
26.98
G- G-Transformer (doc baseline)
25.12
25.52
32.39
27.68
NAT Models
G- G-Trans+GLAAT G-Trans+GAAT-CH
17.05
21.81
13.37
21.16
23.49
29.66
17.97
24.21
21.87
24.16
20.18
24.09
28.80
30.57
23.62
26.27
G-Trans+DA- -rans
21.02
23.48
19.79
23.10
28.64
30.55
23.15
25.71
Table 6: Sentence-level performance with and without document context.
A Experimental Settings
words.
A.1 Datasets
A.3 Model Configuration
We evaluate the models using document-level MT benchmark (Maruf al., 2019), which includes three datasets covering three domains and various data scales for English-German translation.
All the experiments are run on the Base model, which has layers, heads, 512 embedding di- mensions, and 2048 hidden dimensions. We train the models on 4 Tesla V100/A 100 GPUs for both AT and NAT. By default, we use the Tesla V100, but in case out-of-memory happens, we switch Tesla A 100 re-train the model. We do not change the code of existing NAT models, and we obtain the default training and testing arguments from their official code. We update the arguments max- source-positionn and tettaarr-tsssttion to fit the enlarged input and output sequences. We run all main experiments three times and report the me- dian.
TED is from IWSLT17, which is transcribed from TED talks that each talk forms a document. tst2016-2017 is used to test the model, and the rest for development.
News is from News Commentary v11 for train- ing set. For testing and development sets, it uses newstest2016 and newstesr2015, respectively. Europarl from Europarl v7, where the train- ing, development, and testing sets are randomly split.
Table 5 shows the detailed statistics. We prepro- cess the documents by tokenizing and truecasing using MOSES (Koehn et al., 2007) and applying BPE with 30,000 merge operations. We follow Bao al. (2021) split each document into segments with maximum length of 512 tokens.
We assess the speedup on the test set using a batch size of within virtual environment equipped with GPU and 4 CPUs a Tesla V100.
B Contribution Document Context
Previous studies demonstrate that document context can significantly enhance MT perfor- mance (Zhang et al., 2018; Zheng et al., 2021; Bao et al., 2021). We evaluate its contribution as shown in Table 6. Comparing Transformer on sentence- level MT and G-Transformer on document-level that document context enhances B-BLEU by 0.49, 0.55, and 1.05 TED, News, and Europarl, respectively. However, the best NAT results produced by G- G-Trans+GLAT+CCCC on KD data are still lower than the Transformer baseline by 0.47, 0.88, and 0.77 on TED, News, and Europarl,
A.2 Evaluation Metries
We follow Liu et al. (2020), evaluating the model performance in s-BLEU and d-BLEU.
s- 5-BLEU is calculated on each pair of sentences, which are obtained using the sentence alignment between source and target documents.
MT.
d-BLEU is calculated on each pair of segments, taking the whole segment as translation unit to compute the BLEU score.
We calculate the BLEU score (Papineni et al., 2002) using ELU on the detokenized cased

---

Source: companies with high percentage floating rate debt stand the most, Goldman said, outside pure stock plays, consumers stand benefit as well through the rising dollar. savers could see gains well through higher yields at the, though experts differ on how quickly that will take hold.
Target:
Unternehmen mit einem hohen Anteil an flexiblen Zinsen werden meisten verlieren, sagte Goldman. außerhalb der reinen Aktienspiele werden Verbraucher ebenfalls durch den steigenden Dollar profitieren. Sparer könnten Gewinne durch höhere Erträge schen, auch wenn Experten unterschiedllcher Mcinung sind, wie schnell das stattfinden wird.
te te dazu kräften profitieren profitieren profitieren profitieren profitieren durch durch steigenden steigenden proûteren profitieren. die könnten könnten Gewinne Erträge höhere höhere erziclen erzielen erzielen, obwohl sich sich sich sich, schnell schnell diese diese erzielen erzielen erzielen, sich sich sich sich
GLAT: Unternehmen mit einem hohen, freien freien freien erverlieren die die meisten meisten..
FuB wird wird. werden werden sich sich sich sich die die dic.. halten nehmen...
der die die die
Trans+GLAT:: Unternehmen einem hohen Prozentsatz freier Zinsen werden am am meisten verlieren, sagte Goldman Goldman außerhalh reinreinen Aktien würden die Verbraucher chenso durch durch den steigenden Dollar profiticren, die Sparer könnten durch höhere Erträge Erträge Gewinne erzielen, auch sich Experten darüber unterscheiden, wie schnell das das einnehmen wird.
GLAT+CTC: Unternehmen mit hohen der Schulden, die verlieren. autirrall spielt Verhraucher den profiticren. könnten Gewinne höhere Renditen, sich Experten sich, schnell sich. Verbraucher Aktien werden wird Aktien
TTrans+GLAT+CTTE Unternchmen mit einem hohen Prozentsatz freier Zinsen würden am meisten verlieren, , sagte Goldman reine Aktienspielen würden die Verbraucher durch den steigenden Dollar. Sparer könnten Gewinne ebenso durch höhere Renditen sehen, auch sich die Experten darüber sind,, wie schnell dies haben wird.
Table 7: Case study. GLAT and GLAT-CTC show good translation quality at the beginning but poor quality at the end of the document. G- G-TT and GTTTCC show more consistent translation quality for sentences at different positions.
respectively, not mention the G-Transformer baseline. These results indicate that NAT models have a limited ability to utilize document context.
language models, a larger model typically shows a stronger ability to handle long sequences be- cause the increased parameters enable the model to capture more complex long-context dependen- cies. However, for a non-pretraining setting, a larger model does not necessarily show better per- formance on the current document-level MT bench- marks. The longer input makes the model more likely to overfit and larger model will make it worse given the limited training corpus. Take the AT baseline G- G-Transformer as an example. When we increase the model size from Base to Big, the d-BLEU scores decline by 0.36, 1.41, and 0.10 on TED, News, and Europarl, respectively. When we further increase the model size to Large, the d- B-BEUU scores further decline by 16.53, 8.49, and 0.56 on TED, News, and Europarl, respectively.
C Case Study.
As a case in Table 7 shows, even though the source is not a lengthy document, sentence alignment en- hances the translation quality significantly. Specif- ically, on the one hand, G-Trans+GLAT and G- TranC+G estimate the target length more accurately than GLAT and GLAT+CTC because of their fine-grained length prediction on each tar- get sentence. On the other, G-Trans+GLAT and G-Trans+GLAT+CTC show consistent translation quality between the beginning and the end of the document, while GLAT and GLAT+CTC show good quality the beginning and poor quality the end. This case illustrates the challenge of long text sequences NAT models that the translation quality degrades as long as the distance to the begin- ning of the document increases, and demonstrates the advantage of sentence alignment to stop the degradation.
Can the model handle more complex sentence alignments, such as one source sentence divided into multiple target simple sentences? Actu- ally, the current model can handle the case. The sentence-lever alignment between the source and target is achieved by the same group tag assigned to the source tokens and the target tokens. We can treat the multiple simple sentences as whole trans- lation unit, assigning them the same group tag map them to a single complex sentence.
More Discussion
Can larger model handle longer input se- quences? A larger model does not necessarily solve the longer input sequence issue given the same amount of training data. For pre-trained


Reviews:
[{"review_id": "SsJZlWEwmH", "reviewer": "Reviewer_NAdw", "summary": "", "questions": "Do authors have some insight on how the model performances in terms of the model size? Does the larger model can handle the longer input sequence issue? Large model may be more powerful to deal with the multi-modal and misalignment issue. In other word, is this study only useful for small or middle size AT and NAT models?\n\nWhy NAT+CTC and GLAT+CTC is worse than NAT and GLAT for Europarl dataset in Table 1? ", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n**Q1:** Substituting the causal masking layer in the G-Transformer results in a constrained level of innovation. \\\n**A1:** We will revise the paper to clarify it. Substituting the causal masking layer in G-Transformer is only a necessary part at the attention level to apply sentence alignment, which relies model design and training loss to work. As Section 3.3 describes, we update the NAT models to ensure the sentence alignment modeling and training. For example, we introduce new length prediction per sentence in G-Trans+GLAT, improved CTC loss for sentence alignment in G-Trans+GLAT+CTC, and improved DAG design to restrict the transition in sentence in G-Trans+DAT. Without these new designs, the NAT models cannot be reliably trained.\n\n**Q2:** How does the model perform in terms of the model size? Can the larger model handle the longer input sequence issue? \\\n**A2:** We will discuss it in the revised version. A larger model does not necessarily solve the longer input sequence issue given the same amount of training data. For pre-trained language models, a larger model typically shows stronger ability to handle long sequences because the increased parameters enable the model to capture more complex long-context dependencies. However, for a non-pretraining setting, a larger model does not necessarily show better performance on the current document-level MT benchmarks. The longer input makes the model more likely to overfit and a larger model will make it worse given the limited training corpus. Take the AT baseline G-Transformer as an example. When we increase the model size from Base to Big, the d-BLEU scores decline by 0.36, 1.41, and 0.10 on TED, News, and Europarl, respectively. When we further increase the model size to Large, the d-BLEU scores further decline by 16.53, 8.49, and 0.56 on TED, News, and Europarl, respectively. \n\n**Q3:** Why NAT+CTC and GLAT+CTC is worse than NAT and GLAT for the Europarl dataset in Table 1? \\\n**A3:** We will revise the paper to discuss it. This abnormal phenomenon is observed for the first time, and we speculate it is caused by the CTC loss on long sequence. As Table 1 shows, both NAT/GLAT+CTC and NAT/GLAT encounter training failures on the raw Europarl dataset, and the failures are most likely caused by multi-modality and misalignment issues (Section 5.1). However, training with CTC loss on documents is not always stable, leading to occasional failures such as 0.00 d-BLEU scores of NAT/GLAT+CTC on the Europarl raw dataset. We speculate that the instability of CTC loss is caused by the increased alignment space between long input and output (Line 437).\n\n**Q4:** Typos and presentation. \\\n**A4:** We will clarify the difference and the relation between the AT and NAT versions of G-Transformer. **G-Transformer (AT)** has an encoder-decoder architecture, involving two types of multi-head attention. One is for global document, naming global attention, while another is for local sentence, naming group attention. The global attention is simply a normal multi-head attention, which attends to the whole document. The group attention differentiates the sentences in a document by assigning a group tag to each sentence, which are used to calculate an attention mask to avoid cross-sentential attention. The two multi-head attentions are combined using a gate-sum module. G-Transformer (AT) uses group attention on low layers only but combined attention on top 2 layers. **G-Transformer (NAT)** follows the attention design, but removes the decoder casual mask for parallel decoding. Furthermore, the input and output of decoder are changed from autoregressive format to non-autoregressive, and NAT loss is used for training.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper undertakes a captivating exploration of the Neural Autoregressive Translation (NAT) model in the context of document-level machine translation tasks. The experimental results highlight that the proposed G-Transformer-based NAT model surpasses the performance of the Transformer model across three benchmark datasets.\n", "reasons_to_reject": "Substituting the causal masking layer in the G-Transformer results in a constrained level of innovation.", "questions_to_authors": "Do authors have some insight on how the model performances in terms of the model size? Does the larger model can handle the longer input sequence issue? Large model may be more powerful to deal with the multi-modal and misalignment issue. In other word, is this study only useful for small or middle size AT and NAT models?\n\nWhy NAT+CTC and GLAT+CTC is worse than NAT and GLAT for Europarl dataset in Table 1? ", "paper_topic_and_main_contributions": "The primary contribution of this research is the examination of Neural Autoregressive Translation (NAT) in document-level machine translation tasks. This is accomplished by adapting the G-Transformer model from its causal decoder version to the NAT decoder version.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "Yes", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "It is better to clarify the G-Transformer (AT) and G-Transformer (NAT) in the paper."}, "extra_text": {}}, {"review_id": "pb15uKSMwD", "reviewer": "Reviewer_3tvA", "summary": "", "questions": "[Q1]: One limitation of the proposed approach is assuming that there will be one-to-one alignment of sentences on target-and-source. This is true in current benchmark due to they were originally build  for sentence-level translation. However it's not true for all language pairs and writing styles. Some language allows for more complex sentences which in english will be divided into several simple sentences.  What future solution or modification in your solution do you see?\n[Q2]: NAT models are not always fluent, even if the BLEU scores are high. As you point out in your analysis, with discourse metrics. Dis you measure any issues of repetitions cross-sentences, e.g. leak of source context information on adjacent sentences?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n**Q1:** In my view, one main contribution of this paper is to serve as a benchmark for current methods, but it’s not exhaustive with current advances. \\\n**A1:** We will revise the paper to clarify this. Due to the large amount of NAT models (a non-exhaustive search shows more than 60 recent NAT models proposed between 2020 and 2023), existing studies, including XLM-D (Wang et al., 2022), DA-Transformer (Huang et al., 2022), and GLAT (Qian et al., 2021), generally do not conduct an exhaustive comparison with previous methods but instead choose several representatives. We follow the practice and select representative pure (fully) NAT models (Line198) for our investigation -- **understanding the possibility and key challenges to applying NAT to document-level MT**. We leave more complex iterative methods,  semi-autoregressive methods, and pre-training settings for future work (Line616) since they involve external factors, which may complicate the analysis. We hope our preliminary work on pure NAT models can inspire further research on more complex NAT methods. \n\n**Q2:** Current research in document-level MT, including this work, does not update with recent advances in efficient methods for supporting long inputs. This may have an impact on the reported speeding times and training efficiency. \\\n**A2:** We will revise the paper to discuss it. A major challenge for document-level MT is the large alignment space between the long input and output sequences, which may cause the training failure of AT and NAT models. Recent advances in efficient models such as Longformer, Reformer, Linformer, and FlashAttention mainly focus on improving the time and space complexity of the attention mechanism, which does not **address the alignment issue** in document-level MT. As the previous study (G-Transformer) shows that although the local attention mechanism influences the model performance by about 1.78 d-BLEU (average across TED, News, and Europarl), the sentence-level alignment dominates the performance by about 14.68 d-BLEU.\n\nDue to space limitation, we mainly discuss the relative inference acceleration of NAT compared to AT models. Therefore, we leave out general techniques such as Longformer, Reformer, Linformer, and FlashAttention, because these techniques can be applied to both AT and NAT models. We will add relevant discussion.\n\n**Q3:** This work does not include perspectives on realistic scenarios and concerts for measurement speed for NAT like batch sizes and more efficient implementations. \\\n**A3:** We will add comparisons on different batch sizes. Our experiments show that when we increase the batch size from 1 to 4 and 8, the overall speedup of GLAT decreases from 40x to 16x and 11x. If we exclude the time for initializing the models (which takes almost constant time), the inference speedup of GLAT will be 124x, 66x, and 44x for the batch size of 1, 4, and 8, respectively.\n\nMore efficient implementations such as FlashAttention can be applied to both AT and NAT models; therefore, we leave them out of our scope and focus on NAT-related techniques.\n\n**Q4:** Did you measure any issues of repetitions cross-sentences, e.g., leak of source context information on adjacent sentences? \\\n**A4:** We will discuss it in the revised version. We observe serious repetitions on NAT models **without** sentence alignment but we cannot measure the cross-sentence repetitions reliably because these models do not always generate sentence boundaries. We measure the cross-sentence repetitions on NAT models **with** sentence alignment, where the sentence boundaries are assured by the model design. We find that they do not generate obvious cross-sentence repetitions. For example, on the TED test set, G-Trans+GLAT generates translations with cross-sentence repetition ratios of 0.14 and 0.02 for 1 and 2 grams, respectively. G-Trans+GLAT+CTC generates translations with cross-sentence repetition ratios of 0.15 and 0.02 for 1 and 2 grams, respectively. In comparison, the reference translations have cross-sentence repetition ratios of 0.15 and 0.02 for 1 and 2 grams, respectively, which are almost the same as G-Trans+GLAT and G-Trans+GLAT+CTC.\n\n**Q5:** Some language allows for more complex sentences which in English will be divided into several simple sentences. What future solution or modification in your solution do you see? \\\n**A5:** We appreciate your insightful suggestion. Actually, the current model can handle the case. The sentence-level alignment between the source and target is achieved by the same group tag assigned to the source tokens and the target tokens. We can treat the multiple simple sentences as a whole translation unit, assigning them the same group tag to map them to a single complex sentence.\n\n**Q6:** Significance scores are missing. \\\n**A6:** We will revise the paper to include significance scores. G-Trans+GLAT and G-Trans+GLAT+CTC outperform GLAT and GLAT+CTC on TED, News, and Europarl (both Raw and KD) with a statistical significance at the level of p<0.01 using t-test. G-Trans+DA-Trans outperforms DA-Trans on TED and News (Raw and KD) with a statistical significance at the level of p<0.01.\n\n**Q7:** Missing references. \\\n**A7:** Thanks for the suggestions and we will add and discuss them. We will add the missing references. We will add XLM-D [1] as a representative iterative model in related work. We will add a new paragraph in related work to discuss efficient Transformers for long sequences, taking FlashAttention [2] and Long Range Arena [3] as two related studies to discuss their relation and difference to our study. We will add a new paragraph in Section 4.3 – Breakdown Results to discuss the speedup on different batch sizes, taking previous studies [4] [5] as references.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This work establishes a benchmark for NAT at document-level. \n\n- Consistent improvement shows by decoding sentences instead of documents.\n", "reasons_to_reject": "- Significance scores are missing to make the work complete and to validate several conclusions in the results description about which method works better.\n\n- One concern is that current research in document-level machine translation, including this work, does not update with recent advances on efficient methods for supporting long inputs [2,3]. This may have an impact on the reported speeding times and also training efficiency. In my view one main contribution of this paper is to serve as a benchmark for current methods but it’s not exhaustive with current advances. \n\n- Another concert is that this work does not include perspectives on realistic scenarios and concerts for measurement speed for NAT like batch sizes and more efficient implementations [4,5]. And consistent evaluation metrics [5]. In my view one main contribution of this paper is to serve as a benchmark but it’s not exhaustive with current advances. \n", "questions_to_authors": "[Q1]: One limitation of the proposed approach is assuming that there will be one-to-one alignment of sentences on target-and-source. This is true in current benchmark due to they were originally build  for sentence-level translation. However it's not true for all language pairs and writing styles. Some language allows for more complex sentences which in english will be divided into several simple sentences.  What future solution or modification in your solution do you see?\n[Q2]: NAT models are not always fluent, even if the BLEU scores are high. As you point out in your analysis, with discourse metrics. Dis you measure any issues of repetitions cross-sentences, e.g. leak of source context information on adjacent sentences?", "paper_topic_and_main_contributions": "This paper develops two points: 1) an evaluation of current non-autoregressive methods on document-level MT 2) an proposed technique to use sentence alignments so the MT decoder works at sentence level while the encoder at document-level. \nThe selected NAT methods are implemented in the same backbone model (base model from FairSeq). One important missing comparison is: Xlm-d which showed higher scores than the studied ones although having a different backbone model. The sentence alignment shows consistent improvements with respect to the full document decoding. In all evaluations significance scores are missing.\n", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "[1] https://aclanthology.org/2022.emnlp-main.466.pdf \n\n[2] https://proceedings.neurips.cc/paper_files/paper/2022/file/67d57c32e20fd0a7a302cb81d36e40d5-Paper-Conference.pdf\n\n[3] https://arxiv.org/pdf/2011.04006.pdf\n\n[4] https://aclanthology.org/2022.naacl-main.129.pdf\n\n[5] https://arxiv.org/pdf/2205.10577.pdf"}}, {"review_id": "pYv5U6Bohx", "reviewer": "Reviewer_HkNb", "summary": "", "questions": "- In Equation 2, is the parameter set \\theta of the length model the same as that of the translation model?\n- Line 357, why not \"big\" model for a stronger baseline?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback.\n\n**Q1:** Experiments on more language pairs and bigger datasets? \\\n**A1:** We will revise the paper to include one more language pair and a bigger dataset. Specifically, we consider Zh-En literary translation from WMT23, which contains 1.9M sentences. Our preliminary experiments show positive results as for En-De datasets: failure happens on GLAT, CTC improves the performance, and sentence alignment further enhances the model. We will add a new section to talk about it in the revised version if the space allows. \n\nThe latest document-level MT datasets available are at the same level as the Europarl v7 dataset (1.7M sentences, 118K documents) that we use. For example, Zh-En literary translation from WMT23 has about 1.9M sentences, and En-De Europarl v10 from WMT23 has about 1.8M sentences. A larger dataset could probably better demonstrate the advantage of our model on document-level MT, and we will discuss the potential impact.\n\n\n**Q2:** In Equation 2, is the parameter set \\theta of the length model the same as that of the translation model? \\\n**A2:** For simplicity, we take the model as a whole and use the \\theta to denote all the parameters, which includes the parameters of the length model and the translation model. We will revise the paper to clarify it.\n\n**Q3:** Line 357, why not “big” model for a stronger baseline? \\\n**A3:** A bigger model does not necessarily provide a stronger baseline on the current document-level MT benchmarks because a longer input makes the model more likely to overfit and a bigger model will make it worse given the limited training corpus. Take the AT baseline G-Transformer as an example. When we increase the model size from Base to Big, the d-BLEU scores decline by 0.36, 1.41, and 0.10 on TED, News, and Europarl, respectively. When we further increase the model size to Large, the d-BLEU scores further decline by 16.53, 8.49, and 0.56 on TED, News, and Europarl, respectively. We will add the discussion in the revised version.\n\n**Q4:** Missing references. \\\n**A4:** We will add the missing references on BLEU and Fairseq.\n\n**Q5:** Typos and presentation. \\\n**A5:** We will fix the issues and carefully proofread our paper again.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper investigates the potential of non-autoregressive translation models in document-level machine translation, an area that has not yet been thoroughly explored.\n- It introduces a simple but effective sentence alignment strategy that reduces the possible alignment space and offers improvements over the document-level machine translation benchmark.\n- It conducts a thorough investigation of typical non-autoregressive translation models in document-level machine translation, which can provide valuable insights for future research in this field.", "reasons_to_reject": "- The experiments were performed on only one language pair and the datasets used are relatively small.", "questions_to_authors": "- In Equation 2, is the parameter set \\theta of the length model the same as that of the translation model?\n- Line 357, why not \"big\" model for a stronger baseline?", "paper_topic_and_main_contributions": "The paper innovatively explores the capabilities of non-autoregressive translation models in the context of document-level machine translation, an area that has not been explored in depth to date. It presents an effective yet simple strategy for sentence alignment between source and target, and evaluates the effectiveness of the models against one benchmark with three datasets. The authors provide a comprehensive examination of standard non-autoregressive translation models in the field of document-level machine translation and uncover the difficulties these models face in managing discourse phenomena and utilizing document context. And they developed a unique sentence alignment strategy that reduces potential alignment space, providing improvements in the established benchmark. Furthermore, they show that while non-autoregressive translation models significantly speed up decoding in documents, they currently lag behind their autoregressive translation models in terms of overall performance. As a weakness, in addition to the limitations mentioned by the authors, experiments with only one language pair are not very convincing. It would be better to have experiments with other language pairs as well as larger datasets.\n\n", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- \"BLEU: a Method for Automatic Evaluation of Machine Translation\" by Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu\n- \"fairseq: A Fast, Extensible Toolkit for Sequence Modeling\" by Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli", "Typos_Grammar_Style_and_Presentation_Improvements": "- The abbreviation \"KD\" is first used at line 192, so it may be introduced there instead of in the caption of Table 1.\n- Line 208, x may already be introduced with Equation 1.\n- Line 259, does the \\beta(y) define the same as that in Equation 3? If so, it may already be introduced with Equation 3.\n- Line 319, K is already used in Equation 7 and may therefore be introduced there.\n- Line 355, the term \"d-BLEU\" is already used in Section 1, which may be introduced there.\n- Line 591, the abbreviation VP is not introduced."}}]

Author rebuttal:

Meta-review: This paper investigates the application of non-autoregressive transformers (NATs) to document-level MT. It proposes an alignment method for ensuring that generated text respects sentence boundaries, and compares NATs to standard autoregressive transformers, finding that the former produce poorer-quality document-level translations.

Reviewers felt that this was a substantial and novel contribution that establishes useful benchmark results for document-level NATs. They appreciated the comprehensive evaluation with different recent NAT architectures, and the honest comparison that demonstrates underperformance. Negatives included the use of small datasets for only one language pair, lack of statistical significance, and insufficient exploration of known techniques for speeding up the autoregressive baselines.

This is a solid contribution that establishes a useful new baseline technique and corresponding benchmark result for document-level work with NATs. It will serve as a good point of reference for people hoping to extend this line of work. The main downside is the somewhat narrow scope of the evaluation, but this is unlikely to seriously affect the conclusions.

I would like to highlight the use of BLEU rather than more accurate neural metrics such as COMET and BLEURT. Since this is typical of broad areas of MT research, it would be unfair to penalize this particular paper, but this is something that should change in the future, for high-resource settings.
Decision: Accept-Findings