Putting Machine Translation in Context with the Noisy Channel Model
Authors: Lei Yu
Laurent Sartran
Wojciech Stokowiec
Wang Ling
Lingpeng Kong
Phil Blunsom
Chris Dyer
Venue:  (2020)
Keywords: machine translation
context-aware machine translation
bayes rule

Abstract: We show that Bayes' rule provides a compelling mechanism for controlling unconditional document language models, using the long-standing challenge of effectively leveraging document context in machine translation. In our formulation, we estimate the probability of a candidate translation as the product of the unconditional probability of the candidate output document and the ``reverse translation probability'' of translating the candidate output back into the input source language document---the so-called ``noisy channel'' decomposition. A particular advantage of our model is that it requires only parallel sentences to train, rather than parallel documents, which are not always available. Using a new beam search reranking approximation to solve the decoding problem, we find that document language models outperform language models that assume independence between sentences, and that using either a document or sentence language model outperform comparable models that directly estimate the translation probability. We obtain the best-published results on the NIST Chinese--English translation task, a standard task for evaluating document translation. Our model also outperforms the benchmark Transformer model by approximately 2.5 BLEU on the WMT19 Chinese--English translation task.

Under review as a conference paper at ICLR 2020
PUTTING MACHINE TRANSLATION IN CONTEXT WITH THE NOISY CHANNEL MODEL
Anonymous authors Paper under double-blind review
ABSTRACT
We show that Bayes' rule provides compelling mechanism for controlling un- conditional document language models, using the long-standing challenge of ef- fectively leveraging document context in machine translation. In our formulation, we estimate the probability of a candidate translation as the product of the uncon- ditional probability of the candidate output document and the 'reverse translation probability" of translating the candidate output back into the input source lan- guage document- the so-called "noisy channel" decomposition. A particular ad- vantage of our model is that requires only parallel sentences to train, rather than parallel documents, which are not always available. Using reranking approximation to solve the decoding problem, we find that document language models outperform language models that assume independence between sentences, and that using either document or sentence language model outper- forms comparable models that directly estimate the translation probability. We obtain the best-published results on the NIST Chinese-Engllsse English translation task, a standard task for evaluating document translation. Our model also outperforms the benchmark Transformer model by approximately 2.5 BLEU on the WMT19 Chinese-English translation task.
beam search
INTRODUCTION
There have been many recent demonstrations that neural language models based on Transform- ers (Vaswani et al.) 2017 Dai et al. 2019) are capable of learning to generate remarkably coherent
documents with few (Zellers et al., 2019) or no (Radford et al., 2019) conditioning variables. Al- though large unconditional language models are frequently used to provide representations for nat- ural language understanding applications (Devlin et al. 2019 Yang et al. 2019 Peters et al. 2018). how to use them for conditional generation applications remains open question.
In this paper, we employ Bayes' rule to use powerful unconditional document language models to improve conditional task: machine translation. The application of Bayes' rule to transform the translation modeling problem p(y x). where y is the target language, and æ is the source language, has long tradition and the dominant paradigm in speech and language processing for many years (Brown et al.) 1993). In this context is known as the noisy channel (Shannon & Weaver 1949). by analogy an information theoretic conception of Bayes' rule model for reasoning about messages being sent across an unreliable communication medium. In the noisy channel decomposition (:2), rather than directly modeling the conditional distribution, we rewrite as p(y æ) x p(y) x xp(x y). More importantly, this changes the learning problem from estimating single complex conditional distribution to learning two different distributions: a language model p(y), which provides unconditional estimates of the output (in this paper, documents); and p(æ: y), which provides the probability of translating a candidate output y into the (observed) source æ.
As we will discuss below, although the problems of estimating p(y | x)) and p(= y) are similar, independence assumptions made in p(x 3) are less statistically costly than they might otherwise be since, at test time, we will be conditioning over y. which will be jointly dependent on all (conditionally independent) parts of x. This statisti- cal facch is the same trick that gives naive Bayes classifiers their expressiveness and ease of estimation- peenettsettss us to assume independence between sentence translations in the reverse trans- lation model, and therefore to use parallel sentences (rather than parallel documents) to train it. In
and reasoning about posterior distribution

---

Under review as a conference paper at ICLR 2020
the posterior, we thus have an implicit estimate of document-level translation system, even though we made no use of parallel documents. This is particularly useful since parallel sentences are much more readily available than parallel documents. A second benefit of our approach is that the un- conditional language model can be estimated from nonparallel data, which exist in vast quantities. The benefits of strong language models estimated from nonparallel data has long been recognized in translation (Brants et al., (007). and more recently in noisy channel approaches to translation based
on neural network component distributions (Yu et al. 2017 Yee et al. 2019 Ng al. 2019).
Although the noisy channel model is ideal for exploiting the data resources that naturally exist in the world (large corpora of parallel but independent sentences, and large corpora of monolingual documents), we are faced with a much harder decoding problem (t3). To address this problem, we propose new beam-search algorithm, exploiting fact that our document language model operates left-to-right, and our reverse translation model treats sentences independently. The search guided by proposal distribution that provides candidate continuations of document prefix, and these are reranked according to the posterior distribution. In particular, compare two proposal models: one based on estimates of independent sentence translations (Vaswani et al. 2017) and one that conditions the source document context (Zhang et al. 2018).
To explore the performance of our proposed model, we focus on Chinese- -ngllsh translation, fol- lowing a series of papers document translation (Zhang et al. 2018 Werlen et al. 2018 Tu al. 2018 Xiong et al., 2019). Although in general it is unreasonable to expect that independent transla- tions of sentences would lead coherent translations of documents between any pair of languages, the task of translating Chinese into English poses some particularly acute challenges. As Chinese is less morphosyntatically complex than English, and the relevant clues for predicting, e.g., what tense a verb should be in, or whether a noun is singular or plural, may be spread throughout a document, it is crucial that extra sentential context is used.
Our experiments assumptions when modelling documents (i.e., whether sentences are generated independently or not), (3) different amounts of language modelling data, and (4) different proposal models. Briefly summarized, we find that document-context language models significantly improve the translation quality obtained with our system, both in terms of BLEU scores, and in terms of human evaluation. Targeted error analysis demonstrates the document prior is capable of enforcing consistency of tense and number and lexical choice across documents.
explore: (1) different approaches to reranking, (2) different independence
2 MODEL DESCRIPTION
We define x = ( x²) as the source document with sentences, and similarly, y (y', yy....,,y" y²) as the target document with J sentences. During the (human) translation pro- cess, translators may split or recombine sentences, but we will assume that = / Let x'
x'r) represent the ith sentence in the document, consisting of M words; like- wise y' (vi.w)....,w denote the ith sentence in the target document, containing N words.
The translation of document æ is determined by finding the document ÿ, where p(O
2) is optimal.
arg max pyy x).
y
Instead of modelling the probability pyy Ø)) directly, we factorize it using Bayes' rule:
p(
y)
ply)
ý arg max
arg max p(æ
X
P
(2)
p
chanoel model
language model
We further assume that sentences are independently translated, and that the sentences are generated by a left-to-right factorization according to the chain rule. Therefore, we have
arg max
I p(æ
v²)
p(y"
Size mismatches addressed by merging sentences using sentence alignment algorithms.
2

---

Under review as a conference paper at ICLR 2020
y
y
y
y
y
aF
x
a
Figure 1: Graphical model showing the factorization of our noisy channel model where y' indicates the ith target language sentence and x' indicates the ith source language sentence. In the prior (left) the target sentences (the y's) only influence the corresponding source sentence and therefore be learned and modeled independently, but at test time (right), when the target is not observed, each y is dependent every a:".
denotes a document prefix consisting of the first i target sentences. Thus conceived, this is a generative model parallel documents that makes a particular indepen- dence assumption; we illustrate the corresponding graphical model on the left side of Figure
where y "
..... . 1)
2.1 IMPACT OF THE CONDITIONAL INDEPENDENCE ASSUMPTION
At first glance, the conditional independence assumption we have made might seem to be the very in- dependence assumption that bedevils traditional neural document translation systems translations of sentence i appear to be uninfluenced by the translation of any sentence j * i. However, while this is the case during training, this is not the case at test time. Then, we will be conditioning on the at 's (the source language sentences), and reasoning about the posterior distribution over the "under- lying" y ss. By conditioning on the child variables, conditional dependencies between all y, 's and between each y, and all the posterior distribution are shown in the right part of Figure
created (Shachter, 1998). The (in)dependencies that are present in
Thus, although modelling pyy Ø)) or p(z y) would appear to be superficially similar, the statistical impact making a conditional independence assumption is quite different. This fortunate, as makes it straightforward to use parallel sentences, rather than assuming which are less often available (Voita et al. 2019: Zhang et al. 2018: Maruf et al. 2019 inter alia). Finally, since we only need to learn model the likelihood of sentence translations (rather than document translations), the challenges of guiding the learners to make robust generalizations in
have parallel documents
direct document translation models (Voita al. 2019 Zhang et al. 2018 Maruf al. 2019 inter alia) are neatly avoided.
2.2 LEARNING
We can parameterize the channel probability p(a" y using any sequence-to-scquenca model and parameterize the language model p(y' y (²) using any language model. It is straightforward to leam our model: we simply optimize the channel model and the language model separately on par- allel data and monolingual data, respectively. We remark that it is significant practical advantage of this parameterization that we can retrain the channel and language models independently, for ex- ample if acquire more monolingual data, or use different language models with the channel model conditioned on the domain of the source text.
3 DECODING
Because of the global dependencies in the posterior distribution, decoding in our noisy channel model computationally complex. one hand, similarly to the decoding problem faced in stan- dard sequence-to-sequens
models, we must search over the space of all possible outputs with model that makes no Markov assumptions. On the other hand, unlike traditional neural MT models, we have to have complete guess of y: before we can reliably compute the score of p(æ; y:) according to our channel model, making traditional greedy and near- sear-greddy algorithms ineffective. To deal with this issue, leverage auxiliary proposal model g x) which approximates the posterior distribution using a direct model, and we use this to focus on our search on promising parts of the output space.
3

---

Under review as a conference paper at ICLR 2020
Phase 1: sample candidate translations
Phase beam search
9la
""""""""""""
1
1.103
2,1
3,1
ac
Proposal model
0.11
2
y 3,2
y
4.2
y'
1,30
2,3
3.3
219
1p(a
Figure 2:
The decoding process. In Phase 1, the auxiliary proposal model generates candidate trans- lations (3 candidates in the diagram) for each sentence in the document (containing 4 sentences). In Phase 2, beam search is employed to search for the best path from the candidate translations.
Because of the autoregressive factorization the language model, and the independent sentence translation assumption in the channel model, to-right beam search strategy with the aid of our proposal model. Figure 2 iillustrates the decoding process. For imput document, each sentence in the document. We then search for the best path from these candidate translations. We use beam search under the objective defined by the following linear model,
carry the reranking process using left-
let the proposal model propose K candidate translations for
Ogst Å log x S() + X2 log 1(x
y S log p(y
S1
where ly denotes the total number tokens in the partial document y Note that Equation a generalization of Equation 2 in log space-if we set λ As and take the log of Equation
the two objectives are equivalent. The extra factors- the proposal probability and the length of the
output-provide improvements (e.g.. by calibrating the expected length of the output), and can be
incorporated at no cost in the model; they are widely used prior work (Yu et al. 2017 Yee et al.
2019
Ng al. 2019).
EXPERIMENTS
We evaluate our model translation tasks, the NIST Chinese-English task and the WMT19 Chinese-English task. On both tasks, we use the standard parallel training data, and compare our model with strong Transformer baseline, as well as related models from prior work.
4.1 DATASET DESCRIPTION
The NIST training data, containing 1.5M sentence pairs, is composed from LDC-distributed news articles and broadcasting scripts. The document-level parallel corpus a subset of the full training set, including 55K documents with 1.2M sentences. We the MT06 dataset validation set and MT03, MT04, MTOS, and MT008 as test sets. There are 79 documents and 1649 sentences in the vali- dation set and in total 509 documents and 5146 sentences in the set. We preprocess the dataset by doing punctuation normalization, tokenization, and lower casing. We use byte pair encoding (Sen-l nrich et al. 2016b) with 32K merges segment words into sub-word units for both Chinese and English. The evaluation metric is case-insensitive BLEU calculated using mult -bleu. perl, which is consistent with prior work on this task.
The training data for the WMT19 Chinese Chinese-English task includes the UN corpus, CWMT, and news commentary. The total number of sentence pairs 18M after filtering the data by removing dupli- cate sentences and sentences longer than 250 words. The validation sets that we use in the experi- ment are newstest2017 and newstest2018. 88, which contains 169 documents, 2001 sentences and 275 documents, 3981 sentences, respectively. The test set is newstest2019 containing 163 documents and 2000 sentences. The dataset is preprocessed by segmenting Chinese sentences and normaliz- ing punctuation, tokenizing, and true casing English sentences. As for NIST. we learn byte pair encoding (Sennrich et al., (2066b) with 32K merges to segment words into sub-word units for both Chinese and English. The evaluation metric is sacreBI.EU (Post 2018).

---

Under review as a conference paper at ICLR 2020
Method
Model
MT06
MT03
MT04
MT05
MT08
Wang et al.
2017)
RNNsearch
37.76
36.89
27.57
Kuang al.
2017
RNNsearch
38.40
32.90
31.86
Kuang et al.
201
Transformer
48.14
48.05
47.91
48.53
38.38
Zhang et al.
2018
Doc-transformer
49.69
50.21
49.73
49.46
39.69
Sent-transformer
47.72
47.21
49.08
46.86
40.18
Baseline
Doc-transformer
49.79
49.29
50.17
48.99
41.70
Sent-reranker 51.33
52.23
52.36
51.63
43.63
This work
Doc-reranker 51.99
52.77
52.84
51.84
44.17
Table I: Comparison with prior work NIST Chinese Chinese-Engllih translation task. The evaluation metric is tokenized case-insensitive BLEU. The first four rows are numbers reported in the papers of prior work. The first two baselines are the results that got by running the Transformer (Vaswani et al. 2017) and the Document Transformer (Zhang et al. 2018) on the NIST dataset. The sent- reranker is variation of our model in which sentences in documents are assumed be independent.
4.2 MODEL CONFIGURATION
For NIST, we use the Transformer (Vaswani et al. 2017 as the channel model and the Document Transformer (Zhang et al. 2018) as the proposal model. The hyperparameters for training the Trans- former is the same transformer base (Vaswani et al. 2017), i.e. 512 hidden size, 2048 filter size, attention heads, and 6 layers for both the encoder and decoder. We follow Zhang et al. (2018) 's configuration to train the Document Transformer: context length is set to 2 and all other hyperpa- rameters are the same as transformer base. Both models are optimized using Adam (Kingma & Ba 2015) with approximately 24K BPE tokens per mini-batch. For the language model, train the Transformer-XI. (Dai et al.) (019)) on a combination of the English side of NIST training data as well as three sections of Gigaword: XIN, AFP, APW, resulting in total of 7.3M documents and 115M sentences. We use an architecture with 24 layers, 16 attention heads, and embeddings of dimension 1024. The input sequences the language model are encoded into bytes using the byte-level encoder provided by GPT2 (Radford et al. 2019).
For WMT19, we use the Transformer (Vaswani al. 2017) as both the channel and proposal model. The hyperparameters for training the Transformer is the same as transformer big (Vaswani et al.) 2017]. i.e. 1024 hidden size, 4096 filter size, 16 attention heads, and layers. The model trained on 8 GPUs with batch size of 4096. The setup for the language model is the same that of NIST except that the training data is the English side of the parallel training data and Gigaword.
For both tasks, the weights À are selected using grid search, from J0.8, 1., 1.5, 2., 2.2, 2.5, 3.] for the weights of channel model X2 and proposal model A1. and from (0.2, 0.5, 0.8, 1.] for the length penalty λg. The size of the n-best list used in the reranker is set to 50.
4.3 EXPERIMENTAL RESULTS
Table presents the best result from our model (doc-reranker) in comparison with prior work the NIST Chinese-Enghh Englishh translation task. The first four rows are numbers reported in prior work. Wang et al. (2017) incorporate document context by introducing hierarchical RNN
LSTM requence-to-sequene model. Kuang et al. (2017) use a cache to store previously translated words sentences, which they then in sequence-to-sequene conteneetto-sequueeee models. [Zhang et al. (2018) tend the Transformer model with an extra context encoder to capture information from previous source sentences. Apart from prior work, we also compare our doc-reranker with three baselines: the Transformer (Vaswani et al. 2017) and Document Transformer (Zhang et al. 2018), and the sentence-level reranker rsenkere In the sent-reranker, we assume sentences the document are independent (formulation ÿ = arg max,, [||=] p(x' yy) x p(y')), and therefore we train a sentence-level language model and rerank each sentence independently. This sent-reranker setup is close to the work from Yee et al. (2019) and Ng al. (2019) with the difference that rather than using a language model trained on documents we use a language model trained on sentences, which is more statistically consistent. Table T shows that our reranker outperforms previous models as
5

---

Under review as a conference paper at ICLR 2020
Proposal model
Language model
Sent-reranker
Doc-reranker
LSTM: NIST Transformer-XI. NIST
49.92
50.24
Sent-transformer
50.29
50.56
Transformer-XI. NIST + Gigaword
50.19
50.93
LSTM: NIST Transformer- NIST
50.75
51.20
tro-trrasformer
51.27
51.68
Transformer-XL: NIST + Gigaword
51.33
51.99
Table 2: BLEU scores on NIST dev set MT06 from rerankers which are incorporated with various language models. In the language model column X: Y means the language model X is trained on dataset Y. A bigger language model improves the doc-reranker but does not help the sent-reranker.
Independence
Architecture
Data
Perplexity
Transformer- XI
NIST
83.3
Indep. Sentences
Transformer-X XL.
NIST Gigaword
96.5
LSTM
NIST
71,6
Indep. Documents
Transformer-X -XL
NIST
43.8
Transformer-X
NIST Gigaword
43.4
Table 3: Perplexity per word (whitespace delimited token) of language models on NIST dev set.
well as strong Transformer baselines by a significant aaaginn-aproximaatte 2.5 BLEU on all test sets achheeing state of the art. Although the gap between the doc-reranker and sent-reranker smaller, as we will show in A. and } that translations generated by doc-reranker are preferred by humans and more consistent across documents, in line with concerns about the reliability of using BLEU at assessing cross-sentential consistency (Voita et al. 2019).
To understand the rerankers better, investigate the effect of different proposal models, different language models, and various numbers of candidates in the n-best list. Table Sable and Figure 3 show that better proposal models and bigger ne-best lists lead to consistently better reranking results. This is an appealing behaviour showing that our reranker is able to pick better translations from higher quality and more diverse candidate pools generated by better proposal models and bigger n-best lists. To compare the effect of language models, we train an LSTM language model (Merity et al.) 2018a(b) (b) and a Transformer-x -L. language model on the English side of NIST parallel training data in addition to the Transformer-XL. trained on NIST and Gigaword. Table lists the perplexity per word on the NIST validation set for different language models. Given the same training data, the Transformer-Xi XL performs significantly better than the LSTM- language model, which in turn results in a higher BLEU score from the doc-reranker. By adding more training data, the Transformer-XI. language model achieves the performance of the doc-reranker. Notably, when the strong Transformer-X language model is incorporated into the doc-reranker, the best weight ratio of the channel and language model is 1, indicating that the doc-reranker depends heavily on the language model. By contrast, if a weak language model is incorporated, the best ratio is approximately 2 1. A further observation that although a larger-scale language model improves the doc-reranker, it does not help the sent-reranker.
lower perplexity and that gives further boost to
We perform an ablation study to explore what each component of the doc-reranker contributes to the overall performance. Table shows BLEU scores on the NIST validation set for the optimal in- terpolation of various component models. No gains are observed if the language model is combined with the proposal model (a probabilistically unsound combination, although one that often worked in pre-neural approaches to statistical translation). We find that as we increase the weight the lan- guage model, the results become worse. The interpolation of the proposal model and channel model slightly outperforms the proposal model baseline but considerably underperforms the interpolation of the proposal model, channel model, and the language model. This difference indicates the key roles that the language model plays in the doc-reranker. When the channel model is combined with the language model the performance of the doc-reranker comparable that with all three compo-

---

Under review as a conference paper at ICLR 2020
Reranker
Models
MT06
323
Doc-transformet
49.79
0 3:0
Proposal LM
49.79
315
Channel L.M
51.93
53.
Doc-reranker
Proposal Channel
50.40
Proposal + Channel LM
51.99
Figure 3: Effect of n-best list.
Table 4: Effect of different components in doc-reranker.
Method
Model
Unpaired Data
LM PPL
Test17
Testl 8
Test19
Baseline
Transformer big
23.9
23,9
24.5
WMT
106.3
24.9
26.0
27.1
This work
Doc-reranker
Gigaword + WMT
63.8
25.5
26.3
27.1
Table Lac of different models WMT19 validation and sets and perplexity per word of the language models on the English side of WMT19 validation set.
nents included. We conclude from the ablation study that both the channel and language models are consistent with our Bayes' rule account of this model.
indispensable for the doc-reranker,
Table | presents the results of our model together with baselines on the WMT19 Ehglese-English translation task. We find that the results follow the same pattern as those NIST: better language model leads to better translation results and overall the reranker outperforms the Transformer-big by approximately 2.5 BLEU. The two best systems submitted to the WMT19 Chinese Seiinse-Engglshh trans- lation task are Microsoft Research Asia's 's system (Xia et al. 2019) and Baidu's 's system (Sun et al.) 2019) both of which employ multiple techniques improve upon the Transformer big model. Here mainly compare our results with those from Xia et al. (2019) because we use the same evaluation metric BLCU and the same validation and test sets. Using extra parallel training data and the techniques of masked sequence-tor -osnnnce----suuunne pretraining (Song al.) (2019), sequence-level knowl- edge distillation (Kim & Rush 2016), and backtranslation (Edunov et 2018), the best model from (Xia et al. 2019) achieves 30.8, 30.9, and 39.3 newstest2017. newstest2018, and newstest2019, respectively. Although our best results are lower than this, it is notable that our reranker achieves comparable results to their model trained on 56M parallel data, than times more training data than we use. However, our method is orthogonal to these works and can be combined with other techniques make further improvement.
5 ANALYSIS
To investigate how the rerankers improve translation quality, analyze the output from different models: the Document Transformer (Zhang et al. 2018) (our proposal model), the sent-reranker, and the doc-reranker. We observe that general the doc-reranker improves adequacy of translations and can generate more fluent and natural sentences compared to the Document Transformer. More importantly, our doc-reranker shows its superiority over the others in terms of exploiting context, improving consistency of tense, number, and lexical choice across entire articles. Tables 6 and i in Appendix A present example output from the aforementioned systems. In Example 1, the pronoun he is omitted in the Chinese sentence. While the Document Transformer misses this pronoun result- ing a translation completely different meaning, the doc-reranker is able to recover Likewise, in Example 7 them is dropped in the source sentence and this pronoun can only be inferred from pre- vious context. Although both rerankers recover some pronoun, only the doc-reranker gets right, pross-sententia context. Example 2 is good example showing that the doc-reranker better at generating adequate translations than the proposal model: the Document Transformer ignores the phrases with these people and at present, but the doc-reranker covers them.
by relying
7

---

Under review as a conference paper at ICLR 2020
Chinese does not mark grammatical number, it therefore has to be inferred for the English transla- tions. It is not possible for a sentence-level MT system to capture this information if the relevant context is not from the current sentence. In Example 4 and 6 the plural problems and identities can only be inferred from previous sentences (the immediate previous sentence in Example 4 and the sentence 4-5 sentences away from the current one in Example 6). While neither the Document Transformer nor the sent-reranker makes the right predictions in both examples, the doc-reranker translates correctly, indicating its strength in capturing extra-sentential information. In addition to making inference across sentences, the doc-reranker is also capable of maintaining consistency of tense and lexical choice, as demonstrated in Examples 5, 8, and 10. Furthermore, impressively, improves the consistency of writing style. To illustrate, in Example 9, the context is that of list of bullet points starting with continue. The doc-reranker follows this style by starting the translation with the verb continue. However, the sent-reranker starts the sentence with we should continue. Although both translations are reasonable, the former one is more natural within the document.
RELATED WORK
Our work closely related to three lines of research: context-aware neural machine translation, large-scale language models for language understanding, and semi- supervised machine translation. Recent studies (Tiedemann & Scherrer 2017; Bawden et al.] 2018 inter alia) have shown that exploiting document-level context improves translation performance, and in particular improves lexical consistency and coherence of the translated text. Existing work in the area of context-aware NMT typically adapts the MT system take additional context input, either few previous
sentences (Jean et al. 2017] Wang et al.) 2017]] Tu et al. [0018] Titov et al., 2018 Zhang et al.)
2018 Werlen et al. (0188) or the full document Haffari & Maruf 2018 Maruf al. | 2019). The
idea of adding monolingual document-level data has been explored in two most recent works (Voita
al. 2019
2019). Both use backtranslation (Edunov et al.. 2018 Sennrich
it al. 2016a) to create synthetic parallel documents as additional training data. We tram a large-
scale language model and use it to refine the consistency between sentences under a noisy channel framework. An advantage of our model over backtranslation is that the language model is portable across domain and language pairs.
Investigating ways of adding monolingual data to MT systems is an active research area (Gülçehre et al. 2015 Cheng et al. 2016) inter alia). Backtranslation (Edunov et al.] 20188 Sennrich et a 2016a), originally invented for semi-supervisee MT, has been employed as standard approach for unsupervised MT (Lample et al. 20188b al Artetxe et al.] 2019 2019)2018) Noisy channel decompositions have been standard approach in statistical machine translation (Brown al., 1993 Koehn al., 2007) and recently have been applied to neural models (Yu et al.) 2017] Yee et al. 2019 Ng et al,,
2019 Unlike prior work, we adopt noisy channel models for document-level MT where language models are responsible for capturing cross-sentence context.
Large-scale pretrained language models have achieved great success in improving systems in lan- guage understanding, leading to state-of-the-art results on a wide range of tasks (Peters et al. 2018
Devlin et al. 2019 Radford et al. 2018 McCann et al. 2017: Yang et al. a 2019 Chronopoulou al., 2019 Lample & Conncaul (2199). Language generation is another area where pretrained language
models have been applied to, with existing work focusing on fine-tuning approaches for repurposing an unconditional language model (Zhang et al. 2019: Edunov et al. 2019: [ong et al.) 2019 Dong
ctt al. 2019; Ziegler et al., 2019 de Oliveira & Rodrigo 2019). In contrast to our work which uses probabilities from language models, that work uses their internal representations.
7 CONCLUSION
We have presented noisy channel reranker and empirically validated it on Chinese-Englist document-level translation tasks. The noisy channel formulation requires only parallel sentences (rather than documents) but we can use abundant monolingual documents to train the language model component. Experiments show that our proposed model considerably improves translation quality-it it achieves approximately 2.5 BLEU higher than transformer baselines. Subjective eval- uation further confirms that the language model helps enforce consistency of tense, number, and lexical choice across documents.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Mikel Artetxe, Gorka Labaka, Eneko Agirre, and Kyunghyun Cho. Unsupervised neural machine translation. Proceedings of ICLR, 2018.
Mikel Artetxe, Gorka Labaka, and Encko Agirre. An effective approach to unsupervised machine translation. Proceedings of ACL., 2019.
Rachel Bawden, Rico Sennrich, Alexandra Birch, and Barry Haddow. Evaluating discourse phe- nomena in neural machine translation. In Proceedings of NAACL-HLT, 2018.
Thorsten Brants, Ashok C. Popat, Peng Xu, Franz J. Och, and Jeffrey Dean. Large language models in machine translation. In Proceedings of EMNLP-CONEE June 2007.
Peter F. Brown, Stephen Della Pietra, Vincent J. Della Pietra, and Robert L.. Mercer. The mathe- matics of statistical machine translation: Parameter estimation. Computational Linguistics, 19(2): 263-311, 1993.
Yong Cheng, Wei Xu, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. Semi- supervised learning for neural machine translation. In Proceedings of ACI., 2016.
Alexandra Chronopoulou, Christos Baziotis, and Alexandros Potamianos. An embarrassingly sim- ple approach for transfer learning from pretrained language models. In Proceedings of NAACL- HLT, 2019.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime G. Carbonell, Quoc Viet Le, and Ruslan Salakhutdi- nov. Transformer-xl: Attentive language models beyond fixed-length context. In Proceedings of ACL, 2019.
Luke de Oliveira and Alfredo Láinez Rodrigo. Repurposing decoder-transformee language models for abstractive summarization. ArXiv, abs/1909.00325, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, 2019.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Wuen Hon. Unified language model pre-training for natural language understanding and generation. CoRR, abs/1905.03197, 2019.
Sergey Edunov, Myle Out, Michael Auli, and David Grangier. Understanding hack-translation at scale. In Proceedings of EMNLP, 2018.
Sergey Edunov, Alexei Baevski, and Michael Auli. Pre-trained language model representations for language generation. In Proceedings of NAACL-HLT, 2019.
Çaglar Gülçehre, Orhan Firat, Kelvin Xu, Kyunghyun Cho, Loic Barrault, Huei-Chi Lin, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. On using monolingual corpora in neural ma- chine translation. CoRR, abs/1503.00335, 2015.
Gholamreza Haffari and Sameen Maruf. Document context neural machine translation with memory networks. In Proceedings of ACI., 2018.
Sébastien Jean, Stanislas Lauly, Orhan Firat, and Kyunghyun Cho. Does neural machine translation benefit from larger context? CoRR, abs/1704.05133. 2017.
Marcin Junczys-Dommunt, Microsoft translator at WMT 2019: Towards large-scale document-level neural machine translation. In Proceedings of WMT. 2019.
Yoon Kim and Alexander M. Rush. Sequence-level knowledge distillation. In Proceedings of EMNLP, 2016.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proceedings of ICLR, 2015.

---

Under review as a conference paper at ICLR 2020
Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. Moses: Open source toolkit for statistical machine translation. In Proceedings of ACI., 2007.
Shaohui Kuang, Deyi Xiong, Weihua Luo, and Guodong Zhou. Cache-based document-level neural machine translation. CoRR, abs/1711.112211 2017.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. CoRR, abs/1901.07221, 2019.
Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Unsupervised machine translation using monolingual corpora only. In Proceedings of ICLR, 2018a.
Guillaume Lample. Myle Ott, Alexis Conneau, Ludovic Denoyer, and Marc' Aurelio Ranzato. Phrase-based neural unsupervised machine translation. In Proceedings of EMNLP, 201
Sameen Maruf, André F. T. Martins, and Gholamreza Haffari. Selective attention for context- neural machine translation. In Proceedings of NAACL-HLT, 2019.
Bryan McCann, James Bradbury, Caiming Xiong, and Richard Socher. Learned in translation: Contextualized word vectors. In Proceedings of NeurlPS, 2017.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing LSTM language models. In Proceedings of ICLR, 2018a.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. An analysis of neural language modeling at multiple scales. CoRR, abs/1 180.088200, 2018b.
Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov. Facebook fair's WMT19 news translation task submission. In Proceedings of WMT, 2019.
Matthew E. Peters, Mark Neumann, Mohit lyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. In Proceedings of NAACL, 2018.
Matt Post. A call for clarity in reporting BLEU scores. Proceedings of WMT, October 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training. 2018
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 1(8), 2019.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Proceedings of ACL. 2016a.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of ACI., 2016b.
Ross D. Shachter. Bayes-ball: The rational pastime (for determining irrelevance and requisite infor- mation in belief networks and influence diagrams). Proceedings of UAI, 1998.
Claude E. Shannon and Warren Weaver. The Mathematical Theory of Communication. University of Illinois Press, Urbana and Chicago, 1949.
Matthew Snover, Bonnie Dorr, Richard Schwarz, Linnea Micciulla, and John Makhoul. A study of translation edit rate with targeted human annotation. In Proceedings of AMTA, 2006.
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. MASS: masked sequence to sequence pre-training for language generation. In Proceedings of ICML, 2019.
Meng Sun, Bojian Jiang, Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. Baidu neural machine translation systems for wmt19. In Proceedings of WMT, 2019.
10

---

Under review as a conference paper at ICLR 2020
Jörg Tiedemann and Yves Scherrer. Neural machine translation with extended context. In Proceed- ings of DiscoMTe EMNLP, 2017.
Ivan Titov, Rico Sennrich, Pavel Serdyukov, and Elena Voita. Context-aware neural machine trans- lation learns anaphora resolution. In Proceedings of ACL, 2018.
Zhaopeng Tu, Yang Liu, Shuming Shi, and Tong Zhang. Learning to remember translation history with a continuous cache. TACL, 6:407-420, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of NeurIPS, 2017.
Elena Voita, Rico Sennrich, and Ivan Titov, When a good translation is wrong in context: Context- aware machine translation improves on deixis, ellipsis, and lexical cohesion. In Proceedings of ACL, 2019.
Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. Exploiting cross-sentence context for neural machine translation. In Proceedings of EMNLP, 2017.
Lesly Miculicich Werlen, Dhananjay Ram, Nikolaos Pappas, and James Henderson. Document- level neural machine translation with hierarchical attention networks. In Proceedings of EMNLP, 2018.
Yingce Xia, Xu Tan, Fei Tian, Fei Gao, Di He, Weicong Chen, Yang Fan, Linyuan Gong, Yichong Leng. Renqian Luo, et al. Microsoft research asia's 's systems for wmt19, In Proceedings of WMT, 2019.
Hao Xiong, Zhongjun He, Hua Wu, and Haifeng Wang. Modeling coherence for discourse neural machine translation. In Proceedings of AAAI, 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237, 2019.
Kyra Yee, Yann Dauphin, and Michael Auli. Simple and effective noisy channel modeling for neural machine translation. In Proceedings of EMNLP. 2019.
Lei Yu, Phil Blunsom, Chris Dyer, Edward Grefenstette, and Tomás Kocisky. The neural noisy channel. In Proceedings of ICLR, 2017.
Rowan Zellers, Ari Holtzman, Hannah Rashkin, Yonatan Bisk, Ali Farhadi, Franziska Roesner, and 2019.
Yejin Choi. Defending against neural fake news. arXiv preprint
Haoyu Zhang, Yeyun Gong, Yu Yan, Nan Duan, Jianjun Xu, Ji Wang, Ming Gong, and Ming Zhou. Fretraining-base natural language generation for text summarization. CoRR, abs/ abs/1992..0923,, 2019.
Jiacheng Zhang, Huanbo Luan, Maosong Sun, Feifei Zhai, Jingfang Xu, Min Zhang, and Yang Liu. Improving the transformer translation model with document-level context. In Proceedings of EMNLP, 2018
Zachary M. Ziegler, Luke Melas-Kyriaz) Sehastian Gehrmann, and Alexander M. Rush. Encoder- agnostic adaptation for conditional language generation. CoRR, hhs//1908.06933 2019.
11

---

Under review as a conference paper at ICLR 2020
A APPENDIX
A.1 HUMAN EVALUATION
We selected 50 translation triplets (reference translation, translation from the doc-reranker, transla- tion from the sent-reranker) sampled from the validation and test sets of NIST for evaluation by native English speakers. The samples are selected by taking the triplets where the output from the sent-reranker and the doc-reranker have translation edit rate (Snover et al. 2006) above 17.5%.
For each of these documents, they were presented with reference translation and with two transla- tions labelled A and B, one generated by the doc-reranker and one generated by the sent-reranker. They were tasked with indicating which of these two they found better overall, considering fluency, idiomaticness and conrectness (relatively to the reference).
Each of the human evaluators preferred a majority of doc-reranker translations. When aggregated for each document by majority vote, the doc-reranker translations were considered better in 25 documents, worse for 13, and tied for 12. This is a statistically significant preference at p < 0.05 according to exact one-tailed Binomial test (n 38).
A.2 COMPARISON OUTPUT FROM DIFFERENT SYSTEMS
To investigate how the rerankers improve translation quality, we manually inspect the output from three different systems: the Document Transformer (Zhang et al., 2018), the sent-reranker, and the doc-reranker. Tables 6 and 7 present the comparison between the output from the Document Transformer (Zhang et al. 2018) and sent-reranker and between the output from sent-reranker and doc-reranker, respectively. In general, we find that the doc-reranker outperforms other systems in terms maintaining consistency of tense, number, and lexical choices across documents. For detailed analysis, refer readers to
12

---

Under review as a conference paper at ICLR 2020
Example 1:
#
source:
P #0
reference:
cbs news magazine 60 minutes, hoffman softly sighed that in such period he would truly do anything to get drugs and alcohol. in an interview with the cbs news magazine "60 minutes", hoffmann sighed that those days were really unscrupulous in getting drugs and alcohol.
in an interview
transformer:
reranker:
in an interview with the cbs news magazine "60 minutes", hoffmann sighed that at that time in order to obtain drugs and alcohol, he was really unscrupulous,
Example 2:
source:
S S THE T. ME
f H 1
reference:
the meantime, more than 10 chinese personnel working in same place with these people have been called back to karachi. at present they are emotionally
stabilized.
transformer:
at the same time, more than ten chinese personnel working at the same site have also withdrawn to karachi. their sentiments are now stable.
reranker:
the same time, more than ten chinese personnel working with these people on the same site have also withdrawn to karachi. at present, their sentiments are stable.
Example
source:
O PA china is a world navigation power and it would not be proper to lack a navigation festival when the country has so many other celebrations.
reference:
transformer:
china is a major navigation country in the world. it is not appropriate to have no maritime festival during the many holidays.
reranker:
china is major maritime country in the world, and it is not appropriate to have no maritime festival during the many festivals.
Example 4:
# ft 9E?
source:
context:
however, legislator yeung, i wish to tell you what am doing today is to ensure every matter can proceed smoothly after the political review starts. therefore, we have to solve some basic problems first and this is different thing all together. what are the basic problems?
reference:
transformer:
what is the basic problem?
reranker:
what are the basic questions?
Example 5:
source:
# TX
context:
sword of justice: prospects for 2006
reference:
author: sword of justice
transformer:
author: the sword of righteousness
reranker:
author: the sword of justice
Table 6: Example outputs from the Document Transformer (proposal model) and our doc-reranker.
13

---

Under review as a conference paper at ICLR 2020
Example 6:
T , М
source:
context:
the criminal used submachine gun to fire a barrage of shots, and three engineers died unfortunately.
reference:
the same time, in china verified the identities of the dead within the shortest possible time.
sent reranker:
at the same time, we spent the shortest time in china to verify the identity of the deceased.
doc reranker:
at the same time, we spent the shortest time china to verify identities of the deceased.
Example 7:
source:
context:
when the plane carrying the three survivors and 11 other personnel arrived in Hefei, people waiting at the airport heaved a long sigh of relief. after the inci- dent occurred, it made proper arrangements for them.
reference:
they will also be escorted home safely. now they have send it home safely.
sent reranker:
doc reranker:
now they want to send them safely their homes.
Example 8:
AA T i [ #
source:
context:
traffic accident occurred the 58 kilometer point of the beijing-harbin high- way, with a spill from an tanker leading to the closure of a section the high- way. it was learned that the oil tanker contained waste oil from charcoal pro- duction...
reference:
the section of the highway from harbin shuangcheng closed, with many vehicles detoured.
sent reranker:
part of the roads heading towards shuangcheng in harbin are closed, and many vehicles are bypassing.
doc reranker:
part of the road from harbin to shuangcheng was closed, and many vehicles were bypassing.
Example 9:
source:
context:
with regard coalmine safety this year, saws will effectively carry out the following three tasks: -continue to effectively tackle the tough issue of controlling methane...
reference:
continue effectively tackle the tough issue of restructuring and shutting down. we should continue make a success the rectification and closure battle. continue to fight the battle of rectification and closure.
sent reranker:
doc reranker:
Example 10:
source:
context:
first, such abuse of "quota" restricts the thorough implementation of world trade organization's free trade principle. tone about "free trade". on the other hand, it re-establishes trade barriers and stabs your back at will with "quotas". does it appear too arbitrary and unfair? second, "quota" limits the nice growth trend in sino-america trade relation. second, the "restriction" restricts the good development momentum of sino-us eco- nomic and trade relations.
one hand, is talking in high-sounding
reference:
sent reranker:
doc reranker:
second, the "quota" restricts the good development momentum of sino-us eco- nomic and trade relations.
Table 7: Example outputs from the sent-reranker and the doc-reranker.
14


Reviews:
[{"review_id": "r1lfMoQ_YH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "** Paper summary **\nIn this paper, the authors propose a new re-ranking mechanism leveraging document-level information. Let X and Y denote two languages for ease of reference. The authors focus on X->Y translation and Y->X is a model used for re-ranking. Specifically,\n(1)\tTwo translation models X->Y and Y->X are trained, where X->Y is a document Transformer and Y->X is a sentence transformer.\n(2)\tTrain a language model P(Y) on document-level corpus (rather than sentence-level LM).\n(3)\tGiven a document with $I$ sentences (x^1, …, x^I), translate each source sentence $x^i$ to K candidates.\n(4)\tUsing beam search guided by Eqn.(4) to find optimal translation paths, which is a combination of X->Y translation, Y->X translation, document-level language model and the number of words.\nThe authors work on NIST Chinese-to-English translation and WMT’19 Zh->En translation to verify the proposed algorithm.\n\n** Novelty **\nThe novelty is limited. Compared to the paper “the Neural Noisy Channel” (Yu et. al, 2017), the authors use document Transformer and document-level LM for re-ranking, which is of limited novelty. \n\n** Details **\n1.\tSome baselines are missing from this paper: (A) dual inference baseline [ref1]; (B) X->Y is sentence-level transformer and LM is sentence-level LM, i.e., (Yu et. al, 2017), where P(Y|X) and P(X|Y) are sentence-level translation models.\n2.\tIn Table 1, the improvement of doc-reranker is not very significant compared to sent-reranker, ranging from 0.21 to 0.66. \n3.  In Table 4, “Channel + LM” and \"Proposal + Channel + LM\" achieved almost the same results. Does it mean that the \"proposal\" component do not work?\n4.\tMany models are used in this framework. I am not sure whether simple re-ranking or ensemble can outperform this baseline, e.g., 2 Zh->En + 1 En->Zh\n\n[ref1] Dual Inference for Machine Learning, Yingce Xia, Jiang Bian, Tao Qin, Nenghai Yu, Tie-Yan Liu, IJCAI’17", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "Sk97bk0tS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a simple approach for document-level machine translation. The idea is to use a language model on the target side and a reverse translation model to choose the best document-level translation. This is theoretically justified by Bayes’ rule and the assumption that the sentences are conditionally independent. The authors implement this idea using a reranking model that rescores 50 candidate translations generated by a standard Transformer model for forward translation. \n\nThis is interesting work and the experimental results demonstrate the effectiveness of the approach. However, I am concerned about the (missing) comparison between the proposed approach and the approach that combines backtranslation and a document-level translator (e.g.  Doc-transformer). It seems to me that one could backtranslate a large monolingual corpus and use the resulting parallel documents as additional training data for a document-level translation model. How does the proposed approach compare to such a backtranslation approach?\n\nAnother concern is the speed of translation. It seems to me that the computational cost required for generating 50 candidates and reranking them is quite high. I would like to see some experimental results on the actual speed of translation. The aforementioned backtranslation approach should not have this problem, which also makes me unsure about the usefulness of the proposed approach in practice.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJlp8ls19B", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe paper describes a noisy channel approach for document-level translation, which does not rely on parallel documents to train. The approach relies on a sentence-level translation model (from target-to-source languages) and a document level language model (on target language), each is trained separately. For decoding, the paper relies on another proposal model (i.e., a sentence level translation model from source to target) and performs beam-search weighted by a linear combination of the scores of all three models. Experiments show strong results on two standard translation benchmarks.\n\nComments:\n-  The proposed approach is strongly based on the neural noisy channel model of Yu et al. 2017 but mainly extends it to context aware translation. While the paper is referenced, I believe more emphasis should be put on the differences of the proposed approach\n-  It seems that the Document Transformer uses parallel-documents to train, so I am wondering if you can still claim that your approach does not require parallel documents.\n-  In general, I think the paper is well written and results are compelling.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose using a noisy channel formulation which allows them to combine a sentence level target-source translation model with a language model trained over target side document-level information. They use reranking of a 50-best list generated by a standard Transformer model for forward translation and show reasonably strong results.  The reviewers were concerned about the efficiency of this approach and the limited novelty as compared to the sentence-level noisy channel research Yu et al. 2017. The authors responded in depth, adding results with another baseline which includes backtranslated data. I feel that although this paper is interesting, it is not compelling enough for inclusion in ICLR. 
Decision: Reject