Title: PUTTING MACHINE TRANSLATION IN CONTEXT WITH THE NOISY CHANNEL MODEL

Abstract: We show that Bayes' rule provides a compelling mechanism for controlling unconditional document language models, using the long-standing challenge of effectively leveraging document context in machine translation. In our formulation, we estimate the probability of a candidate translation as the product of the unconditional probability of the candidate output document and the "reverse translation probability" of translating the candidate output back into the input source language document-the so-called "noisy channel" decomposition. A particular advantage of our model is that it requires only parallel sentences to train, rather than parallel documents, which are not always available. Using a new beam search reranking approximation to solve the decoding problem, we find that document language models outperform language models that assume independence between sentences, and that using either a document or sentence language model outperforms comparable models that directly estimate the translation probability. We obtain the best-published results on the NIST Chinese-English translation task, a standard task for evaluating document translation. Our model also outperforms the benchmark Transformer model by approximately 2.5 BLEU on the WMT19 Chinese-English translation task.

1. INTRODUCTION:

There have been many recent demonstrations that neural language models based on Transformers (Vaswani et al., 2017; Dai et al., 2019) are capable of learning to generate remarkably coherent documents with few (Zellers et al., 2019) or no (Radford et al., 2019) conditioning variables. Although large unconditional language models are frequently used to provide representations for natural language understanding applications (Devlin et al., 2019; Yang et al., 2019; Peters et al., 2018) , how to use them for conditional generation applications remains an open question.

In this paper, we employ Bayes' rule to use powerful unconditional document language models to improve a conditional task: machine translation. The application of Bayes' rule to transform the translation modeling problem p(y | x), where y is the target language, and x is the source language, has a long tradition and was the dominant paradigm in speech and language processing for many years (Brown et al., 1993) . In this context it is known as the noisy channel (Shannon & Weaver, 1949) , by analogy to an information theoretic conception of Bayes' rule as a model for reasoning about messages being sent across an unreliable communication medium. In the noisy channel decomposition ( §2), rather than directly modeling the conditional distribution, we rewrite it as p(y | x) ∝ p(y)×p(x | y). More importantly, this changes the learning problem from estimating a single complex conditional distribution to learning two different distributions: a language model p(y), which provides unconditional estimates of the output (in this paper, documents); and p(x | y), which provides the probability of translating a candidate output y into the (observed) source x.

As we will discuss below, although the problems of estimating p(y | x) and p(x | y) are similar, independence assumptions made in p(x | y) are less statistically costly than they might otherwise be since, at test time, we will be conditioning on x and reasoning about a posterior distribution over y, which will be jointly dependent on all (conditionally independent) parts of x. This statistical fact-which is the same trick that gives naïve Bayes classifiers their expressiveness and ease of estimation-permits us to assume independence between sentence translations in the reverse translation model, and therefore to use parallel sentences (rather than parallel documents) to train it. In the posterior, we thus have an implicit estimate of a document-level translation system, even though we made no use of parallel documents. This is particularly useful since parallel sentences are much more readily available than parallel documents. A second benefit of our approach is that the unconditional language model can be estimated from nonparallel data, which exist in vast quantities. The benefits of strong language models estimated from nonparallel data has long been recognized in translation (Brants et al., 2007) , and more recently in noisy channel approaches to translation based on neural network component distributions (Yu et al., 2017; Yee et al., 2019; Ng et al., 2019) .

Although the noisy channel model is ideal for exploiting the data resources that naturally exist in the world (large corpora of parallel but independent sentences, and large corpora of monolingual documents), we are faced with a much harder decoding problem ( §3). To address this problem, we propose a new beam-search algorithm, exploiting the fact that our document language model operates left-to-right, and our reverse translation model treats sentences independently. The search is guided by a proposal distribution that provides candidate continuations of a document prefix, and these are reranked according to the posterior distribution. In particular, we compare two proposal models: one based on estimates of independent sentence translations (Vaswani et al., 2017) and one that conditions on the source document context (Zhang et al., 2018) .

To explore the performance of our proposed model, we focus on Chinese-English translation, following a series of papers on document translation (Zhang et al., 2018; Werlen et al., 2018; Tu et al., 2018; Xiong et al., 2019) . Although in general it is unreasonable to expect that independent translations of sentences would lead to coherent translations of documents between any pair of languages, the task of translating Chinese into English poses some particularly acute challenges. As Chinese is less morphosyntatically complex than English, and the relevant clues for predicting, e.g., what tense a verb should be in, or whether a noun is singular or plural, may be spread throughout a document, it is crucial that extra sentential context is used.

Our experiments ( §4) explore: (1) different approaches to reranking, (2) different independence assumptions when modelling documents (i.e., whether sentences are generated independently or not), (3) different amounts of language modelling data, and (4) different proposal models. Briefly summarized, we find that document-context language models significantly improve the translation quality obtained with our system, both in terms of BLEU scores, and in terms of a human evaluation. Targeted error analysis demonstrates the document prior is capable of enforcing consistency of tense and number and lexical choice across documents.

2. MODEL DESCRIPTION:

We define x = (x1 , x 2 , . . . , x I ) as the source document with I sentences, and similarly, y = (y 1 , y 2 , . . . , y J ) as the target document with J sentences. During the (human) translation process, translators may split or recombine sentences, but we will assume that I = J. 1 Let x i = (x i 1 , x i 2 , . . . , x i M ) represent the ith sentence in the document, consisting of M words; likewise y i = (y i 1 , y i 2 , . . . , y i N ) denote the ith sentence in the target document, containing N words. The translation of a document x is determined by finding the document ŷ, where p(ŷ | x) is optimal.

ŷ = arg max y p(y | x).

(1)

Instead of modelling the probability p(y | x) directly, we factorize it using Bayes' rule:

EQUATION

)

We further assume that sentences are independently translated, and that the sentences are generated by a left-to-right factorization according to the chain rule. Therefore, we have

EQUATION

x 1

< l a t e x i t s h a _ b a s e = " h W n F q q i C c f A

G P + z e Z E y A T n R o = " > A A A B + X i c b V D N S g M x G P y / t X t + r R S A I n s p u F f R Y O K x g m F d i Z b N q G Z p M l y R b L j f x k E R r J N / G b L s H b R I G W a + j w m T D j T x v O + n d L a + s b m V n m s r O t / g H h t U w V o S i u V Q P I d a U M F b h h l O H x J F c R x y g n H N n f m V C l m R T Z p r Q I M Z D w Q a M Y G O l v u v Q s k j P Y t l T N H v + W / V q h x o l f g F q U K B Z t / k W S p D E V h n C s d d f E h N k W B l G O J V e q m m C S Z j P K R d S w W O q Q y e f I Z O r N K h A Z S S M M m q u / N z I c z y c n Y y x G e l l L x f / q p G V w H G R N J a q g g i c G K U d G o r w G F D F F i e F T S z B R z G Z F Z I Q V J s a W V b E l + M t f X i X t e s / q N X v L q u N e l F H G U g F M B h y t o w C o Q U E J v A M r / D m Z M L + L E Z L T r F z D H / g f P A s u T Q = = < / l a t e x i t > x 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " t U a f i f D x w m r I 0 Y Z J P C s z V w m G T 6 M = " > A A A B + X i c b V B P S 8 M w H P 1 1 / p v z X 9 W j l + I Q P I 2 2 C n o c e P E 4 w W 3 C V k e a p l t Y m p Q k H Y 6 y b + L F g y J e / S b e / D a m W w + 6 + S D k 8 d 7 v R 1 5 e m D K q t O t + W 5 W 1 9 Y 3 N r e p 2 b W d 3 b / / A P j z q K J F J T N p Y M C E f Q q Q I o 5 y 0 N d W M P K S S o C R k p B u O b w q / O y F S U c H v 9 T Q l Q Y K G n M Y U I 2 2 k g W 3 3 Q 8 E i N U 3 M l T / N H v 2 B X X c b 7 h z O K v F K U o c S r Y H 9 1 Y 8 E z h L C N W Z I q Z 7 n p j r I k d Q U M z K r 9 T N F U o T H a E h 6 h n K U E B X k 8 + Q z 5 8 w o k R M L a Q 7 X z l z 9 v Z G j R B X h z G S C 9 E g t e 4 X 4 n 9 f L d H w d 5 J S n m S Y c L x 6 K M + Z o 4 R Q 1 O B G V B G s 2 N Q R h S U 1 W B 4 + Q R F i b s m q m B G / 5 y 6 u k 4 z e 8 i 4 Z / d 1 l v + m U d V T i B U z g H D 6 6 g C b f Q g j Z g m M A z v M K b l V s v 1 r v 1 s R i t W O X O M f y B 9 f k D + E + T 0 g = = < / l a t e x i t >

x 3

< l a t e x i t s h a 1 _ b a s e 6 4 = " m 1 L W 7 u e C 3 g p / S t t A H U i S c r z 0 m D g = " > A

A A B + X i c b V C 7 T s M w F L 0 p r 1 J e A U a W i A q J q U p a J B g r s T A W i T 6 k N l S O 4 7 R W H T u y n Y o q 6 p + w M I A Q K 3 / C x t / g t B m g 5 U i W j 8 6 5 V z 4 + Q c K o 0 q 7 7 b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 M j + / i k o 0 Q q M W l j w Y T s B U g R R j l p a 6 o Z 6 S W S o D h g p B t M b n O / O y V S U c E f 9 C w h f o x G n E Y U I 2 2 k o W 0 P A s F C N Y v N l T 3 N H x t D u + r W 3 A W c d e I V p A o F W k P 7 a x A K n M a E a 8 y Q U n 3 P T b S f I a k p Z m R e G a S K J A h P 0 I j 0 D e U o J s r P F s n n z o V R Q i c S 0 h y u n Y X 6 e y N D s c r D m c k Y 6 b F a 9 X L x P 6 + f 6 u j G z y h P U k 0 4 X j 4 U p c z R w s l r c E I q C d Z s Z g j C k p q s D h 4 j i b A 2 Z V V M C d 7 q l 9 d J p 1 7 z G r X 6 / V W 1 W S / q K M M Z n M M l e H A N T b i D F r Q B w

x S e 4 R X e r M x 6 s d 6 t j + V o y S p 2 T u E P r M 8 f + d O T 0 w = = < / l a t e x i t >

x 4

< l a t e x i t s h a 1 _ b a s e 6 4 = " T h

i u A r a D T D C E + Y h a + h z B 7 w a p n H o = " > A A A B + X i c b V C 7 T s M w F L 0 p r 1 J e A U Y W i w q J q U p K J R g r s T A W i T 6 k N l S O 4 7 R W H S e y n Y o q 6 p + w M I A Q K 3 / C x t / g t B m g 5 U i W j 8 6 5 V z 4 + f s K Z 0 o 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 M j + / i k o + J U E t o m M Y 9 l z 8 e K c i Z o W z P N a S + R F E c + p 1 1 / c p v 7 3 S m V i s X i Q c 8 S 6 k V 4 J F j I C N Z G G t r 2 w I 9 5 o G a R u b K n + W N j a F e d m r M A W i d u Q a p Q o D W 0 v w Z B T N K I C k 0 4 V q r v O o n 2 M i w 1 I 5 z O K 4 N U 0 Q S T C R 7 R v q E C R 1 R 5 2 S L 5 H F 0 Y J U B h L M 0 R G i 3 U 3 x s Z j l Q e z k x G W I / V q p e L / 3 n 9 V I c 3 X s Z E k m o q y P K h M O V I x y i v A Q V M U q L 5 z B B M J D N Z E R l j i Y k 2 Z V V M C e 7 q l 9 d J p 1 5 z r 2 r 1 + 0 a 1 W S / q K M M Z n M M l u H A N T b i D F r S B w B S e 4

R X e r M x 6 s d 6 t j + V o y S p 2 T u E P r M 8 f + 1 e T 1 A = = < / l a t e x i t >

x 1

< l a t e x i t s h a 1 _ b a s e 6 4 = " h W n F q q i C c f A

G P + z e Z 1 1 E y A T 4 n R o = " > A A A B + X i c b V D N S g M x G P y 2 / t X 6 t + r R S 7 A I n s p u F f R Y 8 O K x g m 2 F d i 3 Z b N q G Z p M l y R b L 0 j f x 4 k E R r 7 6 J N 9 / G b L s H b R 0 I G W a + j 0 w m T D j T x v O + n d L a + s b m V n m 7 s r O 7 t 3 / g H h 6 1 t U w V o S 0 i u V Q P I d a U M 0 F b h h l O H x J F c R x y 2 g n H N 7 n f m V C l m R T 3 Z p r Q I M Z D w Q a M Y G O l v u v 2 Q s k j P Y 3 t l T 3 N H v 2 + W / V q 3 h x o l f g F q U K B Z t / 9 6 k W S p D E V h n C s d d f 3 E h N k W B l G O J 1 V e q m m C S Z j P K R d S w W O q Q 6 y e f I Z O r N K h A Z S 2 S M M m q u / N z I c 6 z y c n Y y x G e l l L x f / 8 7 q p G V w H G R N J a q g g i 4 c G K U d G o r w G F D F F i e F T S z B R z G Z F Z I Q V J s a W V b E l + M t f X i X t e s 2 / q N X v L q u N e l F H G U 7 g F M 7 B h y t o w C 0 0 o Q U E J v A M r / D m Z M 6 L 8 + 5 8 L E Z L T r F z D H / g f P 4 A 9 s u T 0 Q = = < / l a t e x i t > x 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " t U a f i f D x w m r I 0 Y Z J P C s z V w m G T 6 M = " > A A A B + X i c b V B P S 8 M w H P 1 1 / p v z X 9 W j l + I Q P I 2 2 C n o c e P E 4 w W 3 C V k e a p l t Y m p Q k H Y 6 y b + L F g y J e / S b e / D a m W w + 6 + S D k 8 d 7 v R 1 5 e m D K q t O t + W 5 W 1 9 Y 3 N r e p 2 b W d 3 b / / A P j z q K J F J T N p Y M C E f Q q Q I o 5 y 0 N d W M P K S S o C R k p B u O b w q / O y F S U c H v 9 T Q l Q Y K G n M Y U I 2 2 k g W 3 3 Q 8 E i N U 3 M l T / N H v 2 B X X c b 7 h z O K v F K U o c S r Y H 9 1 Y 8 E z h L C N W Z I q Z 7 n p j r I k d Q U M z K r 9 T N F U o T H a E h 6 h n K U E B X k 8 + Q z 5 8 w o k R M L a Q 7 X z l z 9 v Z G j R B X h z G S C 9 E g t e 4 X 4 n 9 f L d H w d 5 J S n m S Y c L x 6 K M + Z o 4 R Q 1 O B G V B G s 2 N Q R h S U 1 W B 4 + Q R F i b s m q m B G / 5 y 6 u k 4 z e 8 i 4 Z / d 1 l v + m U d V T i B U z g H D 6 6 g C b f Q g j Z g m M A z v M K b l V s v 1 r v 1 s R i t W O X O M f y B 9 f k D + E + T 0 g = = < / l a t e x i t >

x 3

< l a t e x i t s h a 1 _ b a s e 6 4 = " m 1 L W 7 u e C 3 g p / S t t A H U i S c r z 0 m D g = " > A A A B + X i c b V C 7 T s M w F L 0 p r 1 J e A U a W i A q J q U p a J B g r s T A W i T 6 k N l S O 4 7 R W H T u y n Y o q 6 p + w M I A Q K 3 / C x t / g t B m g 5 U i W j 8 6 5 V z 4 + Q c K o 0 q 7 7 b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 M j

+ / i k o 0 Q q M W l j w Y T s B U g R R j l p a 6 o Z 6 S W S o D h g p B t M b n O / O y V S U c E f 9 C w h f o x G n E Y U I 2 2 k o W 0 P A s F C N Y v N l T 3 N H x t D u + r W 3 A W c d e I V p A o F W k P 7 a x A K n M a E a 8 y Q U n 3 P T b S f I a k p Z m R e G a S K J A h P 0 I j 0 D e U o J s r P F s n n z o V R Q i c S 0 h y u n Y X 6 e y N D s c r D m c k Y 6 b F a 9 X L x P 6 + f 6 u j G z y h P U k 0 4 X j 4 U p c z R w s l r c E I q C d Z s Z g j C k p q s D h 4 j i b A 2 Z V V M C d 7 q l 9 d J p 1 7 z G r X 6 / V W 1 W S / q K M M Z n M M l e H A N T b i D F r Q B w

x S e 4 R X e r M x 6 s d 6 t j + V o y S p 2 T u E P r M 8 f + d O T 0 w = = < / l a t e x i t >

x 4

< l a t e x i t s h a 1 _ b a s e 6 4 = " T h i u A r a D T D C E + Y h a + h z B 7 w a p n H o = " > A A A B + X i c b V C 7 T s M w F L 0 p r 1 J e A U Y W i w q J q U p K J R g r s T A W i T 6 k N l S O 4 7 R W H S e y n Y o q 6 p + w M I A Q K 3 / C x t / g t B m g 5 U i W j 8 6 5 V z 4 + f s K Z 0 o 7 z b Z U 2 N r e 2 d 8 q 7 l b 3 9 g 8 M j

+ / i k o + J U E t o m M Y 9 l z 8 e K c i Z o W z P N a S + R F E c + p 1 1 / c p v 7 3 S m V i s X i Q c 8 S 6 k V 4 J F j I C N Z G G t r 2 w I 9 5 o G a R u b K n + W N j a F e d m r M A W i d u Q a p Q o D W 0 v w Z B T N K I C k 0 4 V q r v O o n 2 M i w 1 I 5 z O K 4 N U 0 Q S T C R 7 R v q E C R 1 R 5 2 S L 5 H F 0 Y J U B h L M 0 R G i 3 U 3 x s Z j l Q e z k x G W I / V q p e L / 3 n 9 V I c 3 X s Z E k m o q y P K h M O V I x y i v A Q V M U q L 5 z B B M J D N Z E R l j i Y k 2 Z V V

M C e 7 q l 9 d J p 1 5 z r 2 r 1 + 0 a 1 W S / q K M M Z n M M l u H A N T b i D F r S B w B S e 4 R X e r M x 6 s d 6 t j + V o y S p 2 T u E P r M 8 f + 1 e T 1 A = = < / l a t e x i t > y 1

< l a t e x i t s h a _ b a s e = " + F

U j h F t z L K Q k b b C h d n c i o H k = " > A A A B + X i c b V D L S s N A F L p r p f U Z d u B o v g q i R V G X B j c s K g F t L J P J p B m Y S Z S S G E / o k b F q U / c + T d O i y c A w h P u Z c c P + F M a c f t i o b m v b O X d t + w e G R f X z S V X E q C e Q m M e y N F O R O o n m t J I i i O f / v S v o x K x W L x q L O E e h E e C x Y y g r W R R r Y G M e q C w y V N n y R X X c a z g J o n b g l q U O J s j + G g Y x S S M q N O F Y q Y H r J N r L s d S M c D q v D V N F E y m e E w H h g o c U e X l i + R z d G G U A I W x N E d o t F B / b + Q U k U M x l h P V G r X i H + w S H d O R N J q q k g y f C l C M d o I G F D B J i e a Z I Z h I Z r I i M s E S E K q p k S N U v r N u s + F e N Z o P / V W s y j C m d w D p f g w g B a A E C M i G V i z c u v F e

r c + l q M V q w h T + w P n A + F K T g = = < / l a t e x i t > y 2

< l a t e x i t s h a 1 _ b a s e 6 4 = " d P 3

B b I V q R V T L l b 3 4 m 7 u B 5 k o 9 P F Q = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i R V 0 G X B j c s K 9 g F t L J P J p B 0 6 m Y S Z S S G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c A w h 3 P u Z c 4 c P + F M a c f 5 t i o b m 1 v b O 9 X d 2 t 7 + w e G R f X z S V X E q C e 2 Q m M e y 7 2 N F O R O 0 o 5 n m t J 9 I i i O f 0 5 4 / v S v 8 3 o x K x W L x q L O E e h E e C x Y y g r W R R r Y 9 9 G M e q C w y V 5 7 N n 5 o j u + 4 0 n A X Q O n F L U o c S 7 Z H 9 N Q x i k k Z U a M K x U g P X S b S X Y 6 k Z 4 X R e G 6 a K J p h M 8 Z g O D B U 4 o s r L F 8 n n 6 M I o A Q p j a Y 7 Q a K H + 3 s h x p I p w Z j L C e q J W v U L 8 z x u k O r z 1 c i a S V F N B l g + F K U c 6 R k U N K G C S E s 0 z Q z C R z G R F Z I I l J t q U V T M l u K t f X i f d Z s O 9 a j Q f r u u t Z l l H F c 7 g H C 7 B h R t o w T 2 0 o Q M E Z v A M r / B m 5 d a L 9 W 5 9 L E c r V r l z C n 9 g f f 4 A + d a T 0 w = = < / l a t e x i t > y 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " T o f i a n d 4 u W 2 + q r Y B 1 w l D a c 4 p Z L 8 = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B F c l a Q V d F l w 4 7 K C f U A b y 2 Q y a Y d O J m F m U g i h f + L G h S J u / R N 3 / o 2 T N g t t P T D M 4 Z x 7 m T P H T z h T 2 n G + r Y 3 N r e 2 d 3 c p e d f / g 8 O j Y P j n t q j i V h H Z I z G P Z 9 7 G i n A n a 0 U x z 2 k 8 k x Z H P a c + f 3 h V + b 0 a l Y r F 4 1 F l C v Q i P B Q s Z w d p I I 9 s e + j E P V B a Z K 8 / m T 8 2 R X X P q z g J o n b g l q U G J 9 s j + G g Y x S S M q N O F Y q Y H r J N r L s d S M c D q v D l N F E 0 y m e E w H h g o c U e X l i + R z d G m U A I W x N E d o t F B / b + Q 4 U k U 4 M x l h P V G r X i H + 5 w 1 S H d 5 6 O R N J q q k g y 4 f C l C M d o 6 I G F D B J i e a Z I Z h I Z r I i M s E S E 2 3 K q p o S 3 N U v r 5 N u o + 4 2 6 4 2 H 6 1 q r U d Z R g X O 4 g C t w 4 Q Z a c A 9 t 6 A C B G T z D K 7 x Z u f V i v V

s f y 9 E N q 9 w 5 g z + w P n 8 A + 1 q T 1 A = = < / l a t e x i t > y 4

< l a t e x i t s h a 1 _ b a s e 6 4

= " O m G 2 O I Z L l Q + L J 8 k 6 v K 0 p q Y 6 4 8 2 w = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B F c l a Q W d F l w 4 7 K C f U A b y 2 Q y a Y d O J m F m U g i h f + L G h S J u / R N 3 / o 2 T N g t t P T D M 4 Z x 7 m T P H T z h T 2 n G + r Y 3 N r e 2 d 3 c p e d f / g 8 O j Y P j n t q j i V h H Z I z G P Z 9 7 G i n A n a 0 U x z 2 k 8 k x Z H P a c + f 3 h V + b 0 a l Y r F 4 1 F l C v Q i P B Q s Z w d p I I 9 s e + j E P V B a Z K 8 / m T 8 2 R X X P q z g J o n b g l q U G J 9 s j + G g Y x S S M q N O F Y q Y H r J N r L s d S M c D q v D l N F E 0 y m e E w H h g o c U e X l i + R z d G m U A I W x N E d o t F B / b + Q 4 U k U 4 M x l h P V G r X i H + 5 w 1 S H d 5 6 O R N J q q k g y 4 f C l C M d o 6 I G F D B J i e a Z I Z h I Z r I i M s E S E 2 3 K q p o S 3 N U v r 5 N u o + 5 e 1 x s P z V q r U d Z R g X O 4 g C t w 4 Q Z a c A 9 t 6 A C B G T z D K 7 x Z u f V i v V s f y 9 E N q 9 w 5 g z + w P n 8 A / N 6 T 1 Q = = < / l a t e x i t > y 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " + F U j h F t z L K Q 0 k b b C h d n c 9 i o H k 8 4 = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i R V 0 G X B j c s K 9 g F t L J P J p B 0 6 m Y S Z S S G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c A w h 3 P u Z c 4 c P + F M a c f 5 t i o b m 1 v b O 9 X d 2 t 7 + w e G R f X z S V X E q C e 2 Q m M e y 7 2 N F O R O 0 o 5 n m t J 9 I i i O f 0 5 4 / v S v 8 3 o x K x W L x q L O E e h E e C x Y y g r W R R r Y 9 9 G M e q C w y V 5 7 N n 9 y R X X c a z g J o n b g l q U O J 9 s j + G g Y x S S M q N O F Y q Y H r J N r L s d S M c D q v D V N F E 0 y m e E w H h g o c U e X l i + R z d G G U A I W x N E d o t F B / b + Q 4 U k U 4 M x l h P V G r X i H + 5 w 1 S H d 5 6 O R N J q q k g y 4 f C l C M d o 6 I G F D B J i e a Z I Z h I Z r I i M s E S E 2 3 K q p k S 3 N U v r 5 N u s + F e N Z o P 1 / V W s 6 y j C m d w D p f g w g 2 0 4 B 7 a 0 A E C M 3 i G V 3 i z c u v F e r c + l q M V q 9 w 5 h T + w P n 8 A + F K T 0 g = = < / l a t e x i t > y 2 < l a t e x i t s h a 1 _ b a s e 6 4 = " d P 3 B b I V q R V T L l b 3 4 m 7 u B 5 k o 9 P F Q = " > A A A B + X i c b V D L S s N A F L 2 p r 1 p f U Z d u B o v g q i R V 0 G X B j c s K 9 g F t L J P J p B 0 6 m Y S Z S S G E / o k b F 4 q 4 9 U / c + T d O 2 i y 0 9 c A w h 3 P u Z c 4 c P + F M a c f 5 t i o b m 1 v b O 9 X d 2 t 7 + w e G R f X z S V X E q C e 2 Q m M e y 7 2 N F O R O 0 o 5 n m t J 9 I i i O f 0 5 4 / v S v 8 3 o x K x W L x q L O E e h E e C x Y y g r W R R r Y 9 9 G M e q C w y V 5 7 N n 5 o j u + 4 0 n A X Q O n F L U o c S 7 Z H 9 N Q x i k k Z U a M K x U g P X S b S X Y 6 k Z 4 X R e G 6 a K J p h M 8 Z g O D B U 4 o s r L F 8 n n 6 M I o A Q p j a Y 7 Q a K H + 3 s h x p I p w Z j L C e q J W v U L 8 z x u k O r z 1 c i a S V F N B l g + F K U c 6 R k U N K G C S E s 0 z Q z C R z G R F Z I I l J t q U V T M l u K t f X i f d Z s O 9 a j Q f r u u t Z l l H F c 7 g H C 7 B h R t o w T 2 0 o Q M E Z v A M r / B m 5 d a L 9 W 5 9 L E c r V r l z C n 9 g f f 4 A + d a T 0 w = = < / l a t e x i t > y 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " T o f i a n d 4 u W 2 + q r Y B 1 w l D a c 4 p Z L 8 = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B F c l a Q V d F l w 4 7 K C f U A b y 2 Q y a Y d O J m F m U g i h f + L G h S J u / R N 3 / o 2 T N g t t P T D M 4 Z x 7 m T P H T z h T 2 n G + r Y 3 N r e 2 d 3 c p e d f / g 8 O j Y P j n t q j i V h H Z I z G P Z 9 7 G i n A n a 0 U x z 2 k 8 k x Z H P a c + f 3 h V + b 0 a l Y r F 4 1 F l C v Q i P B Q s Z w d p I I 9 s e + j E P V B a Z K 8 / m T 8 2 R X X P q z g J o n b g l q U G J 9 s j + G g Y x S S M q N O F Y q Y H r J N r L s d S M c D q v D l N F E 0 y m e E w H h g o c U e X l i + R z d G m U A I W x N E d o t F B / b + Q 4 U k U 4 M x l h P V G r X i H + 5 w 1 S H d 5 6 O R N J q q k g y 4 f C l C M d o 6 I G F D B J i e a Z I Z h I Z r I i M s E S E 2 3 K q p o S 3 N U v r 5 N u o + 4 2 6 4 2 H 6 1 q r U d Z R g X O 4 g C t w 4 Q Z a c A 9 t 6 A C B G T z D K 7 x Z u f V i v V s f y 9 E N q 9 w 5 g z + w P n 8 A + 1 q T 1 A = = < / l a t e x i t > y 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " O m G 2 O I Z L l Q + L J 8 k 6 v K 0 p q Y 6 4 8 2 w = " > A A A B + X i c b V D L S s N A F L 3 x W e s r 6 t L N Y B F c l a Q W d F l w 4 7 K C f U A b y 2 Q y a Y d O J m F m U g i h f + L G h S J u / R N 3 / o 2 T N g t t P T D M 4 Z x 7 m T P H T z h T 2 n G + r Y 3 N r e 2 d 3 c p e d f / g 8 O j Y P j n t q j i V h H Z I z G P Z 9 7 G i n A n a 0 U x z 2 k 8 k x Z H P a c + f 3 h V + b 0 a l Y r F 4 1 F l C v Q i P B Q s Z w d p I I 9 s e + j E P V B a Z K 8 / m T 8 2 R X X P q z g J o n b g l q U G J 9 s j + G g Y x S S M q N O F Y q Y H r J N r L s d S M c D q v D l N F E 0 y m e E w H h g o c U e X l i + R z d G m U A I W x N E d o t F B / b + Q 4 U k U 4 M x l h P V G r X i H + 5 w 1 S H d 5 6 O R N J q q k g y 4 f C l C M d o 6 I G F D B J i e a Z I Z h I Z r I i M s E S E 2 3 K q p o S 3 N U v r 5 N u o + 5 e 1 x s P z V q r U d Z R g X O 4 g C t w 4 Q Z a c A 9 t 6 A C B G T z D K 7 x Z u f V i v V s f y 9 E N q 9 w 5 g z + w P n 8 A / N 6 T 1 Q = = < / l a t e x i t >

Figure 1 : Graphical model showing the factorization of our noisy channel model where y i indicates the ith target language sentence and x i indicates the ith source language sentence. In the prior (left) the target sentences (the y i 's) only influence the corresponding source sentence and therefore can be learned and modeled independently, but at test time (right), when the target is not observed, each y i is dependent on every x i .

where y <i = (y 1 , . . . , y i-1 ) denotes a document prefix consisting of the first i -1 target sentences. Thus conceived, this is a generative model of parallel documents that makes a particular independence assumption; we illustrate the corresponding graphical model on the left side of Figure 1 .

2.1. IMPACT OF THE CONDITIONAL INDEPENDENCE ASSUMPTION:

At first glance, the conditional independence assumption we have made might seem to be the very independence assumption that bedevils traditional neural document translation systems-translations of sentence i appear to be uninfluenced by the translation of any sentence j = i. However, while this is the case during training, this is not the case at test time. Then, we will be conditioning on the x i 's (the source language sentences), and reasoning about the posterior distribution over the "underlying" y i 's. By conditioning on the child variables, conditional dependencies between all y i 's and between each y i and all x i 's are created (Shachter, 1998) . The (in)dependencies that are present in the posterior distribution are shown in the right part of Figure 1 .

Thus, although modelling p(y | x) or p(x | y) would appear to be superficially similar, the statistical impact of making a conditional independence assumption is quite different. This is fortunate, as it makes it straightforward to use parallel sentences, rather than assuming we have parallel documents which are less often available (Voita et al., 2019; Zhang et al., 2018; Maruf et al., 2019, inter alia) .

Finally, since we only need to learn to model the likelihood of sentence translations (rather than document translations), the challenges of guiding the learners to make robust generalizations in direct document translation models (Voita et al., 2019; Zhang et al., 2018; Maruf et al., 2019 , inter alia) are neatly avoided.

2.2. LEARNING:

We can parameterize the channel probability p(x i | y i ) using any sequence-to-sequence model and parameterize the language model p(y i | y <i ) using any language model. It is straightforward to learn our model: we simply optimize the channel model and the language model separately on parallel data and monolingual data, respectively. We remark that it is a significant practical advantage of this parameterization that we can retrain the channel and language models independently, for example if we acquire more monolingual data, or use different language models with the same channel model conditioned on the domain of the source text.

3. DECODING:

Because of the global dependencies in the posterior distribution, decoding in our noisy channel model is computationally complex. On one hand, similarly to the decoding problem faced in standard sequence-to-sequence models, we must search over the space of all possible outputs with a model that makes no Markov assumptions. On the other hand, unlike traditional neural MT models, we have to have a complete guess of y i before we can reliably compute the score of p(x i | y i ) according to our channel model, making traditional greedy and near-greedy algorithms ineffective.

To deal with this issue, we leverage an auxiliary proposal model q(y | x) which approximates the posterior distribution using a direct model, and we use this to focus on our search on promising parts of the output space.

Proposal model

x i < l a t e x i t s h a 1 _ b a s e 6 4 = " d P C l q X 7 0 L Z n y y T

x Z L 7 9 m O Y / Y U 2 s = " > A A A B + X i c b V D N S g M x G P y 2 / t X 6 t + r R S 7 A I n s p u F f R Y 8 O K x g m 2 F d i 3 Z b N q G Z p M l y R b L 0 j f x 4 k E R r 7 6 J N 9 / G b L s H b R 0 I G W a + j 0 w m T D j T x v O + n d L a + s b m V n m 7 s r O 7 t 3 / g H h 6 1 t U w V o S 0 i u V Q P I d a U M 0 F b h h l O H x J F c R x y 2 g n H N 7 n f m V C l m R T 3 Z p r Q I M Z D w Q a M Y G O l v u v 2 Q s k j P Y 3 t l T 3 N H l n f r X o 1 b w 6 0 S v y C V K F A s + 9 + 9 S J J 0 p g K Q z j W u u t 7 i Q k y r A w j n M 4 q v V T T B J M x H t K u p Q L H V A f Z P P k M n V k l Q g O p 7 B E G z d X f G x m O d R 7 O T s b Y j P S y l 4 v / e d 3 U D K 6 D j I k k N V S Q x U O D l C M j U V 4 D i p i i x P C p J Z g o Z r M i M s I K E 2 P L q t g S / O U v r 5 J 2 v e Z f 1 O p 3 l 9 V G v a i j D C d w C u f g w x U 0 4 B a a 0 A I C E 3 i G V 3 h z M u f F e X c

+ F q M l p 9 g 5 h j 9 w P n 8 A S 7 q U C Q = = < / l a t e x i t > y i,1

< l a t e x i t s h a 1 _ b a s e 6 4 = " U s N e O m w 0 J

V G c 3 P U j 1 8 W Z V J n 1 Q u k = " > A A A B / n i c b V D N S 8 M w H E 3 n 1 5 x f V f H k J T g E D z L a K e h x 4 M X j B P c B W x 1 p m m 5 h a V K S V C i l 4 L / i x Y M i X v 0 7 v P n f m G 4 9 6 O a D k M d 7 v x 9 5 e X 7 M q N K O 8 2 1 V V l b X 1 j e q m 7 W t 7 Z 3 d P X v / o K t E I j H p Y M G E 7 P t I E U Y 5 6 W i q G e n H k q D I Z 6 T n T 2 8 K v / d I p K K C 3 + s 0 J l 6 E x p y G F C N t p J F 9 N P Q F C 1 Q a m S t L 8 4 e M n k M 3 H 9 l 1 p + H M A J e J W 5 I 6 K N E e 2 V / D Q O A k I l x j h p Q a u E 6 s v Q x J T T E j e W 2 Y K B I j P E V j M j C U o 4 g o L 5 v F z + G p U Q I Y C m k O 1 3 C m / t 7 I U K S K h G Y y Q n q i F r 1 C / M 8 b J D q 8 9 j L K 4 0 Q T j u c P h Q m D W s C i C x h Q S b B m q S E I S 2 q y Q j x B E m F t G q u Z E t z F L y + T b r P h X j S a d 5 f 1 V r O s o w q O w Q k 4 A y 6 4 A i 1 w C 9 q g A z D I w D N 4 B W / W k / V i v V s f 8 9 G K V e 4 c g j + w P n 8 A Y W W V s Q = = < / l a t e x i t > y i,2 < l a t e x i t s h a 1 _ b a s e 6 4 = " l p E b O 9 6 k V 3 F Z J j Q O P X O 0 o 4 c P k P E = " > A A A B / n i c b V D N S 8 M w H E 3 n 1 5 x f U / H k J T g E D z L a K u h x 4 M X j B P c B W x 1 p m m 5 h a V K S V C i l 4 L / i x Y M i X v 0 7 v P n f m G 4 9 6 O a D k M d 7 v x 9 5 e X 7 M q N K 2 / W 1 V V l b X 1 j e q m 7 W t 7 Z 3 d v f r + Q V e J R G L S w Y I J 2 f e R I o x y 0 t F U M 9 K P J U G R z 0 j P n 9 4 U f u + R S E U F v 9 d p T L w I j T k N K U b a S K P 6 0 d A X L F B p Z K 4 s z R 8 y e g 7 d f F R v 2 E 1 7 B r h M n J I 0 Q I n 2 q P 4 1 D A R O I s I 1 Z k i p g W P H 2 s u Q 1 B Q z k t e G i S I x w l M 0 J g N D O Y q I 8 r J Z / B y e G i W A o Z D m c A 1 n 6 u + N D E W q S G g m I 6 Q n a t E r x P + 8 Q a L D a y + j P E 4 0 4 X j + U J g w q A U s u o A B l Q R r l h q C s K Q m K 8 Q T J B H W p r G a K c F Z / P I y 6 b p N 5 6 L p 3 l 0 2 W m 5 Z R x U c g x N w B h x w B V r g F r R B B 2 C Q g W f w C t 6 s J + v F e r c + 5 q M V q 9 w 5 B H 9 g f f 4 A Y u q V s g = = < / l a t e x i t > y i,3

< l a t e x i t s h a 1 _ b a s e 6 4 = " R T 1 C z P R 8 S s 4 S 1 9 v U 2 O f f + l W 9 I z 4 = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U a 7 C X o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o p + K 9 4 8 a C I V / 8 O b / 4 3 p l s P u v k g 5 P H e 7 0 d e n h 8 z q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2

F U i k Z h 0 s G B C 9 n 2 k C K O c d D T V j P R j S V D k M 9 L z p z e F 3 3 s k U l H B 7 3 U a E y 9 C Y 0 5 D i p E 2 0 s g + H v q C B S q N z J W l + U N G L 2 A z H 9 k 1 p + 7 M A J e J W 5 I a K N E e 2 V / D Q O A k I l x j h p Q a u E 6 s v Q x J T T E j e X W Y K B I j P E V j M j C U o 4 g o L 5 v F z + G Z U Q I Y C m k O 1 3 C m / t 7 I U K S K h G Y y Q n q i F r 1 C / M 8 b J D q 8 9 j L K 4 0 Q T j u c P h Q m D W s C i C x h Q S b B m q S E I S 2 q y Q j x B E m F t G q u a E t z F L y + T b q P u N u u N u 8 t a q 1 H W U Q E n 4 B S c A x d c g R a 4 B W 3 Q A R h k 4 B

m 8 g j f r y X q x 3 q 2 P + e i K V e 4 c g T + w P n 8 A Z G + V s w = = < / l a t e x i t >

y 1,3 < l a t e x i t s h a 1 _ b a s e 6 4 = " R Q t 1 Q a g h 6 I C v g 0 D p m v o k q R U T d R g = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U a 7 C X o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o p + K 9 4 8 a C I V / 8

O b / 4 3 p l s P u v k g 5 P H e 7 0 d e n h 8 z q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2

F U i k Z h 0 s G B C 9 n 2 k C K O c d D T V j P R j S V D k M 9 L z p z e F 3 3 s k U l H B 7 3 U a E y 9 C Y 0 5 D i p E 2 0 s g + H v q C B S q N z J W l + U P m X s B m P r J r T t 2 Z A S 4 T t y Q 1 U K I 9 s r + G g c B J R L j G D C k 1 c J 1 Y e x m S m m J G 8 u o w U S R G e I r G Z G A o R x F R X j a L n 8 M z o w Q w F N I c r u F M / b 2 R o U g V C c 1 k h P R E L X q F + J 8 3 S H R 4 7 W W U x 4 k m H M 8 f C h M G t Y B F F z C g k m D N U k M Q l t R k h X i C J M L a N F Y 1 J b i L X 1 4 m 3 U b d b d Y b d 5 e 1 V q O s o w J O w C k 4 B y 6 4 A i 1 w C 9 q g A z D I w D N 4 B W / W k / V i v V s f 8 9 E V q 9 w 5 A n 9 g f f 4 A D q + V e w = = < / l a t e x i t > y 1,1

< l a t e x i t s h a 1 _ b a s e 6 4 = " 2 9 g 3 m P q W H R P

D / F 2 i m / a 2 9 d Y 2 H 1 4 = " > A A A B / n i c b V D N S 8 M w H E 3 n 1 5 x f V f H k J T g E D z L a K e h x 4 M X j B P c B W x 1 p m m 5 h a V K S V C i l 4 L / i x Y M i X v 0 7 v P n f m G 4 9 6 O a D k M d 7 v x 9 5 e X 7 M q N K O 8 2 1 V V l b X 1 j e q m 7 W t 7 Z 3 d P X v / o K t E I j H p Y M G E 7 P t I E U Y 5 6 W i q G e n H k q D I Z 6 T n T 2 8 K v / d I p K K C 3 + s 0 J l 6 E x p y G F C N t p J F 9 N P Q F C 1 Q a m S t L 8 4 f M P Y d u P r L r T s O Z A S 4 T t y R 1 U K I 9 s r + G g c B J R L j G D C k 1 c J 1 Y e x m S m m J G 8 t o w U S R G e I r G Z G A o R x F R X j a L n 8 N T o w Q w F N I c r u F M / b 2 R o U g V C c 1 k h P R E L X q F + J 8 3 S H R 4 7 W W U x 4 k m H M 8 f C h M G t Y B F F z C g k m D N U k M Q l t R k h X i C J M L a N F Y z J b i L X 1 4 m 3 W b D v W g 0 7 y 7 r r W Z Z R x U c g x N w B l x w B V r g F r R B B 2 C Q g W f w C t 6 s J + v F e r c + 5 q M V q 9 w 5 B H 9 g f f 4 A C 6 W V e Q = = < / l a t e x i t > y 1,2 < l a t e x i t s h a 1 _ b a s e 6 4 = " C D N o D Z s V d 5 w T T t 5 L J Z W 4 h 4 B / a 3 s = " > A A A B / n i c b V D N S 8 M w H E 3 n 1 5 x f U / H k J T g E D z L a K u h x 4 M X j B P c B W x 1 p m m 5 h a V K S V C i l 4 L / i x Y M i X v 0 7 v P n f m G 4 9 6 O a D k M d 7 v x 9 5 e X 7 M q N K 2 / W 1 V V l b X 1 j e q m 7 W t 7 Z 3 d v f r + Q V e J R G L S w Y I J 2 f e R I o x y 0 t F U M 9 K P J U G R z 0 j P n 9 4 U f u + R S E U F v 9 d p T L w I j T k N K U b a S K P 6 0 d A X L F B p Z K 4 s z R 8 y 5 x y 6 + a j e s J v 2 D H C Z O C V p g B L t U f 1 r G A i c R I R r z J B S A 8 e O t Z c h q S l m J K 8 N E 0 V i h K d o T A a G c h Q R 5 W W z + D k 8 N U o A Q y H N 4 R r O 1 N 8 b G Y p U k d B M R k h P 1 K J X i P 9 5 g 0 S H 1 1 5 G e Z x o w v H 8 o T B h U A t Y d A E D K g n W L D U E Y U l N V o g n S C K s T W M 1 U 4 K z + O V l 0 n W b z k X T v b t s t N y y j i o 4 B i f g D D j g C r T A L W i D D s A g A 8 / g F b x Z T 9 a L 9 W 5 9 z E c r V r l z C P 7 A + v w B D S q V e g = = < / l a t e x i t > y 2,2 < l a t e x i t s h a 1 _ b a s e 6 4 = " G q k T V c Y r h 8 8 h K o H h l V J F U B i b J L Q = " > A A A B / n i c b V D N S 8 M w H E 3 n 1 5 x f U / H k J T g E D z L a K u h x 4 M X j B P c B W x 1 p m m 5 h a V K S V C i l 4 L / i x Y M i X v 0 7 v P n f m G 4 9 6 O a D k M d 7 v x 9 5 e X 7 M q N K 2 / W 1 V V l b X 1 j e q m 7 W t 7 Z 3 d v f r + Q V e J R G L S w Y I J 2 f e R I o x y 0 t F U M 9 K P J U G R z 0 j P n 9 4 U f u + R S E U F v 9 d p T L w I j T k N K U b a S K P 6 0 d A X L F B p Z K 4 s z R 8 y 9 x y 6 + a j e s J v 2 D H C Z O C V p g B L t U f 1 r G A i c R I R r z J B S A 8 e O t Z c h q S l m J K 8 N E 0 V i h K d o T A a G c h Q R 5 W W z + D k 8 N U o A Q y H N 4 R r O 1 N 8 b G Y p U k d B M R k h P 1 K J X i P 9 5 g 0 S H 1 1 5 G e Z x o w v H 8 o T B h U A t Y d A E D K g n W L D U E Y U l N V o g n S C K s T W M 1 U 4 K z + O V l 0 n W b z k X T v b t s t N y y j i o 4 B i f g D D j g C r T A L W i D D s A g A 8 / g F b x Z T 9 a L 9 W 5 9 z E c r V r l z C P 7 A + v w B D r K V e w = = < / l a t e x i t > y 2,1

< l a t e x i t s h a 1 _ b a s e 6 4 = " 3 l k 3 8 c / w 2 y a L a h S n g D s d 

b C W e X t U = " > A A A B / n i c b V D N S 8 M w H E 3 n 1 5 x f U / H k J T g E D z L a K u h x 4 M X j B P c B W x 1 p m m 5 h a V K S V C i l 4 L / i x Y M i X v 0 7 v P n f m G 4 9 6 O a D k M d 7 v x 9 5 e X 7 M q N K 2 / W 1 V V l b X 1 j e q m 7 W t 7 Z 3 d v f r + Q V e J R G L S w Y I J 2 f e R I o x y 0 t F U M 9 K P J U G R z 0 j P n 9 4 U f u + R S E U F v 9 d p T L w I j T k N K U b a S K P 6 0 d A X L F B p Z K 4 s z R 8 y 9 x w 6 + a j e s J v 2 D H C Z O C V p g B L t U f 1 r G A i c R I R r z J B S A 8 e O t Z c h q S l m J K 8 N E 0 V i h K d o T A a G c h Q R 5 W W z + D k 8 N U o A Q y H N 4 R r O 1 N 8 b G Y p U k d B M R k h P 1 K J X i P 9 5 g 0 S H 1 1 5 G e Z x o w v H 8 o T B h U A t Y d A E D K g n W L D U E Y U l N V o g n S C K s T W M 1 U 4 K z + O V l 0 n W b z k X T v b t s t N y y j i o 4 B i f g D D j g C r T A L W i D D s A g A 8 /

E i l Z i 0 s W B C 9 g K k C K O c t D X V j P Q S S V A c M N I N J j e F 3 3 0 k U l H B 7 3 W W E D 9 G I 0 4 j i p E 2 0 t A + H g S C h S q L z Z V n 0 4 f c u 4 C N 6 d C u O X V n B r h M 3 J L U Q I n W 0 P 4 a h A K n M e E a M 6 R U 3 3 U S 7 e d I a o o Z m V Y H q S I J w h M 0 I n 1 D O Y q J 8 v N Z / C k 8 M 0 o I I y H N 4 R r O 1 N 8 b O Y p V k d B M x k i P 1 a J X i P 9 5 / V R H 1 3 5 O e Z J q w v H 8 o S h l U A t Y d A F D K g n W L D M E Y U l N V o j H S C K s T W N V U 4 K 7 + O V l 0 v H q b q P u 3 V 3 W m l 5 Z R w W c g F N w D l x w B Z r g F r R A G 2 C Q g 2 f w C t 6 s J + v F e

U i k Z h 0 s G B C 9 n 2 k C K O c d D T V j P R j S V D k M 9 L z p z e F 3 3 s k U l H B 7 3 U a E y 9 C Y 0 5 D i p E 2 0 s g + H v q C B S q N z J W l + U P W v I D N f G T X n L o z A 1 w m b k l q o E R 7 Z H 8 N A 4 G T i H C N G V J q 4 D q x 9 j I k N c W M 5 N V h o k i M 8 B S N y c B Q j i K i v G w W P 4 d n R g l g K K Q 5 X M O Z + n s j Q 5 E q E p r J C O m J W v Q K 8 T 9 v k O j w 2 s s o j x N N O J 4 / F C Y M a g G L L m B A J c G a p Y Y g L K n J C v E E S Y S 1 a a x q S n A X v 7 x M u o 2 6 2 6 w 3 7 i 5 r r U Z Z R w W c g F N w D l x w B V r g F r R B B 2 C Q g W f w C t 6 s J + v F e r c + 5 q M r V r l z B P 7 A + v w B E b + V f Q = = < / l a t e x i t > y 3,2

< l a t e x i t s h a 1 _ b a s e 6 4 = " E j 5 n 8 W L D y i 2 z H D + w j g k 6 6 3 p F 7 / g = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U b b C X o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o Z + K 9 4 8 a C I V / 8 O b / 4 3 p l s P u v k g 5 P H e 7 0 d e X p A w q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2

F E i l Z i 0 s W B C 9 g K k C K O c t D X V j P Q S S V A c M N I N J j e F 3 3 0 k U l H B 7 3 W W E D 9 G I 0 4 j i p E 2 0 t A + H g S C h S q L z Z V n 0 4 e 8 c Q G 9 6 d C u O X V n B r h M 3 J L U Q I n W 0 P 4 a h A K n M e E a M 6 R U 3 3 U S 7 e d I a o o Z m V Y H q S I J w h M 0 I n 1 D O Y q J 8 v N Z / C k 8 M 0 o I I y H N 4 R r O 1 N 8 b O Y p V k d B M x k i P 1 a J X i P 9 5 / V R H 1 3 5 O e Z J q w v H 8 o S h l U A t Y d A F D K g n W L D M E Y U l N V o j H S C K s T W N V U 4 K 7 + O V l 0 v H q b q P u 3 V 3 W m l 5 Z R w W c g F N w D l x w B Z r g F r R A G 2 C Q g 2 f w C t 6 s J + v F e r c + 5 q M r V r l z B P 7 A + v w B E D q V f A = = < / l a t e x i t > y 3,1

< l a t e x i t s h a 1 _ b a s e 6 4 = " b d P A Y 8 9 k 1 w I B v 8 e 4 9 v A 7 n c B A h 5 g = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U a 7 C X o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o p + K 9 4 8 a C I V / 8 O b / 4 3 p l s P u v k g 5 P H e 7 0 d e n h 8 z q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2

F U i k Z h 0 s G B C 9 n 2 k C K O c d D T V j P R j S V D k M 9 L z p z e F 3 3 s k U l H B 7 3 U a E y 9 C Y 0 5 D i p E 2 0 s g + H v q C B S q N z J W l + U P W v I B u P r J r T t 2 Z A S 4 T t y Q 1 U K I 9 s r + G g c B J R L j G D C k 1 c J 1 Y e x m S m m J G 8 u o w U S R G e I r G Z G A o R x F R X j a L n 8 M z o w Q w F N I c r u F M / b 2 R o U g V C c 1 k h P R E L X q F + J 8 3 S H R 4 7 W W U x 4 k m H M 8 f C h M G t Y B F F z C g k m D N U k M Q l t R k h X i C J M L a N F Y 1 J b i L X 1 4 m 3 U b d b d Y b d 5 e 1 V q O s o w J O w C k 4 B y 6 4 A i 1 w C 9 q g A z D I w D N 4 B W / W k / V i v V s f 8 9 E V q 9 w 5 A n 9 g f f 4 A D r W V e w = = < / l a t e x i t > y 4,1

< l a t e x i t s h a 1 _ b a s e 6 4 = " L y 5 C n Y j p K i 3 z T i n a 1 7 O K a C l q h o g = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U Y 7 B 3 o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o p + K 9 4 8 a C I V / 8 O b / 4 3 p l s P u v k g 5 P H e 7 0 d e n h 8 z q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2 

F U i k Z h 0 s G B C 9 n 2 k C K O c d D T V j P R j S V D k M 9 L z p z e F 3 3 s k U l H B 7 3 U a E y 9 C Y 0 5 D i p E 2 0 s g + H v q C B S q N z J W l + U P W v I B u P r J r T t 2 Z A S 4 T t y Q 1 U K I 9 s r + G g c B J R L j G D C k 1 c J 1 Y e x m S m m J G 8 u o w U S R G e I r G Z G A o R x F R X j a L n 8 M z o w Q w F N I c r u F M / b 2 R o U g V C c 1 k h P R E L X q F + J 8 3 S H R 4 7 W W U x 4 k m H M 8 f C h M G t Y B F F z C g k m D N U k M Q l t R k h X i C J M L a N F Y 1 J b i L X 1 4 m 3 U b d v a w 3 7 p q 1 V q O s o w J O w C k 4 B y 6 4 A i 1 w C 9 q g A z D I w D N 4 B W / W k / V i v V s f 8 9 E V q 9 w 5 A n 9 g f f 4 A E D 2 V f A = = <

E i l Z i 0 s W B C 9 g K k C K O c t D X V j P Q S S V A c M N I N J j e F 3 3 0 k U l H B 7 3 W W E D 9 G I 0 4 j i p E 2 0 t A + H g S C h S q L z Z V n 0 4 e 8 c Q G 9 6 d C u O X V n B r h M 3 J L U Q I n W 0 P 4 a h A K n M e E a M 6 R U 3 3 U S 7 e d I a o o Z m V Y H q S I J w h M 0 I n 1 D O Y q J 8 v N Z / C k 8 M 0 o I I y H N 4 R r O 1 N 8 b O Y p V k d B M x k i P 1 a J X i P 9 5 / V R H 1 3 5 O e Z J q w v H 8 o S h l U A t Y d A F D K g n W L D M E Y U l N V o j H S C K s T W N V U 4 K 7 + O V l 0 v H q 7 m X d u 2 v U m l 5 Z R w W c g F N w D l x w B Z r g F r R A G 2 C Q g 2 f w C t 6 s J + v F e r c + 5 q M r V r l z B P 7 A + v w B E c K V f Q = = < / l a t e x i t > y 4,3

< l a t e x i t s h a 1 _ b a s e 6 4 = " 

I x m + 5 R F d b U a w 2 G B M 4 5 S q h Q 5 M m 4 w = " > A A A B / n i c b V D N S 8 M w H E 3 n 1 5 x f V f H k J T g E D z L a b a D H g R e P E 9 w H b H W k a b q F p U l J U q G U g f + K F w + K e P X v 8 O Z / Y 7 r 1 o J s P Q h 7 v / X 7 k 5 f k x o 0 o 7 z r d V W l v f 2 N w q b 1 d 2 d v f 2 D + z D o 6 4 S i c S k g w U T s u 8 j R R j l p K O p Z q Q f S 4 I i n 5 G e P 7 3 J / d 4 j k Y o K f q / T m H g R G n M a U o y 0 k U b 2 y d A X L F B p Z K 4 s n T 1 k z U v Y m I 3 s q l N z 5 o C r x C 1 I F R R o j + y v Y S B w E h G u M U N K D V w n 1 l 6 G p K a Y k V l l m C g S I z x F Y z I w l K O I K C + b x 5 / B c 6 M E M B T S H K 7 h X P 2 9 k a F I 5 Q n N Z I T 0 R C 1 7 u f i f N 0 h 0 e O 1 l l M e J J h w v H g o T B r W A e R c w o J J g z V J D E J b U Z I V 4 g i T C 2 j R W M S W 4 y 1 9 e J d 1 6 z W 3 U 6 n f N a q t e 1 F E G p + A M X A A X X I E W u A V t 0 A E Y Z O A Z v I I

v 5 B + f C o p Z N M M W y y R C S q H V K N g k t s G m 4 E t l O F N A 4 F P o a j m 5 n / + I R K 8 0 Q + m H G K Q U w H k k e c U W O l e 8 + 9 7 J U r n u v N Q V a J n 5 M K 5 G j 0 y l / d f s K y G K V h g m r d 8 b 3 U B B O q D G c C p 6 V u p j G l b E Q H 2 L F U 0 h h 1 M J m f O i V n V u m T K F G 2 p C F z 9 f f E h M Z a j + P Q d s b U D P W y N x P / 8 z q Z i a 6 D C Z d p Z l C y x a I o E 8 Q k Z P Y 3 6 X O F z I i x J Z Q p b m 8 l b E g V Z c a m U 7 I h + M s v r 5 J W z f X P 3 d r d R a V

l i J y 2 p l D z W S 2 D V A 4 K t g D / A u U M = " > A A A B 6 n i c b V B N S 8 N A E J 3 4 W e t X 1 a O X x S L 0 V J J W 0 G P B i 8 e K 9 g P a U D b b T b t 0 s w m 7 E 6 G E / g Q v H h T x 6 i / y 5 r 9 x 2 + a g r Q 8 G H u / N M D M v S K Q w 6 L r f z s b m 1 v b O b m G v u H 9 w e H R c O j l t m z j V j L d Y L G P d D a j h U i j e Q o G S d x P N a R R I 3 g k m t 3 O / 8 8 S 1 E b F 6 x G n C / Y i O l A g F o 2 i l B 7 d a H 5 T K b t V d g K w T L y d l y N E c l L 7 6 w 5 i l E V f I J D W m 5 7 k J + h n V K J j k s 2 I / N T y h b E J H v G e p o h E 3 f r Y 4 d U Y u r T I k Y a x t K S Q L 9 f d E R i N j p l F g O y O K Y 7 P q z c X / v F 6 K 4 Y 2 f C Z W k y B V b L g p T S T A m 8 7 / J U G j O U E 4 t o U w L e y t h Y 6 o p Q 5 t O 0 Y b g r b 6 8 T t q 1 q l e v 1 u 6 v y o 1 K H k c B z u E C K u D B N T T g D p r Q A g Y j e I Z X e H O k 8 + K 8 O x / L 1 g 0 n n z m D P 3 A + f w B Q A I 0 T < / l a t e x i t > 0.1 < l a t e x i t s h a 1 _ b a s e 6 4 = " g g n H 0 D 5 j Z 4 Q 0 M / 6 e R b 3 g k j q s u b Q = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 B U Q Y 8 F L x 4 r 2 g 9 o Q 9 l s N + 3 S z S b s T o R S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M J X C o O d 9 O 4 W N z a 3 t n e J u a W / / 4 P C o f H z S M k m m G W + y R C a 6 E 1 L D p V C 8 i Q I l 7 6 S a 0 z i U v B 2 O b + d + + 4 l r I x L 1 i J O U B z E d K h E J R t F K D 5 7 r 9 8 s V z / U W I O v E z 0 k F c j T 6 5 a / e I G F Z z B U y S Y 3 p + l 6 K w Z R q F E z y W a m X G Z 5 S N q Z D 3 r V U 0 Z i b Y L o 4 d U Y u r D I g U a J t K S Q L 9 f f E l M b G T O L Q d s Y U R 2 b V m 4 v / e d 0 M o 5 t g K l S a I V d s u S j K J M G E z P 8 m A 6 E 5 Q z m x h D I t 7 K 2 E j a i m D G 0 6 J R u C v / r y O m n V X P / S r d 1 f V e r V P I 4 i n M E 5 V M G H a 6 j D H T S g C Q y G 8 A y v 8 O Z I 5 8 V 5 d z 6 W r Q U n n z m F P 3 A + f w B M + I 0 R < / l a t e x i t > q(y 2,1 | x 2 )p(x 2 | y 2,1 )p(y 2,1 | y 1,3 )p(y 1,3 ) < l a t e x i t

M h a U X k 1 Q c p T / H P + S b / 8 Z 0 m z C 3 H Q j 5 + L 7 v n J y c 4 8 W M C m l Z P 5 q + t b 2 z u 1 f Y L 5 Y O D o + O j Z P T t o g S j k k L R y z i X Q 8 J w m h I W p J K R r o x J y j w G O l 4 k + d c 7 3 w Q L m g U v s p p T P o B e g v p i G I k F T U 0 v t 5 N 1 4 u Y L 6 a B u t J p N k i d i p 1 B N 6 A + X F Y + s 4 F z F Z u r 1 L r x r 8 S K e U 7 D T a V z y a 7 A 2 q a U O T 8 0 y l b V m g V c B / Y C l M E i m k P j 2 / U j n A Q k l J g h I X q 2 F c t + i r i k m J G s 6 C a C x A h P 0 B v p K R i i g I h + O h t n B i 8 V 4 8 N R x N U J J Z y x y x k p C k T e o n I G S I 7 F q p a T m 7 R e I k d 3 / Z S G c S J J i O c P j R I G Z Q T z 3 U C f c o I l m y q A M K e q V 4 j H i C M s 1 Q a L a g j 2 6 p f X Q d u p 2 r W q 8 3 J d r p u L c R T A O b g A J r D B L a i D B m i C F s B a S b O 1 e + 1 B N / Q b / V F / m l t 1 b Z F z B v 6 F 3 v g F 7 c f H r A = = < / l a t e x i t >

Phase 1: sample candidate translations Phase 2: beam search

q(y 1,3 | x 1 )p(x 1 | y 1,3 )p(y 1,3 ) = 0.6 < l a t e x i t s h a 1 _ b a s e 6 4 = " L T Y O 1 v D Y d 8 1 L Y W e C 7 s o 1 1 K B l s Q g = " > A A A C Y n i c b V F b S 8 M w G E 2 r 0 z l 1 6 9 y j P g S H 0 I G M d h P 1 R R j 4 4 u M E d 4 H d S N N s C 0 s v J q l Y S v + k b z 7 5 4 g 8 x u y C z 8 4 O Q w z n n S 7 6 c O C G j Q l r W p 6 b v 7 e c O D v N H h e O T 0 2 L J K J 9 1 R R B x T D o 4 Y A H v O 0 g Q R n 3 S k V Q y 0 g 8 5 Q Z 7 D S M 9 Z P C 7 1 3 h v h g g b + i 4 x D M v L Q z K d T i p F U 1 M S I X 8 2 h E z B X x J 7 a k j g d J / Z 1 M 4 V D j 7 p w W 3 l P x 3 Y t N L P U r v H 3 i I x 7 z c N m W o M P 0 K r f T o y q V b d W B X e B v Q F V s K n 2 x P g Y u g G O P O J L z J A Q A 9 s K 5 S h B X F L M S F o Y R o K E C C / Q j A w U 9 J F H x C h Z R Z T C K 8 W 4 c B p w t X w J V + x 2 R 4 I 8 s R x U O T 0 k 5 y K r L c n / t E E k p / e j h P p h J I m P 1 x d N I w Z l A J d 5 Q 5 d y g i W L F U C Y U z U r x H P E E Z b q V w o q B D v 7 5 F 3 Q b d T t Z r 3 x f F N t m Z s 4 8 u A c X A I T 2 O A O t M A T a I M O w O B L y 2 l F r a R 9 6 w W 9 r F f W V l 3 b 9 F T A n 9 I v f g D 1 M 7 Z 5 < / l a t e x i t >

Figure 2 : The decoding process. In Phase 1, the auxiliary proposal model generates candidate translations (3 candidates in the diagram) for each sentence in the document (containing 4 sentences). In Phase 2, beam search is employed to search for the best path from the candidate translations.

Because of the autoregressive factorization of the language model, and the independent sentence translation assumption in the channel model, we can carry out the reranking process using a leftto-right beam search strategy with the aid of our proposal model. Figure 2 illustrates the decoding process. For an input document, we let the proposal model propose K candidate translations for each sentence in the document. We then search for the best path from these candidate translations. We use beam search under the objective defined by the following linear model,

EQUATION

where |y ≤i | denotes the total number of tokens in the partial document y ≤i . Note that Equation 4 is a generalization of Equation 2 in log space-if we set λ 1 = λ 3 = 0 and take the log of Equation 2the two objectives are equivalent. The extra factors-the proposal probability and the length of the output-provide improvements (e.g., by calibrating the expected length of the output), and can be incorporated at no cost in the model; they are widely used in prior work (Yu et al., 2017; Yee et al., 2019; Ng et al., 2019) .

4. EXPERIMENTS:

We evaluate our model on two translation tasks, the NIST Chinese-English task and the WMT19 Chinese-English task. On both tasks, we use the standard parallel training data, and compare our model with a strong Transformer baseline, as well as related models from prior work.

4.1. DATASET DESCRIPTION:

The NIST training data, containing 1.5M sentence pairs, is composed from LDC-distributed news articles and broadcasting scripts. The document-level parallel corpus is a subset of the full training set, including 55K documents with 1.2M sentences. We use the MT06 dataset as validation set and MT03, MT04, MT05, and MT08 as test sets. There are 79 documents and 1649 sentences in the validation set and in total 509 documents and 5146 sentences in the test set. We preprocess the dataset by doing punctuation normalization, tokenization, and lower casing. We use byte pair encoding (Sennrich et al., 2016b) with 32K merges to segment words into sub-word units for both Chinese and English. The evaluation metric is case-insensitive BLEU calculated using multi-bleu.perl, which is consistent with prior work on this task.

The training data for the WMT19 Chinese-English task includes the UN corpus, CWMT, and news commentary. The total number of sentence pairs is 18M after filtering the data by removing duplicate sentences and sentences longer than 250 words. The validation sets that we use in the experiment are newstest2017 and newstest2018, which contains 169 documents, 2001 sentences and 275 documents, 3981 sentences, respectively. The test set is newstest2019 containing 163 documents and 2000 sentences. The dataset is preprocessed by segmenting Chinese sentences and normalizing punctuation, tokenizing, and true casing English sentences. As for NIST, we learn a byte pair encoding (Sennrich et al., 2016b) with 32K merges to segment words into sub-word units for both Chinese and English. The evaluation metric is sacreBLEU (Post, 2018) . (Vaswani et al., 2017) and the Document Transformer (Zhang et al., 2018) on the NIST dataset. The sentreranker is a variation of our model in which sentences in documents are assumed to be independent.

4.2. MODEL CONFIGURATION:

For NIST, we use the Transformer (Vaswani et al., 2017) as the channel model and the Document Transformer (Zhang et al., 2018) as the proposal model. The hyperparameters for training the Transformer is the same as transformer base (Vaswani et al., 2017) , i.e. 512 hidden size, 2048 filter size, 8 attention heads, and 6 layers for both the encoder and decoder. We follow Zhang et al. (2018) 's configuration to train the Document Transformer: context length is set to 2 and all other hyperparameters are the same as transformer base. Both models are optimized using Adam (Kingma & Ba, 2015) with approximately 24K BPE tokens per mini-batch. For the language model, we train the Transformer-XL (Dai et al., 2019) on a combination of the English side of NIST training data as well as three sections of Gigaword: XIN, AFP, APW, resulting in a total of 7.3M documents and 115M sentences. We use an architecture with 24 layers, 16 attention heads, and embeddings of dimension 1024. The input sequences to the language model are encoded into bytes using the byte-level encoder provided by GPT2 (Radford et al., 2019) .

For WMT19, we use the Transformer (Vaswani et al., 2017) as both the channel and proposal model. The hyperparameters for training the Transformer is the same as transformer big (Vaswani et al., 2017) , i.e. 1024 hidden size, 4096 filter size, 16 attention heads, and 6 layers. The model is trained on 8 GPUs with batch size of 4096. The setup for the language model is the same as that of NIST except that the training data is the English side of the parallel training data and Gigaword.

For both tasks, the weights λ are selected using grid search, from [0.8, 1., 1.5, 2., 2.2, 2.5, 3.] for the weights of channel model λ 2 and proposal model λ 1 , and from [0.2, 0.5, 0.8, 1.] for the length penalty λ 3 . The size of the n-best list used in the reranker is set to 50.

4.3. EXPERIMENTAL RESULTS:

Table 1 presents the best result from our model (doc-reranker) in comparison with prior work on the NIST Chinese-English translation task. The first four rows are numbers reported in prior work. Wang et al. (2017) incorporate document context by introducing a hierarchical RNN to an LSTM sequence-to-sequence model. Kuang et al. (2017) use a cache to store previously translated words across sentences, which they then use in sequence-to-sequence models. Zhang et al. (2018) extend the Transformer model with an extra context encoder to capture information from previous source sentences. Apart from prior work, we also compare our doc-reranker with three baselines: the Transformer (Vaswani et al., 2017) and Document Transformer (Zhang et al., 2018) , and the sentence-level reranker (sent-reranker). In the sent-reranker, we assume sentences in the document are independent (formulation ŷ = arg max y |x| i=1 p(x i | y i ) × p(y i )), and therefore we train a sentence-level language model and rerank each sentence independently. This sent-reranker setup is close to the work from Yee et al. (2019) and Ng et al. (2019) with the difference that rather than using a language model trained on documents we use a language model trained on sentences, which is more statistically consistent. well as strong Transformer baselines by a significant margin-approximately 2.5 BLEU on all test sets-achieving new state of the art. Although the gap between the doc-reranker and sent-reranker is smaller, as we will show in §A.1 and §5 that translations generated by doc-reranker are preferred by humans and are more consistent across documents, in line with concerns about the reliability of using BLEU at assessing cross-sentential consistency (Voita et al., 2019) .

To understand the rerankers better, we investigate the effect of different proposal models, different language models, and various numbers of candidates in the n-best list. Table 2 and Figure 3 show that better proposal models and bigger n-best lists lead to consistently better reranking results. This is an appealing behaviour showing that our reranker is able to pick better translations from higher quality and more diverse candidate pools generated by better proposal models and bigger n-best lists. To compare the effect of language models, we train an LSTM language model (Merity et al., 2018a; b) and a Transformer-XL language model on the English side of NIST parallel training data in addition to the Transformer-XL trained on NIST and Gigaword. Table 3 lists the perplexity per word on the NIST validation set for different language models. Given the same training data, the Transformer-XL performs significantly better than the LSTM-based language model, which in turn results in a higher BLEU score from the doc-reranker. By adding more training data, the Transformer-XL language model achieves even lower perplexity and that gives a further boost to the performance of the doc-reranker. Notably, when the strong Transformer-XL language model is incorporated into the doc-reranker, the best weight ratio of the channel and language model is 1 : 1, indicating that the doc-reranker depends heavily on the language model. By contrast, if a weak language model is incorporated, the best ratio is approximately 2 : 1. A further observation is that although a larger-scale language model improves the doc-reranker, it does not help the sent-reranker.

We perform an ablation study to explore what each component of the doc-reranker contributes to the overall performance. Table 4 shows BLEU scores on the NIST validation set for the optimal interpolation of various component models. No gains are observed if the language model is combined with the proposal model (a probabilistically unsound combination, although one that often worked in pre-neural approaches to statistical translation). We find that as we increase the weight of the language model, the results become worse. The interpolation of the proposal model and channel model slightly outperforms the proposal model baseline but considerably underperforms the interpolation of the proposal model, channel model, and the language model. This difference indicates the key roles that the language model plays in the doc-reranker. When the channel model is combined with the language model the performance of the doc-reranker is comparable to that with all three compo- nents included. We conclude from the ablation study that both the channel and language models are indispensable for the doc-reranker, as is consistent with our Bayes' rule account of this model.

Table 5 presents the results of our model together with baselines on the WMT19 Chinese-English translation task. We find that the results follow the same pattern as those on NIST: a better language model leads to better translation results and overall the reranker outperforms the Transformer-big by approximately 2.5 BLEU. The two best systems submitted to the WMT19 Chinese-English translation task are Microsoft Research Asia's system (Xia et al., 2019 ) and Baidu's system (Sun et al., 2019) , both of which employ multiple techniques to improve upon the Transformer big model. Here we mainly compare our results with those from Xia et al. (2019) because we use the same evaluation metric SacreBLEU and the same validation and test sets. Using extra parallel training data and the techniques of masked sequence-to-sequence pretraining (Song et al., 2019) , sequence-level knowledge distillation (Kim & Rush, 2016) , and backtranslation (Edunov et al., 2018) , the best model from (Xia et al., 2019) achieves 30.8, 30.9, and 39.3 on newstest2017, newstest2018, and newstest2019, respectively. Although our best results are lower than this, it is notable that our reranker achieves comparable results to their model trained on 56M parallel data, more than two times more training data than we use. However, our method is orthogonal to these works and can be combined with other techniques to make further improvement.

5. ANALYSIS:

To investigate how the rerankers improve translation quality, we analyze the output from different models: the Document Transformer (Zhang et al., 2018 ) (our proposal model), the sent-reranker, and the doc-reranker. We observe that in general the doc-reranker improves adequacy of translations and can generate more fluent and natural sentences compared to the Document Transformer. More importantly, our doc-reranker shows its superiority over the others in terms of exploiting context, improving consistency of tense, number, and lexical choice across entire articles. Tables 6 and 7 in Appendix A present example output from the aforementioned systems. In Example 1, the pronoun he is omitted in the Chinese sentence. While the Document Transformer misses this pronoun resulting in a translation of completely different meaning, the doc-reranker is able to recover it. Likewise, in Example 7 them is dropped in the source sentence and this pronoun can only be inferred from previous context. Although both rerankers recover some pronoun, only the doc-reranker gets it right, by relying on cross-sentential context. Example 2 is a good example showing that the doc-reranker is better at generating adequate translations than the proposal model: the Document Transformer ignores the phrases with these people and at present, but the doc-reranker covers them.

Chinese does not mark grammatical number, it therefore has to be inferred for the English translations. It is not possible for a sentence-level MT system to capture this information if the relevant context is not from the current sentence. In Example 4 and 6 the plural problems and identities can only be inferred from previous sentences (the immediate previous sentence in Example 4 and the sentence 4-5 sentences away from the current one in Example 6). While neither the Document Transformer nor the sent-reranker makes the right predictions in both examples, the doc-reranker translates correctly, indicating its strength in capturing extra-sentential information. In addition to making inference across sentences, the doc-reranker is also capable of maintaining consistency of tense and lexical choice, as demonstrated in Examples 5, 8, and 10. Furthermore, impressively, it improves the consistency of writing style. To illustrate, in Example 9, the context is that of a list of bullet points starting with continue. The doc-reranker follows in this style by starting the translation with the verb continue. However, the sent-reranker starts the sentence with we should continue.

Although both translations are reasonable, the former one is more natural within the document.

6. RELATED WORK:

Our work is closely related to three lines of research: context-aware neural machine translation, large-scale language models for language understanding, and semi-supervised machine translation.

Recent studies (Tiedemann & Scherrer, 2017; Bawden et al., 2018, inter alia) have shown that exploiting document-level context improves translation performance, and in particular improves lexical consistency and coherence of the translated text. Existing work in the area of context-aware NMT typically adapts the MT system to take additional context as input, either a few previous sentences (Jean et al., 2017; Wang et al., 2017; Tu et al., 2018; Titov et al., 2018; Zhang et al., 2018; Werlen et al., 2018) or the full document (Haffari & Maruf, 2018; Maruf et al., 2019) . The idea of adding monolingual document-level data has been explored in two most recent works (Voita et al., 2019; Junczys-Dowmunt, 2019) . Both use backtranslation (Edunov et al., 2018; Sennrich et al., 2016a) to create synthetic parallel documents as additional training data. We train a largescale language model and use it to refine the consistency between sentences under a noisy channel framework. An advantage of our model over backtranslation is that the language model is portable across domain and language pairs. Investigating ways of adding monolingual data to MT systems is an active research area (Gülc ¸ehre et al., 2015; Cheng et al., 2016, inter alia) . Backtranslation (Edunov et al., 2018; Sennrich et al., 2016a) , originally invented for semi-supervised MT, has been employed as a standard approach for unsupervised MT (Lample et al., 2018b; a; Artetxe et al., 2019; 2018) . Noisy channel decompositions have been a standard approach in statistical machine translation (Brown et al., 1993; Koehn et al., 2007) and recently have been applied to neural models (Yu et al., 2017; Yee et al., 2019; Ng et al., 2019) . Unlike prior work, we adopt noisy channel models for document-level MT where language models are responsible for capturing cross-sentence context.

Large-scale pretrained language models have achieved great success in improving systems in language understanding, leading to state-of-the-art results on a wide range of tasks (Peters et al., 2018; Devlin et al., 2019; Radford et al., 2018; McCann et al., 2017; Yang et al., 2019; Chronopoulou et al., 2019; Lample & Conneau, 2019) . Language generation is another area where pretrained language models have been applied to, with existing work focusing on fine-tuning approaches for repurposing an unconditional language model (Zhang et al., 2019; Edunov et al., 2019; Song et al., 2019; Dong et al., 2019; Ziegler et al., 2019; de Oliveira & Rodrigo, 2019) . In contrast to our work which uses probabilities from language models, that work uses their internal representations.

7. CONCLUSION:

We have presented a noisy channel reranker and empirically validated it on Chinese-English document-level translation tasks. The noisy channel formulation requires only parallel sentences (rather than documents) but we can use abundant monolingual documents to train the language model component. Experiments show that our proposed model considerably improves translation quality-it achieves approximately 2.5 BLEU higher than transformer baselines. Subjective evaluation further confirms that the language model helps enforce consistency of tense, number, and lexical choice across documents.

Example 1: source: 霍夫曼在接受美国哥伦比亚广播公司新闻杂志「六十分钟」访问时轻叹,那 段时期为了得到毒品和酒,真是不择手段。 reference:

in an interview on us cbs news magazine 60 minutes, hoffman softly sighed that in such period he would truly do anything to get drugs and alcohol. transformer: in an interview with the cbs news magazine "60 minutes", hoffmann sighed that those days were really unscrupulous in getting drugs and alcohol. reranker:

in an interview with the cbs news magazine "60 minutes", hoffmann sighed that at that time in order to obtain drugs and alcohol, he was really unscrupulous.

Example 2: source: 与此同时，与这几个人在同一工地工作的十多名中方人员也已经撤回到卡 拉奇，目前他们的情绪稳定。 reference:

in the meantime, more than 10 chinese personnel working in the same place with these people have been called back to karachi. at present they are emotionally stabilized. transformer: at the same time, more than ten chinese personnel working at the same site have also withdrawn to karachi. their sentiments are now stable. reranker:

at the same time, more than ten chinese personnel working with these people on the same site have also withdrawn to karachi. at present, their sentiments are stable.

Example 3: source: 中国是世界航海大国,在众多的节日中没有航海节是不应该的。 reference:

china is a world navigation power and it would not be proper to lack a navigation festival when the country has so many other celebrations. transformer: china is a major navigation country in the world. it is not appropriate to have no maritime festival during the many holidays. reranker:

china is a major maritime country in the world, and it is not appropriate to have no maritime festival during the many festivals. author: sword of justice transformer: author: the sword of righteousness reranker:

author: the sword of justice 

:

Size mismatches are addressed by merging sentences using sentence alignment algorithms.

Figures and Tables:

g F b x Z T 9 a L 9 W 5 9 z E c r V r l z C P 7 A + v w B D S 2 V e g = = < / l a t e x i t > y 2,3 < l a t e x i t s h a 1 _ b a s e 6 4 = " p x v q w x U DV 4 + 8 9 d L A q I l Z e O + M N o k = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U b b C X o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o Z + K 9 4 8 a C I V / 8 O b /4 3 p l s P u v k g 5 P H e 7 0 d e X p A w q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2 F

r c + 5 q M r V r l z B P 7 A + v w B E D e V f A = = < / l a t e x i t > y 3,3 < l a t e x i t s h a 1 _ b a s e 6 4 = "V v d j T W + G o J M C d Z d D d a P / X k R E s j 0 = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U a 7 C X o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o p + K 9 4 8 a C I V / 8 O b /4 3 p l s P u v k g 5 P H e 7 0 d e n h 8 z q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2 F

t e x i t s h a 1 _ b a s e 6 4 = " m D g L S w v x T a 9 b e T 4 n y d uL f V x 8 1 1 s = " > A A A B / n i c b V D N S 8 M w H E 3 9 n P O r K p 6 8 B I f g Q U Z b B 3 o c e P E 4 w X 3 A V k e a p l t Y m p Q k F U o Z + K 9 4 8 a C I V / 8 O b /4 3 p l s P u v k g 5 P H e 7 0 d e X p A w q r T j f F s r q 2 v r G 5 u V r e r 2 z u 7 e v n 1 w 2 F

3 6 8 l 6 s d 6 t j 8 V o y S p 2 j s E f W J 8 / E 0 e V f g = = < / l a t e x i t > 0.6 < l a t e x i t s h a 1 _ b a s e 6 4 = " M O d A s W 6 y j V d e p W h r D U U v A h q c B S Y = " > A A A B 6 n i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B a h p 5 B U U Y 8 F L x 4 r 2 l p o Q 9 l s J + 3 S z S b s b o R S + h O 8 e F D E q 7 / I m / / G b Z u D t j 4 Y e L w 3 w 8 y 8 M B V c G 8 / 7 d g p r 6 x u b W 8 X t 0 s 7 u 3

e z e M o w g m c Q h V 8 u I I 6 3 E I D m s B g A M / w C m + O c F 6 c d + d j 0 V p w 8 p l j + A P n 8 w d U j I 0 W < / l a t e x i t > 0.3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 O

s h a 1 _ b a s e 6 4 = " P o b O U b 4 5 r z + 7 r 5 a t k z u r I Q b 4 3 8 w = " > A A A C j n i c b V F b S 8 M w F E 7 r b W 5 e q j 7 6 E h x C h T H a T l Q U c e D L H i e 4 C 6 z b S N P

Figure 3: Effect of n-best list.

Comparison with prior work on NIST Chinese-English translation task. The evaluation metric is tokenized case-insensitive BLEU. The first four rows are numbers reported in the papers of prior work. The first two baselines are the results that we got by running the Transformer

Table content: <table><tr><td>Method</td><td>Model</td><td colspan="4">MT06 MT03 MT04 MT05 MT08</td></tr><tr><td>(Wang et al., 2017)</td><td>RNNsearch</td><td>37.76</td><td>-</td><td>-</td><td>36.89 27.57</td></tr><tr><td>(Kuang et al., 2017)</td><td>RNNsearch</td><td>-</td><td>-</td><td colspan="2">38.40 32.90 31.86</td></tr><tr><td>(Kuang et al., 2017)</td><td>Transformer</td><td colspan="4">48.14 48.05 47.91 48.53 38.38</td></tr><tr><td colspan="6">(Zhang et al., 2018) Doc-transformer 49.69 50.21 49.73 49.46 39.69</td></tr><tr><td/><td colspan="5">Sent-transformer 47.72 47.21 49.08 46.86 40.18</td></tr><tr><td>Baseline</td><td colspan="5">Doc-transformer 49.79 49.29 50.17 48.99 41.70</td></tr><tr><td/><td>Sent-reranker</td><td colspan="4">51.33 52.23 52.36 51.63 43.63</td></tr><tr><td>This work</td><td>Doc-reranker</td><td colspan="4">51.99 52.77 52.84 51.84 44.17</td></tr></table>

Table1shows that our reranker outperforms previous models as BLEU scores on NIST dev set MT06 from rerankers which are incorporated with various language models. In the language model column X: Y means the language model X is trained on dataset Y. A bigger language model improves the doc-reranker but does not help the sent-reranker.

Table content: <table><tr><td>Proposal model</td><td/><td>Language model</td><td/><td>Sent-reranker Doc-reranker</td></tr><tr><td/><td/><td>LSTM: NIST</td><td/><td>49.92</td><td>50.24</td></tr><tr><td>Sent-transformer</td><td colspan="3">Transformer-XL: NIST</td><td>50.29</td><td>50.56</td></tr><tr><td colspan="4">Transformer-XL: NIST + Gigaword</td><td>50.19</td><td>50.93</td></tr><tr><td/><td/><td>LSTM: NIST</td><td/><td>50.75</td><td>51.20</td></tr><tr><td>Doc-transformer</td><td colspan="3">Transformer-XL: NIST</td><td>51.27</td><td>51.68</td></tr><tr><td colspan="4">Transformer-XL: NIST + Gigaword</td><td>51.33</td><td>51.99</td></tr><tr><td>Independence</td><td/><td>Architecture</td><td>Data</td><td>Perplexity</td></tr><tr><td>Indep. Sentences</td><td/><td colspan="3">Transformer-XL NIST Transformer-XL NIST + Gigaword</td><td>83.3 96.5</td></tr><tr><td/><td/><td>LSTM</td><td>NIST</td><td>71.6</td></tr><tr><td colspan="2">Indep. Documents</td><td colspan="2">Transformer-XL NIST</td><td>43.8</td></tr><tr><td/><td/><td colspan="3">Transformer-XL NIST + Gigaword</td><td>43.4</td></tr></table>

Perplexity per word (whitespace delimited token) of language models on NIST dev set.

Table content: <table/>

Effect of different components in doc-reranker.

Table content: <table><tr><td/><td/><td>Reranker</td><td/><td>Models</td><td/><td/><td>MT06</td></tr><tr><td/><td/><td>-</td><td/><td colspan="2">Doc-transformer</td><td/><td>49.79</td></tr><tr><td/><td/><td/><td/><td colspan="2">Proposal + LM</td><td/><td>49.79</td></tr><tr><td/><td/><td colspan="2">Doc-reranker</td><td colspan="3">Channel + LM Proposal + Channel</td><td>51.93 50.40</td></tr><tr><td/><td/><td/><td/><td colspan="4">Proposal + Channel + LM 51.99</td></tr><tr><td>Method</td><td>Model</td><td>Unpaired Data</td><td colspan="5">LM PPL Test17 Test18 Test19</td></tr><tr><td>Baseline</td><td>Transformer big</td><td>-</td><td/><td>-</td><td>23.9</td><td>23.9</td><td>24.5</td></tr><tr><td>This work</td><td>Doc-reranker</td><td>WMT Gigaword + WMT</td><td/><td>106.3 63.8</td><td>24.9 25.5</td><td>26.0 26.3</td><td>27.1 27.1</td></tr></table>

SacreBLEU of different models on WMT19 validation and test sets and perplexity per word of the language models on the English side of WMT19 validation set.

Table content: <table/>

. . . however, legislator yeung, i wish to tell you what i am doing today is to ensure every matter can proceed smoothly after the political review starts. therefore, we have to solve some basic problems first and this is a different thing all together. reference: what are the basic problems? transformer: what is the basic problem? reranker:what are the basic questions?

Table content: <table><tr><td>Example 4:</td><td/></tr><tr><td>source:</td><td>基本的问题是什么呢?</td></tr><tr><td>context:</td><td/></tr><tr><td>Example 5:</td><td/></tr><tr><td>source:</td><td>作者：正义之剑</td></tr><tr><td>context:</td><td>sword of justice: prospects for 2006</td></tr><tr><td>reference:</td><td/></tr></table>

Example outputs from the Document Transformer (proposal model) and our doc-reranker.

Table content: <table/>

Peer review (iclr2020):
Summary: The paper describes a noisy channel approach for document-level translation, which does not rely on parallel documents to train. The approach relies on a sentence-level translation model (from target-to-source languages) and a document level language model (on target language), each is trained separately. For decoding, the paper relies on another proposal model (i.e., a sentence level translation model from source to target) and performs beam-search weighted by a linear combination of the scores of all three models. Experiments show strong results on two standard translation benchmarks. Comments: - The proposed approach is strongly based on the neural noisy channel model of Yu et al. 2017 but mainly extends it to context aware translation. While the paper is referenced, I believe more emphasis should be put on the differences of the proposed approach - It seems that the Document Transformer uses parallel-documents to train, so I am wondering if you can still claim that your approach does not require parallel documents. - In general, I think the paper is well written and results are compelling.