Putting Machine Translation in Context with the Noisy Channel Model

1 INTRODUCTION . There have been many recent demonstrations that neural language models based on Transformers ( Vaswani et al. , 2017 ; Dai et al. , 2019 ) are capable of learning to generate remarkably coherent documents with few ( Zellers et al. , 2019 ) or no ( Radford et al. , 2019 ) conditioning variables . Although large unconditional language models are frequently used to provide representations for natural language understanding applications ( Devlin et al. , 2019 ; Yang et al. , 2019 ; Peters et al. , 2018 ) , how to use them for conditional generation applications remains an open question . In this paper , we employ Bayes ’ rule to use powerful unconditional document language models to improve a conditional task : machine translation . The application of Bayes ’ rule to transform the translation modeling problem p ( y | x ) , where y is the target language , and x is the source language , has a long tradition and was the dominant paradigm in speech and language processing for many years ( Brown et al. , 1993 ) . In this context it is known as the noisy channel ( Shannon & Weaver , 1949 ) , by analogy to an information theoretic conception of Bayes ’ rule as a model for reasoning about messages being sent across an unreliable communication medium . In the noisy channel decomposition ( §2 ) , rather than directly modeling the conditional distribution , we rewrite it as p ( y | x ) ∝ p ( y ) ×p ( x | y ) . More importantly , this changes the learning problem from estimating a single complex conditional distribution to learning two different distributions : a language model p ( y ) , which provides unconditional estimates of the output ( in this paper , documents ) ; and p ( x | y ) , which provides the probability of translating a candidate output y into the ( observed ) source x . As we will discuss below , although the problems of estimating p ( y | x ) and p ( x | y ) are similar , independence assumptions made in p ( x | y ) are less statistically costly than they might otherwise be since , at test time , we will be conditioning on x and reasoning about a posterior distribution over y , which will be jointly dependent on all ( conditionally independent ) parts of x . This statistical fact—which is the same trick that gives naı̈ve Bayes classifiers their expressiveness and ease of estimation—permits us to assume independence between sentence translations in the reverse translation model , and therefore to use parallel sentences ( rather than parallel documents ) to train it . In the posterior , we thus have an implicit estimate of a document-level translation system , even though we made no use of parallel documents . This is particularly useful since parallel sentences are much more readily available than parallel documents . A second benefit of our approach is that the unconditional language model can be estimated from nonparallel data , which exist in vast quantities . The benefits of strong language models estimated from nonparallel data has long been recognized in translation ( Brants et al. , 2007 ) , and more recently in noisy channel approaches to translation based on neural network component distributions ( Yu et al. , 2017 ; Yee et al. , 2019 ; Ng et al. , 2019 ) . Although the noisy channel model is ideal for exploiting the data resources that naturally exist in the world ( large corpora of parallel but independent sentences , and large corpora of monolingual documents ) , we are faced with a much harder decoding problem ( §3 ) . To address this problem , we propose a new beam-search algorithm , exploiting the fact that our document language model operates left-to-right , and our reverse translation model treats sentences independently . The search is guided by a proposal distribution that provides candidate continuations of a document prefix , and these are reranked according to the posterior distribution . In particular , we compare two proposal models : one based on estimates of independent sentence translations ( Vaswani et al. , 2017 ) and one that conditions on the source document context ( Zhang et al. , 2018 ) . To explore the performance of our proposed model , we focus on Chinese–English translation , following a series of papers on document translation ( Zhang et al. , 2018 ; Werlen et al. , 2018 ; Tu et al. , 2018 ; Xiong et al. , 2019 ) . Although in general it is unreasonable to expect that independent translations of sentences would lead to coherent translations of documents between any pair of languages , the task of translating Chinese into English poses some particularly acute challenges . As Chinese is less morphosyntatically complex than English , and the relevant clues for predicting , e.g. , what tense a verb should be in , or whether a noun is singular or plural , may be spread throughout a document , it is crucial that extra sentential context is used . Our experiments ( §4 ) explore : ( 1 ) different approaches to reranking , ( 2 ) different independence assumptions when modelling documents ( i.e. , whether sentences are generated independently or not ) , ( 3 ) different amounts of language modelling data , and ( 4 ) different proposal models . Briefly summarized , we find that document-context language models significantly improve the translation quality obtained with our system , both in terms of BLEU scores , and in terms of a human evaluation . Targeted error analysis demonstrates the document prior is capable of enforcing consistency of tense and number and lexical choice across documents . 2 MODEL DESCRIPTION . We define x = ( x1 , x2 , . . . , xI ) as the source document with I sentences , and similarly , y = ( y1 , y2 , . . . , yJ ) as the target document with J sentences . During the ( human ) translation process , translators may split or recombine sentences , but we will assume that I = J .1 Let xi = ( xi1 , x i 2 , . . . , x i M ) represent the ith sentence in the document , consisting of M words ; likewise yi = ( yi1 , y i 2 , . . . , y i N ) denote the ith sentence in the target document , containing N words . The translation of a document x is determined by finding the document ŷ , where p ( ŷ | x ) is optimal . ŷ = argmax y p ( y | x ) . ( 1 ) Instead of modelling the probability p ( y | x ) directly , we factorize it using Bayes ’ rule : ŷ = argmax y p ( x | y ) × p ( y ) p ( x ) = argmax y p ( x | y ) ︸ ︷︷ ︸ channel model × p ( y ) ︸︷︷︸ language model . ( 2 ) We further assume that sentences are independently translated , and that the sentences are generated by a left-to-right factorization according to the chain rule . Therefore , we have ŷ = argmax y |x|∏ i=1 p ( xi | yi ) × p ( yi | y < i ) , ( 3 ) 1Size mismatches are addressed by merging sentences using sentence alignment algorithms . where y < i = ( y1 , . . . , yi−1 ) denotes a document prefix consisting of the first i−1 target sentences . Thus conceived , this is a generative model of parallel documents that makes a particular independence assumption ; we illustrate the corresponding graphical model on the left side of Figure 1 . 2.1 IMPACT OF THE CONDITIONAL INDEPENDENCE ASSUMPTION . At first glance , the conditional independence assumption we have made might seem to be the very independence assumption that bedevils traditional neural document translation systems—translations of sentence i appear to be uninfluenced by the translation of any sentence j 6= i . However , while this is the case during training , this is not the case at test time . Then , we will be conditioning on the xi ’ s ( the source language sentences ) , and reasoning about the posterior distribution over the “ underlying ” yi ’ s . By conditioning on the child variables , conditional dependencies between all yi ’ s and between each yi and all xi ’ s are created ( Shachter , 1998 ) . The ( in ) dependencies that are present in the posterior distribution are shown in the right part of Figure 1 . Thus , although modelling p ( y | x ) or p ( x | y ) would appear to be superficially similar , the statistical impact of making a conditional independence assumption is quite different . This is fortunate , as it makes it straightforward to use parallel sentences , rather than assuming we have parallel documents which are less often available ( Voita et al. , 2019 ; Zhang et al. , 2018 ; Maruf et al. , 2019 , inter alia ) . Finally , since we only need to learn to model the likelihood of sentence translations ( rather than document translations ) , the challenges of guiding the learners to make robust generalizations in direct document translation models ( Voita et al. , 2019 ; Zhang et al. , 2018 ; Maruf et al. , 2019 , inter alia ) are neatly avoided . 2.2 LEARNING . We can parameterize the channel probability p ( xi | yi ) using any sequence-to-sequence model and parameterize the language model p ( yi | y < i ) using any language model . It is straightforward to learn our model : we simply optimize the channel model and the language model separately on parallel data and monolingual data , respectively . We remark that it is a significant practical advantage of this parameterization that we can retrain the channel and language models independently , for example if we acquire more monolingual data , or use different language models with the same channel model conditioned on the domain of the source text . 3 DECODING . Because of the global dependencies in the posterior distribution , decoding in our noisy channel model is computationally complex . On one hand , similarly to the decoding problem faced in standard sequence-to-sequence models , we must search over the space of all possible outputs with a model that makes no Markov assumptions . On the other hand , unlike traditional neural MT models , we have to have a complete guess of yi before we can reliably compute the score of p ( xi | yi ) according to our channel model , making traditional greedy and near-greedy algorithms ineffective . To deal with this issue , we leverage an auxiliary proposal model q ( y | x ) which approximates the posterior distribution using a direct model , and we use this to focus on our search on promising parts of the output space . Because of the autoregressive factorization of the language model , and the independent sentence translation assumption in the channel model , we can carry out the reranking process using a leftto-right beam search strategy with the aid of our proposal model . Figure 2 illustrates the decoding process . For an input document , we let the proposal model propose K candidate translations for each sentence in the document . We then search for the best path from these candidate translations . We use beam search under the objective defined by the following linear model , Ox≤i , y≤i = λ1 log q ( y ≤i | x≤i ) + λ2 log p ( x≤i | y≤i ) + log p ( y≤i ) + λ3|y≤i| , ( 4 ) where |y≤i| denotes the total number of tokens in the partial document y≤i . Note that Equation 4 is a generalization of Equation 2 in log space—if we set λ1 = λ3 = 0 and take the log of Equation 2 the two objectives are equivalent . The extra factors—the proposal probability and the length of the output—provide improvements ( e.g. , by calibrating the expected length of the output ) , and can be incorporated at no cost in the model ; they are widely used in prior work ( Yu et al. , 2017 ; Yee et al. , 2019 ; Ng et al. , 2019 ) .

Summary: The paper describes a noisy channel approach for document-level translation, which does not rely on parallel documents to train. The approach relies on a sentence-level translation model (from target-to-source languages) and a document level language model (on target language), each is trained separately. For decoding, the paper relies on another proposal model (i.e., a sentence level translation model from source to target) and performs beam-search weighted by a linear combination of the scores of all three models. Experiments show strong results on two standard translation benchmarks.