A Guider Network for Multi-Dual Learning

A large amount of parallel data is needed to train a strong neural machine translation ( NMT ) system . This is a major challenge for low-resource languages . Building on recent work on unsupervised and semi-supervised methods , we propose a multidual learning framework to improve the performance of NMT by using an almost infinite amount of available monolingual data and some parallel data of other languages . Since our framework involves multiple languages and components , we further propose a timing optimization method that uses reinforcement learning ( RL ) to optimally schedule the different components in order to avoid imbalanced training . Experimental results demonstrate the validity of our model , and confirm its superiority to existing dual learning methods . 1 INTRODUCTION . Neural machine translation ( NMT ) has significantly improved the quality of machine translation in recent several years ( He et al. , 2017 ; Cho et al. , 2014 ; Bahdanau et al. , 2014 ; Sutskever et al. , 2014 ) . The availability of high quality large-scale parallel bilingual corpora made this success possible ( He et al. , 2016 ; Lample et al. , 2017 ; Artetxe et al. , 2017 ; Koehn & Knowles , 2017 ) . However , collecting such parallel corpora remained to be a practical challenge for the large majority of language pairs , such as Basque and German-Russian . To address this issue , different methods have been proposed recently . They can be broadly classified into two categories : ( 1 ) low parallel data of source and target languages are required but heavily depend on third-party resources , such as a large amount of parallel data of another languages ( Dong et al. , 2015 ; Firat et al. , 2016a ; Zoph & Knight , 2016 ; Johnson et al. , 2016 ; Libovickỳ & Helcl , 2017 ) , a special dataset in which same word has same distribution between source and target ( Artetxe et al. , 2017 ; Lample et al. , 2017 ) , images or other multi-media resources corresponding explanation languages ( Chen et al. , 2018 ; Cheng et al. , 2017 ; Johnson et al. , 2016 ) . ( 2 ) using monolingual data to improve the NMT performance , but a pre-trained high quality NMT model is required . For example , Zhang & Zong ( 2016 ) ; Zhang et al . ( 2018 ) enlarged the training dataset by using a pre-trained translation model to translate the monolingual data into pseudo bilingual sentence pairs to address the shortage of parallel training data . Recent dual learning provided a way to enhance model ’ s performance using unlabeled data ( He et al. , 2016 ; Xia et al. , 2017 ; Wang et al. , 2018 ; Luo et al. , 2017 ; Yi et al. , 2017 ; Zhu et al. , 2017 ) . As described in the left panel of Figure 1 , dual learning updates the whole system according to Y ′i which is a forward translation can reconstruct X better and has high language fluency ( scored by language model ) among all the sampled Y ′ ( He et al. , 2016 ; Tu et al. , 2017 ) . However , the best direction to update parameters heavily relies on the quality of sampled translations Y ′1 , Y ′ 2 , ... which may be far from real translations Y due to inaccurate translations existing in the sampled ones , especially for the case where there is no enough data to train a NMT model with high performance . In that case , the dual learning framework suffers from learning a poor mapping between X and Y ′i . Existing dual learning can not provide another optimal updating direction to help train a strong NMT system . Inspired by the two kinds of work , we propose a new guided learning framework ( GLF ) that can alleviate the disadvantages in the prior works and address the problem of parallel corpora shortage . The key component in our framework is a guider network ( GN ) , and different from existing dual learning , GN can provide new learning directions for NMT by leveraging monolingual data of different languages . Specifically , the proposed GN is trained using monolingual data and then used to guide the training of NMT.1 Note that GN has a flexible structure , which can reconstruct the input source sentence not only from the encoder but also from the decoders of NMT systems for different languages . In that case , as shown in the right panel in Figure 1 , GN keeps the hidden states of encoder and decoder consistent and forces the NMT system to map them to the same information space . Well matched encoder and decoder will bring a better word alignment and yield better translation as we will see in Section 3 . And GN thus offers a better updating direction for the NMT model in the dual learning process . Further , benefiting from the flexible structure of GN , we expand GLF to address multilingual translation completed by guiding all the decoders of different languages at the meantime through GN . Ensemble output of different languages can be regard as a normalization operation that can accelerate deep network training according to Ioffe & Szegedy ( 2015 ) . With an increasing number of languages , training order for different components in a complex model significantly affects the final training results since different models have different convergence rates . In traditional fixed order sequential training setting , different convergence rates will cause some models to be over-fitted while the others still have not converged . In that case , the over-fitted models will damage the whole system and yield poor performance . To address this problem , we propose a novel timing optimization method by leveraging reinforcement learning ( RL ) ( Mnih et al. , 2013 ; 2015 ; Wu et al. , 2018 ) to schedule the training order . To the best of our knowledge , this issue has not been studied before . 2 BACKGROUND AND RELATED WORK . Multilingual Promotion : Recent NMT studies achieved better performances by extending sequenceto-sequence model architectures to multilingual translation . By allocating separate encoder and decoder for each language , different interactive modes have been found effective , such as , one source to many targets ( Dong et al. , 2015 ) , many sources to one target ( Zoph & Knight , 2016 ; Firat et al. , 2016b ) , many sources to many targets ( Luong et al. , 2015 ; Firat et al. , 2016a ) , while some gain enhancement by leveraging universal encoder-decoder ( Johnson et al. , 2016 ; Ha et al. , 2016 ) . The others use an inter-lingual to improve the translation quality ( Lu et al. , 2018 ) . Monolingual Promotion : The most relevant work to our model is dual learning ( He et al. , 2016 ) , which enables an NMT system to automatically learn from unlabeled data through a dual-learning game . It aims to train two dual NMTs in a semi-supervised way . It translates monolingual data of source to the target side using one of the two NMTs , then translates it back to the original source by the other one . Then the whole process can be implemented from a dual view . This framework considers the language model score and reconstruction quality as rewards and trains using a policy gradient method . Some previous works ( Xia et al. , 2017 ; Wang et al. , 2018 ) utilized language model to enhance NMT systems . Methods mentioned above achieved some improvements , although not in an end-to-end manner . Others take unlabeled data to reinforce the encoder ( Zhang & Zong , 2016 ) or the decoder ( Sennrich et al. , 2016 ) directly . In this paper , we propose a new framework to exploit unlabeled data in an end-to-end way , which improves the encoder and decoder . Module Scheduling : Multi-lingual models naturally come up with a scheduling issue for it consists of several models , i.e. , how to decide which component of the whole system to train next . Recently , methods in Wang et al . ( 2016 ) ; Sung et al . ( 2017 ) used on a “ teacher-like ” critic net to guide the basic task ’ s training in a meta-learning setting . Yang Fan introduces a `` teacher '' network trained by reinforcement learning to obtain the next piece of pre-divided data set , which is able to accelerate the basic task ’ s training . On another purpose of enhancing the basic model performance , ( Wu et al. , 2018 ) suggested that learning a critic can help ease the problem of biased unlabeled data , and increase the classification accuracy . 1The training data includes monolingual data and parallel bilingual data of some other languages . In our work , unlike multi-lingual translation model in Firat et al . ( 2016a ) , which was trained in a fixed schedule , we rely on an additional `` teacher-like '' model to decide which module should train next in order to obtain a balance gain and a better performance . We utilize the deep Q-learning ( Mnih et al. , 2013 ; Wang et al. , 2015 ; Van Hasselt et al. , 2016 ) algorithm to train a critic network to schedule the training of models . 3 GUIDED LEARNING FRAMEWORK ( GLF ) . This section describes the proposed Guided Learning Framework and its main component guider network ( GN ) which is also called the general generator since it can be used as a generator directly . We first describe the correlation between the source and target languages using the bi-directional entropy and information space ( Section 3.1 ) . The correlation is the basis of GN . Then we present the architecture of GLF and the guider network in Section 3.2 . 3.1 INFORMATION SPACE BEHIND HIDDEN STATES . Attention mechanism ( Bahdanau et al. , 2014 ; Luong et al. , 2015 ) is an effective method to improve the performance of sequence-to-sequence ( seq2seq ) models ( Sutskever et al. , 2014 ) ( the basic framework of NMT ) . In fact , attention is a kind of operation in the information space of hidden states . We thus use attention to explain the information space . The well-known attention mechanism can be formulated by ci = ∑n t=1 αtiht where ht is the hidden state of encoder at position t , αti is the weighting coefficient indicating how well ht and hi match , ci is the context vector associated with the hidden state hi in decoder , and then ci is involved to predict the output word directly . Here we propose ci to be a vector in a space expanded by basis ( h1 , h2 , ... , hN ) . We call this space ( which can be used to represent the input sentence ) the Information Space of the Encoder ( ISE ) . Corresponding to this , we also get a Information Space of the Decoder , ( ISD ) . We call a ISE ( or ISD ) CISE ( or CISD ) , if the basis of the ISE ( or ISD ) can fully represent the whole sentence . That is , a one-to-one mapping exists between the words ( with different meanings ) and their corresponding hidden states ( Bahdanau et al. , 2014 ) . CISE can ensure a good translation since it can provide sufficient information of the source sentence which is needed by the decoder to yield a better translation . CISD can indicates the quality of obtained translation . That is because a good translation ’ s hidden states must expand a CISD . However , how to judge whether the ISD ( or ISE ) is CISD ( or CISE ) is the major challenge . Here we propose a bi-direction attention entropy ( BDE ) to judge it : B = − N∑ i=1 M∑ j=1 pij log pij + pji log pji ( 1 ) where pij is the forward matching probability from the i-th hidden state in encoder hi ( column vector ) to the j-th hidden state in decoder hj ( column vector ) , and pji is the backward matching probability . The only difference between the forward and backward matching probability is the softmax scope : pij = eij∑M j=1 eij ; pji = eij∑N i=1 eij , eij = hTi · hj scores how well hTi and hj match . According to information theory and the attention mechanism ( Bahdanau et al. , 2014 ) , it is clear that we will get a low BDE if and only if : 1 ) Each hidden state in source ( or target ) attends to a few hidden states in target ( or source ) , which can also be considered that ISE and ISD are able to represent each other . 2 ) ISE should be CISE ( the hidden states of different words in the input sentence have different and corresponding hidden state representations ) . These two conditions mean that a low BDE ensures a full coverage of the meanings of translation and input sentence . In that case , ISD will tend to be CISD with the decrease of B , and a better translation can be generated . We can arrive at the same conclusions from experiments ( see Figure 2 ) which also demonstrates that B is more sensitive to the change in BLEU score compared to existing indicators ( e.g . PPL , ROUGEL and METEOR ) . Note that BDE doesn ’ t consider the word orders in translation , thus we need to use language models to ensure a fluent translation . Based on the properties of information space , we develop a novel guided learning framework called multi-dual learning framework to use both monolingual data and third party resources to improve the performance of neural machine translation ( see Section 3.2 ) .

Summary: This paper proposes an extension of the dual learning framework, with a guider network and multiple languages included: (1) Each language $i$ has a guider network $GN_i$, that can be used to reconstruct the source sentence from either the output of the encoder or the output of the decoder. (2) Multiple languages are used in this framework, where each language also has a $GN_i$  for guiding the training according to the reconstruction error. The authors work on MultiUN dataset to verify their algorithms.