Multi-Agent Dual Learning

1 INTRODUCTION . Motivated by the dual nature of many tasks , e.g. , English-to-German vs. German-to-English in machine translation , photo-to-Monet vs. Monet-to-photo in image translation , and speech recognition vs. speech synthesis , dual learning is proposed ( He et al. , 2016 ) and applied to many applications including neural machine translation ( NMT ) ( He et al. , 2016 ; Xia et al. , 2017a ; b ) , image-to-image translation ( Zhu et al. , 2017 ; Yi et al. , 2017 ; Luo et al. , 2017 ) , question answering ( Duan et al. , 2017 ; Sun et al. , 2019 ) and image captioning ( Huang et al. , 2018 ) . The basic idea of dual learning is to leverage the duality between the two tasks as the feedback signal to regularize training . The regularization has been implemented with different ways in existing works , such as maximizing the likelihood of data reconstruction ( He et al. , 2016 ) , constraining joint probabilistic consistency ( Xia et al. , 2017b ) , and encouraging the model-level information sharing ( Xia et al. , 2018 ) . Among them , the principle of likelihood maximization of data reconstruction has been mostly adopted ( Zhu et al. , 2017 ; He et al. , 2016 ) , in which dual learning is formulated as a two-agent game : One agent targets at learning the mapping f : X 7→ Y , while the other learning the mapping of opposite direction g : Y 7→ X . Here X and Y are two domains . In dual learning , x ∈ X is first mapped to ŷ ∈ Y through f and then reconstructed to x̂ ∈ X by g ( ŷ ) . The distortion between x and x̂ , denoted as ∆x ( x , x̂ ) , is used as the feedback signal to regularize training . Similar reconstruction error can also be constructed in Y and further used for training . In such a two-agent game , g and f can be viewed as an evaluator for each other . g is used to evaluate the quality of ŷ generated by f and return the feedback signal ∆x ( x , g ( ŷ ) ) back to f , and vice versa . The quality of such evaluation plays an important role to improve the training of the mapping functions . In the current dual learning framework , only one agent g is used to evaluate and provide feedback signals to the mapping function f in the other direction . In this work , we introduce multiple agents into the learning system to further exploit the potential of dual learning . The agents ∗This work was done when the first author was an intern at Microsoft Research . in the same direction have similar capability and certain level of diversity to map one domain to the other domain , i.e. , X → Y or Y → X . Different agents can be obtained by training multiple f ’ s and g ’ s independently with different random seeds for initialization and data access order . Then for the output of each f ( or g ) , multiple g ’ s ( or f ’ s ) will provide feedback signals . Intuitively , more agents can lead to more reliable and robust feedback , just like the majority voting of multiple experts , and it is expected to achieve better final performance . We name this new dual learning framework with multiple agents as multi-agent dual learning1 . Although multiple agents are involved in multi-agent dual learning , we still focus on training two mappings f0 : X → Y and g0 : Y → X , similar to the traditional two-agent dual learning . The additionally introduced agents play the role of facilitating the training of f0 and g0 . The parameters of the additional agents are fixed during training and have no part to play during testing/inference . Therefore , the inference of multi-agent dual learning is of the same complexity as it is of the standard two-agent dual learning . More precisely , apart from the two agents f0 and g0 , we further employ 2 ( N − 1 ) additional agents , N ≥ 2 , which are pre-trained before optimizing f0 and g0 and subsequently held constant . Among them , N − 1 agents are leveraged to model the mapping fi : X 7→ Y and the others N − 1 agents model gi : Y 7→ X , i ∈ { 1 , 2 , · · · , N − 1 } .2 All these 2 ( N − 1 ) agents are used to facilitate the training of f0 and g0 in a similar manner as the standard dual learning with minor revisions : To help the training of f0 , for any x ∈ X , we first map x to ŷ by f0 , build the feedback signals by adaptively summing ∆x ( x , gi ( ŷ ) ) for any i ∈ { 0 , 1 , · · · , N − 1 } , and then regularize training with the feedback signals . The training of g0 is performed similarly . We conduct experiments on two benchmark tasks of dual learning , NMT and image-to-image translation , to verify our proposed framework . For NMT , we work on the English↔German translation task with different scales of bilingual data . On IWSLT 2014 translation task with 153K bilingual data and no monolingual data , we achieve 29.52 and 35.44 BLEU scores for English→German and German→English translations respectively , setting a new record on this task . On WMT 2014 translation task with 4.5M bilingual sentence , we achieve 30.05/31.03 and 33.32/35.64 BLEU scores without/with monolingual data , which are the best results of the same settings . On the recent WMT 2018 English→German translation task , we set a new record of 49.61 BLEU score , improving the previous best system by over 1.3 BLEU score . On WMT 2016 unsupervised NMT task , where no bilingual data is available and only 50M monolingual data is provided , we achieve 19.26/23.85 BLEU scores , outperforming all the previous pure NMT based systems . We work on two groups of tasks for image-to-image translation , paint↔photo translations ( paint including Van Gogh , Monet , Ukiyo-e and Cezanne ) , and cityscapes label↔photo translation . Experimental results show that we can generate better images than CycleGAN . The remaining part of this paper is organized as follows . We introduce the basic framework in Section 2 . The applications to NMT and image translation are presented in Section 3 and Section 4 . We conclude this paper and discuss the future directions in the last section . 2 FRAMEWORK . In this section , we mathematically formulate the multi-agent dual learning framework , and compare it with previous learning settings . Consider two domains of interests , X and Y . Let Dx and Dy denote the collection of training data from X and Y respectively , Dx ⊂ X , Dy ⊂ Y . We aim to learn two agents f0 : X 7→ Y , and g0 : Y → X . Without loss of generality , we refer to learning f0 and g0 as the primal task and the dual task respectively . Let ∆x ( x , x′ ) be a mapping X ×X → R , representing the dissimilarity/distance/error between x and x′ where x , x′ ∈ X , and ∆y ( y , y′ ) denote the dissimilarity between the two elements 1Although with the same term , the “ multi-agent ” or “ agent ” in this paper has no relationship with multiagent reinforcement learning 2The numbers of agents served for two mapping directions are not necessarily the same . For the tasks where the two domains X and Y are fully symmetric , like the language spaces in NMT and image collections in image-to-image translation , learning the two mappings are of similar difficulty . Therefore , we use the same number of agents to assist training in this paper , and leave the study of asymmetric setting to the future work . y and y′ in space Y . The standard dual learning loss ( He et al. , 2016 ) is formulated as 1 |Dx| ∑ x∈Dx ∆x ( x , g0 ( f0 ( x ) ) ) + 1 |Dy| ∑ y∈Dy ∆y ( y , f0 ( g0 ( y ) ) ) , ( 1 ) where |Dx| and |Dy| denote the number of elements in Dx and Dy . In a multi-agent dual learning framework , as a prerequisite there are N − 1 pretrained primal models fi : X → Y and N − 1 dual models gi : Y → X , i = { 1 , 2 , · · · , N − 1 } . Each fi is pretrained by minimizing the cross entropy loss − ∑ ( x , y ) logP ( y|x ; fi ) via stochastic gradient descent where ( x , y ) is the paired data , or using unsupervised learning techniques to obtain like unsupervised NMT ( Lample et al. , 2018 ) or unsupervised image translation ( Zhu et al. , 2017 ) . Different methods can be used to obtain diverse agents fi , i ∈ { 1 , 2 , · · · , N − 1 } , including using various random seeds to affect weight initialization and input order of the training samples , different model architectures , or training over different subsets . The same holds for gi , i ∈ { 1 , 2 , · · · , N−1 } . In the training process of f0 and g0 , all these pretrained 2 ( N − 1 ) models remain fixed , and are linearly aggregated with f0 , g0 into two mixed models Fα and Gβ . Specifically , given any αi ≥ 0 and βi ≥ 0 for any i ∈ { 0 , 1 , · · · , N − 1 } , Fα = N−1∑ i=0 αifi , Gβ = N−1∑ j=0 βjgj ; s.t . N−1∑ i=0 αi = 1 , N−1∑ j=0 βj = 1 . ( 2 ) For multi-agent dual learning , the duality feedback signal is built upon Fα and Gβ . Following the basic framework of dual learning ( He et al. , 2016 ) , for any x ∈ X , all agents first cooperate to generate a ŷ ∈ Y by ŷ = Fα ( x ) , and then jointly reconstruct the x̂ ∈ X through x̂ = Gβ ( ŷ ) . The reconstruction error between y ∈ Y and ŷ = Fα ( Gβ ( y ) ) is similarly constructed . The out-coming dual learning loss is defined as ` dual ( Dx , Dy ; Fα , Gβ ) = 1 |Dx| ∑ x∈Dx ∆x ( x , Gβ ( Fα ( x ) ) ) + 1 |Dy| ∑ y∈Dy ∆y ( y , Fα ( Gβ ( y ) ) ) . ( 3 ) In this way , N agent pairs ( prima-dual model pairs ) are involved to jointly train and improve f0 and g0 . When N = 1 , Our algorithm degenerates to standard dual learning . No labeled information is required for building ` dual . We can apply it on either labeled data or unlabeled data . Note that Eqn . ( 3 ) is the loss about duality . Other training objectives could also be included . For example , in NMT , if bilingual data is available , the cross entropy loss can be included to guide the training ; in image-to-image translation , the GAN loss could also be included to enforce the generated images into the correct categories . In the next two sections , we will discuss how to adapt Eqn . ( 3 ) to different applications . Discussion . While there are existing works using multiple agents to boost the model performance , none of them has leveraged the duality . We use the X → Y mapping task to compare the previous work with our proposed framework , following the notations defined in Section 2 . ( 1 ) Ensemble learning ( Zhou , 2012 ) is a straightforward way to combine multiple models during inference . To predict the label of x ∈ X , all agents vote together and the final label would be arg miny∈Y ∑N−1 i=0 αi ` ( fi ( x ) , y ) , where ` is the loss function over space X × Y . The αi ’ s can be simply set as 1/N or adaptively set according to the quality of each agent . There are several differences between ensemble learning and our work : 1 ) ensemble learning does not use multiple agents in training as we do ; 2 ) our multi-agent dual learning uses only one model f0 in inference , which is more efficient than ensemble learning that uses multiple agents ; and 3 ) duality is not considered in conventional ensemble learning . ( 2 ) Knowledge distillation with multiple agents ( Hinton et al. , 2015 ; Kim & Rush , 2016 ) . Knowledge distillation consists of two steps : first , all fi ’ s generate soft labels for x ∈ X , e.g. , ŷ = arg miny∈Y ∑N−1 i=0 αi ` ( fi ( x ) , y ) ; the generated pairs ( x , ŷ ) ’ s are together used to train a new model . Each ( x , ŷ ) is regarded as labeled data without evaluating the quality of ŷ or considering whether it is good enough for model training . In our proposed framework , we leverage the duality to build a feedback loop so as to evaluate the quality of generated pairs .

Summary: The paper proposes to modify the "Dual Learning" approach to supervised (and unsupervised) translation problems by making use of additional pretrained mappings for both directions (i.e. primal and dual). These pre-trained mappings ("agents") generate targets from the primal to the dual domain, which need to be mapped back to the original input. It is shown that having >=1 additional agents improves training of the BLEU score in standard MT and unsupervised MT tasks. The method is also applied to unsupervised image-to-image "translation" tasks.