MULTI-AGENT DUAL LEARNING
Year: 2019

Abstract:
Dual learning has attracted much attention in machine learning, computer vision and natural language processing communities. The core idea of dual learning is to leverage the duality between the primal task (mapping from domain X to domain Y) and dual task (mapping from domain Y to X ) to boost the performances of both tasks. Existing dual learning framework forms a system with two agents (one primal model and one dual model) to utilize such duality. In this paper, we extend this framework by introducing multiple primal and dual models, and propose the multi-agent dual learning framework. Experiments on neural machine translation and image translation tasks demonstrate the effectiveness of the new framework. In particular, we set a new record on IWSLT 2014 German-to-English translation with a 35.44 BLEU score, achieve a 31.03 BLEU score on WMT 2014 English-toGerman translation with over 2.6 BLEU improvement over the strong Transformer baseline, and set a new record of 49.61 BLEU score on the recent WMT 2018 English-to-German translation.

heading: 1 INTRODUCTION
text: Motivated by the dual nature of many tasks, e.g., English-to-German vs. German-to-English in machine translation, photo-to-Monet vs. Monet-to-photo in image translation, and speech recognition vs. speech synthesis, dual learning is proposed (He et al., 2016) and applied to many applications including neural machine translation (NMT) (He et al., 2016; Xia et al., 2017a;b), image-to-image translation (Zhu et al., 2017; Yi et al., 2017; Luo et al., 2017), question answering (Duan et al., 2017; Sun et al., 2019) and image captioning (Huang et al., 2018). The basic idea of dual learning is to leverage the duality between the two tasks as the feedback signal to regularize training. The regularization has been implemented with different ways in existing works, such as maximizing the likelihood of data reconstruction (He et al., 2016), constraining joint probabilistic consistency (Xia et al., 2017b), and encouraging the model-level information sharing (Xia et al., 2018). Among them, the principle of likelihood maximization of data reconstruction has been mostly adopted (Zhu et al., 2017; He et al., 2016), in which dual learning is formulated as a two-agent game: One agent targets at learning the mapping f : X 7→ Y , while the other learning the mapping of opposite direction g : Y 7→ X . Here X and Y are two domains. In dual learning, x ∈ X is first mapped to ŷ ∈ Y through f and then reconstructed to x̂ ∈ X by g(ŷ). The distortion between x and x̂, denoted as ∆x(x, x̂), is used as the feedback signal to regularize training. Similar reconstruction error can also be constructed in Y and further used for training. In such a two-agent game, g and f can be viewed as an evaluator for each other. g is used to evaluate the quality of ŷ generated by f and return the feedback signal ∆x(x, g(ŷ)) back to f , and vice versa. The quality of such evaluation plays an important role to improve the training of the mapping functions. In the current dual learning framework, only one agent g is used to evaluate and provide feedback signals to the mapping function f in the other direction. In this work, we introduce multiple agents into the learning system to further exploit the potential of dual learning. The agents
∗This work was done when the first author was an intern at Microsoft Research.
in the same direction have similar capability and certain level of diversity to map one domain to the other domain, i.e., X → Y or Y → X . Different agents can be obtained by training multiple f ’s and g’s independently with different random seeds for initialization and data access order. Then for the output of each f (or g), multiple g’s (or f ’s) will provide feedback signals. Intuitively, more agents can lead to more reliable and robust feedback, just like the majority voting of multiple experts, and it is expected to achieve better final performance. We name this new dual learning framework with multiple agents as multi-agent dual learning1.
Although multiple agents are involved in multi-agent dual learning, we still focus on training two mappings f0 : X → Y and g0 : Y → X , similar to the traditional two-agent dual learning. The additionally introduced agents play the role of facilitating the training of f0 and g0. The parameters of the additional agents are fixed during training and have no part to play during testing/inference. Therefore, the inference of multi-agent dual learning is of the same complexity as it is of the standard two-agent dual learning. More precisely, apart from the two agents f0 and g0, we further employ 2(N − 1) additional agents, N ≥ 2, which are pre-trained before optimizing f0 and g0 and subsequently held constant. Among them, N − 1 agents are leveraged to model the mapping fi : X 7→ Y and the others N − 1 agents model gi : Y 7→ X , i ∈ {1, 2, · · · , N − 1}.2 All these 2(N − 1) agents are used to facilitate the training of f0 and g0 in a similar manner as the standard dual learning with minor revisions: To help the training of f0, for any x ∈ X , we first map x to ŷ by f0, build the feedback signals by adaptively summing ∆x(x, gi(ŷ)) for any i ∈ {0, 1, · · · , N − 1}, and then regularize training with the feedback signals. The training of g0 is performed similarly.
We conduct experiments on two benchmark tasks of dual learning, NMT and image-to-image translation, to verify our proposed framework. For NMT, we work on the English↔German translation task with different scales of bilingual data. On IWSLT 2014 translation task with 153K bilingual data and no monolingual data, we achieve 29.52 and 35.44 BLEU scores for English→German and German→English translations respectively, setting a new record on this task. On WMT 2014 translation task with 4.5M bilingual sentence, we achieve 30.05/31.03 and 33.32/35.64 BLEU scores without/with monolingual data, which are the best results of the same settings. On the recent WMT 2018 English→German translation task, we set a new record of 49.61 BLEU score, improving the previous best system by over 1.3 BLEU score. On WMT 2016 unsupervised NMT task, where no bilingual data is available and only 50M monolingual data is provided, we achieve 19.26/23.85 BLEU scores, outperforming all the previous pure NMT based systems. We work on two groups of tasks for image-to-image translation, paint↔photo translations (paint including Van Gogh, Monet, Ukiyo-e and Cezanne), and cityscapes label↔photo translation. Experimental results show that we can generate better images than CycleGAN.
The remaining part of this paper is organized as follows. We introduce the basic framework in Section 2. The applications to NMT and image translation are presented in Section 3 and Section 4. We conclude this paper and discuss the future directions in the last section.

heading: 2 FRAMEWORK
text: In this section, we mathematically formulate the multi-agent dual learning framework, and compare it with previous learning settings.
Consider two domains of interests, X and Y . Let Dx and Dy denote the collection of training data from X and Y respectively, Dx ⊂ X , Dy ⊂ Y . We aim to learn two agents f0 : X 7→ Y , and g0 : Y → X . Without loss of generality, we refer to learning f0 and g0 as the primal task and the dual task respectively. Let ∆x(x, x′) be a mapping X ×X → R, representing the dissimilarity/distance/error between x and x′ where x, x′ ∈ X , and ∆y(y, y′) denote the dissimilarity between the two elements
1Although with the same term, the “multi-agent” or “agent” in this paper has no relationship with multiagent reinforcement learning
2The numbers of agents served for two mapping directions are not necessarily the same. For the tasks where the two domains X and Y are fully symmetric, like the language spaces in NMT and image collections in image-to-image translation, learning the two mappings are of similar difficulty. Therefore, we use the same number of agents to assist training in this paper, and leave the study of asymmetric setting to the future work.
y and y′ in space Y . The standard dual learning loss (He et al., 2016) is formulated as
1 |Dx| ∑ x∈Dx ∆x(x, g0(f0(x))) + 1 |Dy| ∑ y∈Dy ∆y(y, f0(g0(y))), (1)
where |Dx| and |Dy| denote the number of elements in Dx and Dy . In a multi-agent dual learning framework, as a prerequisite there are N − 1 pretrained primal models fi : X → Y and N − 1 dual models gi : Y → X , i = {1, 2, · · · , N − 1}. Each fi is pretrained by minimizing the cross entropy loss − ∑ (x,y) logP (y|x; fi) via stochastic gradient descent where (x, y) is the paired data, or using unsupervised learning techniques to obtain like unsupervised NMT (Lample et al., 2018) or unsupervised image translation (Zhu et al., 2017). Different methods can be used to obtain diverse agents fi, i ∈ {1, 2, · · · , N − 1}, including using various random seeds to affect weight initialization and input order of the training samples, different model architectures, or training over different subsets. The same holds for gi, i ∈ {1, 2, · · · , N−1}. In the training process of f0 and g0, all these pretrained 2(N − 1) models remain fixed, and are linearly aggregated with f0, g0 into two mixed models Fα and Gβ. Specifically, given any αi ≥ 0 and βi ≥ 0 for any i ∈ {0, 1, · · · , N − 1},
Fα = N−1∑ i=0 αifi, Gβ = N−1∑ j=0 βjgj ; s.t. N−1∑ i=0 αi = 1, N−1∑ j=0 βj = 1. (2)
For multi-agent dual learning, the duality feedback signal is built upon Fα and Gβ. Following the basic framework of dual learning (He et al., 2016), for any x ∈ X , all agents first cooperate to generate a ŷ ∈ Y by ŷ = Fα(x), and then jointly reconstruct the x̂ ∈ X through x̂ = Gβ(ŷ). The reconstruction error between y ∈ Y and ŷ = Fα(Gβ(y)) is similarly constructed. The out-coming dual learning loss is defined as
`dual(Dx,Dy;Fα, Gβ) = 1 |Dx| ∑ x∈Dx ∆x(x,Gβ(Fα(x))) + 1 |Dy| ∑ y∈Dy ∆y(y, Fα(Gβ(y))). (3)
In this way, N agent pairs (prima-dual model pairs) are involved to jointly train and improve f0 and g0. When N = 1, Our algorithm degenerates to standard dual learning. No labeled information is required for building `dual. We can apply it on either labeled data or unlabeled data.
Note that Eqn. (3) is the loss about duality. Other training objectives could also be included. For example, in NMT, if bilingual data is available, the cross entropy loss can be included to guide the training; in image-to-image translation, the GAN loss could also be included to enforce the generated images into the correct categories. In the next two sections, we will discuss how to adapt Eqn. (3) to different applications.
Discussion. While there are existing works using multiple agents to boost the model performance, none of them has leveraged the duality. We use the X → Y mapping task to compare the previous work with our proposed framework, following the notations defined in Section 2.
(1) Ensemble learning (Zhou, 2012) is a straightforward way to combine multiple models during inference. To predict the label of x ∈ X , all agents vote together and the final label would be arg miny∈Y ∑N−1 i=0 αi`(fi(x), y), where ` is the loss function over space X × Y . The αi’s can be simply set as 1/N or adaptively set according to the quality of each agent. There are several differences between ensemble learning and our work: 1) ensemble learning does not use multiple agents in training as we do; 2) our multi-agent dual learning uses only one model f0 in inference, which is more efficient than ensemble learning that uses multiple agents; and 3) duality is not considered in conventional ensemble learning.
(2) Knowledge distillation with multiple agents (Hinton et al., 2015; Kim & Rush, 2016). Knowledge distillation consists of two steps: first, all fi’s generate soft labels for x ∈ X , e.g., ŷ = arg miny∈Y ∑N−1 i=0 αi`(fi(x), y); the generated pairs (x, ŷ)’s are together used to train a new model. Each (x, ŷ) is regarded as labeled data without evaluating the quality of ŷ or considering whether it is good enough for model training. In our proposed framework, we leverage the duality to build a feedback loop so as to evaluate the quality of generated pairs.

heading: 3 APPLICATION TO NEURAL MACHINE TRANSLATION
text: In this section, we introduce how to adapt the proposed multi-agent dual learning framework to Neural Machine Translation (NMT), and present evaluation on several public translation datasets.

heading: 3.1 ADAPTION
text: Note that for NMT, X and Y stands for the collection of all possible sentences of two given languages, while Dx and Dy is the dataset. Denote the parameters of mapping f0 and g0 as θf0 and θ g 0 respectively. Following the common practice in NMT, the ∆x and ∆y is specified as the negative log-likelihood, that is, for any x ∈ Dx or x ∈ X ,
∆x(x,Gβ(Fα(x))) = − log ∑ ŷ∈Y P (Fα(x) = ŷ|x;Fα, Gβ)P (Gβ(ŷ) = x|x, Fα(x) = ŷ;Fα, Gβ)
=− log ∑ ŷ∈Y P (Fα(x) = ŷ|x;Fα)P (Gβ(ŷ) = x|ŷ;Gβ). (4)
For ease of reference, we briefly denote Eqn. (4) as ∆x(x,Gβ(Fα(x))) = − log ∑ ŷ∈Y P (ŷ|x;Fα)P (x|ŷ;Gβ). (5)
Similarly, for any y ∈ Y , we have ∆y(y, Fα(Gβ(y))) = − log ∑ x̂∈X P (x̂|y;Gβ)P (y|x̂;Fα). (6)
Following Hassan et al. (2018), we minimize the upper bounds of ∆x and ∆y for ease of optimization, which are respectively denoted as ∆̄x and ∆̄y:
∆̄x(x,Gβ(Fα(x))) = − ∑ ŷ∈Y P (ŷ|x;Fα) logP (x|ŷ;Gβ) ≥ ∆x(x,Gβ(Fα(x)));
∆̄y(y, Fα(Gβ(y))) = − ∑ x̂∈X P (x̂|y;Gβ) logP (y|x̂;Fα) ≥ ∆y(y, Fα(Gβ(y))). (7)
where two ≥’s hold due to Jensen’s inequality. Then we turn to minimize
˜̀dual(Dx,Dy;Fα, Gβ) = 1 |Dx| ∑ x∈Dx ∆̄x(x,Gβ(Fα(x))) + 1 |Dy| ∑ y∈Dy ∆̄y(y, Fα(Gβ(y))). (8)
Directly calculating the gradients for ∆̄x w.r.t θ f 0 and θ g 0 is difficult in that: i) Y is exponentially large since it represents all possible sentences of a language, which makes the computation intractable . ii) All fi’s and gi’s (i ≥ 1) need to be loaded into the memory, although they will not be updated, which makes the computation inefficient. Similar issues exist for computing the gradients of ∆̄y w.r.t θf0 and θ g 0 .
We use importance sampling to address the above two difficulties. Define γ = (0, 1N−1 , · · · , 1 N−1 ), thus Fγ and Gγ represent the combined model from all the pre-trained agents without the target models f0 and g0 (see Eqn. (2)). Define
δ(x, ŷ) = ( P (ŷ|x;Fα) P (ŷ|x;Fγ) ) logP (x|ŷ;Gβ), φ(y, x̂) = ( P (x̂|y;Gβ) P (x̂|y;Gγ) ) logP (y|x̂;Fα). (9)
Then the gradients for ∆̄x w.r.t θ f 0 is: ∂∆̄x ∂θf0
= − ∑ ŷ∈Y P (ŷ|x;Fγ) ∂δ(x,ŷ)
∂θf0 , and can be estimated by:
(i) Given an x ∈ Dx and y ∈ Dy , sample a ŷ according to the distribution P (·|x;Fγ); sample an x̂ according to the distribution P (·|y;Gγ);
(ii) Estimate the gradients of ∆̄x w.r.t θ f 0 as follows (and similarly for θ g 0 without loss of generality):
∂∆̄x ∂θf0 = Eŷ∼P (·|x;Fγ)
[ −∂δ(x, ŷ)
∂θf0
] ≈ −∂δ(x, ŷ)
∂θf0 . (10)
Our method is summarized in Algorithm 1. We do the offline sampling by sampling x̂’s and ŷ’s in advance (step 3); and computing the probabilities related to fi’s and gi’s i ≥ 1 in advance (step 4). When bilingual data is available, denoted as B = {(xk, yk)}Mk=1 where M is the number of training data, we can also apply the negative log-likelihood loss on bilingual data.
Algorithm 1: Algorithm for multi-agent dual learning. 1 Input: Data Dx and Dy; learning rate η; fi and gi i ∈ {0, 1, · · · , N − 1}; mini-batch size K;
bilingual data B if possible; 2 Define γ = (0, 1N−1 , · · · , 1 N−1 ) and α = β = ( 1 N , 1 N , · · · , 1 N ); 3 Sample the datasets Dx and Dy; obtain D̂y = {ŷ ∼ Fγ(x)|x ∈ Dx}, D̂x = {x̂ ∼ Gγ(y)|y ∈ Dy}; 4 Compute the probabilities P (ŷ|x;Fγ) and P (x̂|y;Gγ); 5 while not converged do 6 Randomly sample two batches of Bxŷ ⊂ Dx × D̂y and Bx̂y ⊂ D̂x ×Dy , each of size K; 7 Following Eqn. (9) and Eqn. (10), calculate Gradf0 and Gradg0 as follows:
Gradf0 ← − 1|Bxŷ| ∑ (x,ŷ)∈Bxŷ ∂δ(x,ŷ) ∂θf0 − 1|Bx̂y| ∑ (x̂,y)∈Bx̂y ∂φ(y,x̂) ∂θf0 and
Gradg0 ← − 1|Bxŷ| ∑ (x,ŷ)∈Bxŷ ∂δ(x,ŷ) ∂θg0 − 1|Bx̂y| ∑ (x̂,y)∈Bx̂y ∂φ(y,x̂) ∂θg0
; 8 If bilingual data is available, sample a batch Bxy ⊂ B of size K, calculate
Gradf0 ← Gradf0 − 1K∇θf0 ∑
(x,y)∈Bxy logP (y|x; f0) and Gradg0 ← Gradg0 − 1K∇θg0 ∑ (x,y)∈Bxy logP (x|y; g0);
9 Update the parameters: θf0 ← θ f 0 − ηGradf0 , θ g 0 ← θ g 0 − ηGradg0 ;
10 end

heading: 3.2 EXPERIMENT SETTINGS
text: Dataset We use multiple benchmark NMT datasets to evaluate the effectiveness of the proposed framework, including IWSLT 2014 English↔German translation, WMT 2014 English↔German translation and WMT 2016 unsupervised English↔German translation. English and German is denoted as “En” and “De” respectively for ease of reference. For IWSLT 2014 En↔De translation, following Edunov et al. (2018b), we lowercase all the sentences, and split them into training/validation/test set with 153k/7k/7k sentences respectively. For WMT 2014 En↔De translation, we choose WMT 2014 training set and filter out 4.5M sentences pairs following Gehring et al. (2017) and Vaswani et al. (2017). We concatenate newstest 2012 and newstest 2013 as the validation set and use newstest 2014 as the test set. We also select 8M English and 8M German monolingual sentences from newscrawl 2013 as monolingual dataset. For unsupervised En↔De, following Lample et al. (2018), we choose 50M monolingual English and German sentences and use newstest 2016 as the test set for a fair comparison with the previous work. We preprocess the words into word-pieces in the same way as Wu et al. (2016).
Model Architecture We use Transformer (Vaswani et al., 2017) as the basic model structure. For IWSLT En↔De, we use the transformer small configuration with 4 and 8 blocks to verify the generality of our model with different structures (a 4-block setting refer to transformer with 4 encoder blocks and 4 decoder blocks), where the word embedding dimension, hidden state dimension and non-linear layer dimension is set to be 256, 256 and 1024 respectively. For the WMT task, following Vaswani et al. (2017), we use the transformer big setting with 6 blocks, where the word embedding dimension, hidden state dimension and non-linear layer dimension is 1024, 1024 and 4096 respectively. For the unsupervised NMT task, we choose transformer base setting following Lample et al. (2018), where the above three dimensions are 512, 512 and 2048. The dropout rates for the three settings are 0.2, 0.1 and 0.1 respectively.
Optimization and Evaluation We choose Adam (Kingma & Ba, 2015) to optimize the network. The η in Algorithm 1 is set as 2 × 10−4. The learning rate decay rule and the two β’s of Adam are the same as Vaswani et al. (2017). For the three tasks, we use one, eight and four M40 GPUs to train those networks for three, five and six days respectively. Beam search is applied to generate translations for all the models, where the beam sizes for the three tasks are 6, 4 and 4 respectively.
The evaluation metric is BLEU score (Papineni et al., 2002), which is the geometric mean of n-gram precisions, n ∈ {1, 2, 3, 4}. Baselines We implement three types of important baselines3, including back translation (Sennrich et al., 2015; Edunov et al., 2018a), knowledge distillation (Kim & Rush, 2016) and the two-agent dual learning (He et al., 2016). Take the f : X 7→ Y translation as an example: 1) To use back translation, a reversed translation model g : Y 7→ X is first trained on the original bilingual dataset, then the dataset {(g(y), y)|y ∈ Dy} is constructed, which is then concatenated with the original dataset and train another model f . 2) To use knowledge distillation, we pre-train a teacher model fT : X → Y , and use it to generate the corresponding dataset {(x, fT (x))|x ∈ Dx}, which is then mixed with the ground truth data and used to train a new model4. For these two baselines, we generate sentences using both a single model and multiple models. 3) Two-agent dual learning, which uses f together with g to build the feedback signal to regularize the training (see Eqn. (1)), which we denote as “Dual-1”. Our multi-agent models are denoted as “Dual-i” (i > 1)

heading: 3.3 RESULTS OF NMT ON IWSLT DATASET
text: We regard the English and German sentences in the IWSLT14 bilingual corpus as Dx and Dy in Algorithm 1. We use 4-block networks (i.e. Fγ in Algorithm 1) to generate translations, which are then used to train both 4-block (4B) and 8-block (8B) networks. We compare dual learning (Duali) with the standard baseline (Standard), knowledge distillation (KD-i) and back translation (BT-i) with different number of primal-dual model pairs i ∈ {1, 5}. Specifially, Dual-1 is the two-agent dual learning baseline, and Dual-5 is our multi-agent model with 4 additional pairs of agents.
The results of IWSLT En↔De are presented in Table 1. We can observe that: 1) Dual learning has brought significant improvement over all the baselines (Standard, KD and BT) in both single-agent and multi-agent settings, demonstrating the effectiveness of dual learning; 2) Involving multiple agents into the learning system leads to better performances, which shows the importance of additional feedback signals from other agents. In particular, Dual-5 outperforms Dual-1 by around 0.5 BLEU in 4B setting and around 0.8 BLEU in 8B setting, which proves that dual learning can benefit from cooperating with more agents and demonstrates the effectiveness of our proposed algorithm. Particularly, dual learning withN = 5 agent pairs under 8B setting sets a new record of 35.44 BLEU in De→En and 29.52 BLEU in En→De translations.
Study on different number of agents. We study the performances of our proposed algorithm with respect to different numbers of agent pairs, i.e., N and explore the optimal value of N . We enumerate within N = {1, · · · , 5} and did not try larger N values due to the limitation of computational resources. As we can see from Figure 1, more agents can bring better performance for all settings. While it appears that the performance becomes better with more pair of agents involved into the learning system, the gain becomes more and more marginal (e.g., the maximum difference for N = 4 and N = 5 across all settings is around 0.1 BLEU). Furthermore, larger N leads to higher computational costs such as GPU resources. Thus, it is the most practical to leverage N = 3 agent pairs, where we can benefit from the substantial gain over baselines without too much computational costs. We employ N = 3 for the rest of our experiments.
3All the implementations are based on the official tensor2tensor release: https://github.com/ tensorflow/tensor2tensor.
4We have tried both mixing with ground truth and not and found that mixing is better. On IWSLT 2014 De→En, with/without mixing can achieve 33.89/33.14 BLEU scores.
Study on diversity of agents. We further study the influence of diversity among the agents. We show that diversity is important but not the focus of our algorithm. Detailed results and discussion is presented in Appendix A due to the space limitation.
Study on translation tasks for other language pairs. We carry out another three groups of IWSLT14 experiments, English↔Spanish (Es), English↔Russian (Ru) and English↔Hebrew (He) translations to verify the generality of our framework on different languages. As is illustrated in Table 2, we achieve consistent improvements across different translation tasks (for all languages and all translation directions), further demonstrating the effectiveness and robustness of our algorithm.

heading: 3.4 RESULTS OF NMT ON WMT DATASET
text: We work with two settings on WMT En↔De task, where (1) only bilingual data is available; and (2) additional monolingual data is provided. The results are summarized in Table 3.
From this table, we can see that the proposed algorithm achieves the best performances for all settings, and we observe that: 1) With WMT 2014 bilingual data only, the proposed multi-agent dual learning (Dual-3) achieves the state-of-the-art results to our knowledge, with 30.05 BLEU for En→De and 33.32 BLEU for De→En5. 2) Compared with the traditional two-agent dual learning in He et al. (2016), we can achieve 0.61 BLEU score improvement on En→De and 0.33 on De→En with bilingual data only. For the setting of using monolingual data, the improvements are 1.10 and 0.66 respectively. 3) Although KD and BT can bring improvements under different settings, the
5Although Edunov et al. (2018a) provides a higher BLEU score, they use WMT’18 dataset, which is different from ours and currently not widely used for WMT14 En↔De tasks in NMT literature.
results are not as consistent as dual learning. For example, the performance of BT is not as good as KD under bilingual setting, but is better than KD under monolingual setting. We conjecture the reason is that with feedback signals to a generated sentence in the dual learning framework, the dataset is not only enlarged through sampling, but also guaranteed with good quality of generation.
Study on generality of the algorithm. We apply our method to the state-of-the-art systems on WMT 2016, 2017 and 2018 English-to-German translation challenge to verify the generality of our method. We work with the degenerated case where f0 is warm started with a state-of-the-art model provided by FAIR6 and g0 is fixed. We adopt the commercial system Google Translator (crawled on Oct 16, 2018), the champion system of WMT18 En→De: MS-Marian (Junczys-Dowmunt, 2018), and FAIR’s single and ensemble models (Edunov et al., 2018a) as our baselines. The models are evaluated by sacreBLEU7. More details are left in Appendix B.
As can be seen from Table 4, our method is capable of further improving the strong model by a large margin. Both single and ensemble models with our algorithm outperform the previous best systems and set state-of-the-art results on these tasks, demonstrating the effectiveness of our method.

heading: 3.5 RESULTS OF NMT WITH MONOLINGUAL DATA ONLY
text: Unsupervised NMT is studied recently to learn two translation models without bilingual data (details in Appendix C). We pre-train two unsupervised NMT models with different initialization, use them to translate the 50M monolingual sentences, and apply KD, BT and dual learning algorithms.
As is shown in Table 5, with one pair of agents, KD-1 and BT-1 obtains almost the same results as the standard baseline, while Dual-1 achieves slightly better results on En→De translation. When we increase the number of agents, there are substantial improvements for KD-3, BT-3 and Dual-3. Especially, multi-agent dual learning (Dual-3) achieves 19.26 and 23.85 BLEU scores, setting new records for unsupervised NMT with pure NMT models.

heading: 4 APPLICATION TO IMAGE TRANSLATION
text: In this section, We apply the multi-agent dual learning framework to image translation tasks. Different from Ciregan et al. which also uses multiple agents (several columns of deep neural networks), we leverage the duality of translation tasks and only use one model during inference. We follow the setting of CycleGAN (Zhu et al., 2017), the most popular implementation of image translation that combines the ideas of GAN and dual learning.
6https://s3.amazonaws.com/fairseq-py/models/wmt18.en-de.ensemble.tar.bz2 7sig=BLEU+case.mixed+lang.en-de+numrefs.1+smooth.exp+test.wmt{16,17,18}+tok.13a+version.1.2.11

heading: 4.1 ADAPTION
text: Following the common practice in image translation (Zhu et al., 2017), for each term in Eqn. (3), the ∆x and ∆y are specified as the L1 norm of pixel-level difference between two images. We set α = β = ( 1N , 1 N , · · · , 1 N ). Since there is few literature focusing on generating consensus images using multiple models, we switch Eqn. (3) to a simpler form:
`dual(Dx,Dy;Fα, Gβ) = 1 |Dx| ∑ x∈Dx ‖x−Gβ(f0(x))‖1 + 1 |Dy| ∑ y∈Dy ‖y − Fα(g0(y))‖1, (11)
where Dx and Dy are the datasets of domain X and domain Y . The remaining parts of our model, including the model components, the objective functions, are exactly the same as the standard CycleGAN. There are two discriminators dX : X 7→ [0, 1] and dY : Y 7→ [0, 1] used to differentiate the generated images from natural images. The outputs of dX and dY represent the probability that the input image is a natural one. The GAN loss (Goodfellow et al., 2014) is defined as
`GAN = 1 |Dx| ∑ x∈X [log dX (x)+log ( 1−dY(f0(x)) ) ]+ 1 |Dy| ∑ y∈Y [log dY(y)+log ( 1−dX (g0(y)) ) ].
During the optimization process, the two discriminators will try to maximize `GAN, while the f0 and g0 will minimize `dual(Dx,Dy;Fα, Gβ) + λ`GAN in cooperation with other provided agents, where λ is a hyper-parameter to be tuned. In the experiments, we follow Zhu et al. (2017) to set λ as 10.

heading: 4.2 SETTINGS
text: We use the same datasets, model configuration and optimization algorithm as Zhu et al. (2017). We implement our method based on CycleGAN8. We choose two tasks, the photo↔painting translations where the art styles include Monet, Van Gogh, Ukiyo-e, Cezanne, and cityscapes label↔photo translation. All the images are cropped to 256 × 256. The configuration details can be found at Appendix D.1, which follows the same settings as Zhu et al. (2017).

heading: 4.3 RESULTS OF PAINTING-TO-PHOTO TRANSLATION
text: The task of painting→photo translation is to translate an input painting with specific art style to the original photo captured by a camera. We present case studies on painting→photo translations in Figure 2. Indeed, there are several cases that our algorithm is clearly better than CycleGAN: For the images in the top-left corner, CycleGAN fails to recover the tree while our algorithm succeeds, with many details correctly included like the branches and leaves. In the bottom right corner, the texture of the ground is kept by our results, which is missing from the images generated by CycleGAN.
We further present quantitative evaluation with Fréchet Inception Distance (briefly, FID), which is introduced in Heusel et al. (2017) and verified to be a reasonable metric by Lucic et al. (2018), to evaluate the quality of painting→photo translations. FID captures the distance between generated images and the real ones, and a smaller FID score indicates the better model quality. As is shown in Table 6, our method yields consistent improvements across all datasets. More results in photo→painting are left in Appendix D.2 due to space limitation.
8https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

heading: 4.4 RESULTS OF CITYSCAPES LABEL-TO-PHOTO TRANSLATION
text: We present case studies and quantitative evaluation of label→photo translation on cityscapes images in Figure 3 and Table 7. As can be seen from Figure 3, both CycleGAN and our method can transfer the input label to a matching natural scene, while results of ours are clearer with less noise. For example, in the top left corner, our method can generate flatten roads, clearer cars and buildings and the baseline model fails. In Table 7, we present quantitative evaluation with “FCN-scores” following Isola et al. (2017). Our method achieves better results on the three evaluation metrics. More details about FCN-score and results on photo→label translation are left in Appendix D.3.

heading: 5 CONCLUSION
text: In this paper, we proposed a new framework, multi-agent dual learning, in which multiple primal models and dual models are involved in the learning process. We empirically verified the effectiveness of the proposed framework on multiple machine translation tasks and image translation tasks.
Extending the multi-agent dual learning framework to more applications would be an interesting direction for future research. For example, we can apply our framework to improve the recent non-autoregressive machine translation, which is known to have high inference efficiency yet poor translation quality (Gu et al., 2018; Wang et al., 2019; Guo et al., 2019). It is also worth studying how to apply the framework to other dual learning paradigms, including dual supervised learning (Xia et al., 2017b) and dual transfer learning (Wang et al., 2018). Furthermore, it is challenging yet important to improve the training efficiency while maintaining the substantial improvements.

heading: ACKNOWLEDGEMENT
text: This work is partially supported by the National Science Foundation under Grant No. 1801652.

heading: A STUDY ON DIVERSITY OF AGENTS
text: We further study how the diversity among agents influences the performances of our algorithm on the IWSLT De↔En dataset. We compare three group of agents: (1) Agents with independent runs (“Independent Run”): We use the same agents as Section 3.3, which have the same model architecture and hyperparameters, and are trained through independent runs with different random seeds. (2) Agents with different architectures (“Different Arch”): The agents have different hidden dimensions (256, 512) with different numbers of heads (2, 4). They are also trained through independent runs with different random seeds. Intuitively, this group of agents with different architectures could be more diverse than the first group. (3) Agents from a same run (“Homogeneous”): In comparison to the first two groups where different methods (i.e., different random seeds, different model architecture) are used to encourage diversity among agents, in this group, we use model checkpoints at different (but very close) iterations from a same run. At the late stage of training, the learning rate becomes smaller. Thus the diversity among agents would naturally be very poor.
We train 8-block (8B) networks as Section 3.3 with N = {1, ..., 5} pairs of agents from the above three groups respectively. The same training and inference procedures as Section 3.3 are applied here. As is shown from results in Figure 4, we have the following observations and conclusions:
1. Diversity among agents plays an important role in our method. The Homogeneous agents with poor diversity has the worst performances among the three groups. That is, introducing more homogeneous agents brings very marginal gain.
2. A more diverse group of agents can potentially bring further improvements, yet diversity is not the key focus of our algorithm. Independent Run achieves comparable performances with Different Arch, suggesting that our framework can achieve substantial improvements with a reasonable level of diversity among agents.

heading: B DETAILS ON WMT18 EN→DE CHALLENGES
text: For the study on generality of the algorithm on WMT18 En→De challenges, the training data, Fγ(·) and Gγ(·) are the same as those used in Section 3.4. We work with a degenerated case where g0 (i.e., the De→En model) is pre-trained before optimizing f0 and subsequently hold constant. The f0 is warm started by the models from https://dl.fbaipublicfiles.com/fairseq/ models/wmt18.en-de.ensemble.tar.bz2, which are well trained with tremendous resources. We degenerate our algorithm to optimize the single direction due to the limitation that only En→De model checkpoints are released. We train the model on 8 M40 GPUs for 2 days with PyTorch implementation of our algorithm based on FairSeq Toolkit9 . The batch size is 4096 tokens. We use Adam optimizer and the learning rate is automatically calculated by the system based on the reloaded checkpoint. The dropout ratio is 0.3. We use beam search with beam size 5 to generate candidates.
9https://github.com/pytorch/fairseq
C INTRODUCTION TO UNSUPERVISED NMT
We briefly introduce the recent state-of-the-art unsupervised NMT algorithm in (Lample et al., 2018). Similar to the standard NMT model with bilingual training data, an unsupervised NMT model is also based on the encoder-decoder architecture. The source sentences and target sentences are mapped into a same vocabulary using BPE techniques. The shared embedding is pretrained with fastText Bojanowski et al. (2017) to get good initial values. The proposed model can handle both source-to-target translation and target-to-source translation like Johnson et al. (2017).
The training loss usually consists of two parts, a language model loss and a back-translation loss. Let PX→Y denote the translation from space X to space Y , and so for the other similar notations. (1) Language model loss, implemented by a denoising autoencoder. Mathematically,
L1 =Ex∼X [− logPX→X (x|C(x))] + Ey∼Y [− logPY→Y(y|C(y))], (12) where C(·) is a noise model with randomly dropping several words, swapping words, etc. (2) Back translation loss, implemented by back-translating the monolingual data and feeding into the reversed models. Mathematically,
L2 =Ex∼X [− logPY→X (x|ŷ(x))] + Ey∼Y [− logPX→Y(y|x̂(y))], (13) in which ŷ(x) = arg maxu∈Y PX→Y(u|x) and x̂(y) = arg maxv∈X PY→X (v|y). Note that the four P···’s are implemented in a single encoder-decoder based model, where each translation task has a different “task embedding”, i.e., a learnable vector indicating the translating directions.
D IMAGE-TO-IMAGE TRANSLATION
D.1 NETWORK ARCHITECTURE
We follow Zhu et al. (2017) to configure the model architectures for all tasks on image-to-image translation. For the generator, the network contains 2 stride-2 convolutions, 9 residual blocks, and 2 convolutional layers with stride 12 , (i.e., the transposed convolutional layers). Instance normalization is applied to the network. For the discriminator, the 70× 70 PatchGANs (Isola et al., 2017), which aim to classify whether 70 × 70 overlapping image patches are real or fake. Such a patch-level discriminator architecture has fewer parameters than a full-image discriminator and can work on arbitrarily-sized images in a fully convolutional fashion.
D.2 RESULTS OF PHOTO-TO-PAINTING TRANSLATION
We present results on photo→painting translation in Figure 5. As we can see, both standard CycleGAN and our algorithm successfully translates the inputs to the corresponding target domains. Our model outperforms the baseline in two aspects: 1) multi-agent dual learning can generate clearer images with less block artifact (the Monet column and Ukiyo-e column); and 2) the images generated by our algorithm are faithful to the original images in semantics. For example, the sun is missing from CycleGAN generated images in the top Ukiyo-e column and the bottom Cezanne column, while our algorithm is able to keep the sun.
D.3 RESULTS OF CITYSCAPES PHOTO-TO-LABEL TRANSLATION
FCN Evaluation FCN-scores are used in Zhu et al. (2017) and Isola et al. (2017) to quantitatively evaluate Cityscape photo to semantic label translation. A pre-trained fully convolutional network (Long et al., 2015), briefly, FCN, is used to convert the photo to the corresponding semantic label, and then calculate the per-pixel accuracy, per-class accuracy and mean class IntersectionOver-Union(Class IOU) (Cordts et al., 2016) against the ground truth labels.
Results on photo→label translation The example cases and quantitative evaluation with FCNscores of photo→labels on cityscapes images are shown in Figure 6 and Table 810 respectively. We
10The baseline performance is different from Zhu et al. (2017) since it is unclear how the RGB based labels is converted to integer labels without the official evaluation code released. This is a common issue at https://github.com/phillipi/pix2pix/issues/115.
can observe improvements but the gap is not significant. Note that CycleGAN is a very strong baseline, and we leave further improving photo→label translation to future work.

Review summaries:
The authors propose an extension of dual learning (DL). In DL, one leverages the duality of a dataset, by predicting both forward and backward, e.g. English to German, and German back to English. It’s been shown that training models using this duality is beneficial. This paper extends DL by introducing multiple models for the forward and backward, and using their output to regularise the training of the two main agents.

The paper proposes to modify the "Dual Learning" approach to supervised (and unsupervised) translation problems by making use of additional pretrained mappings for both directions (i.e. primal and dual). These pre-trained mappings ("agents") generate targets from the primal to the dual domain, which need to be mapped back to the original input. It is shown that having >=1 additional agents improves training of the BLEU score in standard MT and unsupervised MT tasks. The method is also applied to unsupervised image-to-image "translation" tasks.