Title: MULTI-AGENT DUAL LEARNING

Abstract: Dual learning has attracted much attention in machine learning, computer vision and natural language processing communities. The core idea of dual learning is to leverage the duality between the primal task (mapping from domain X to domain Y) and dual task (mapping from domain Y to X ) to boost the performances of both tasks. Existing dual learning framework forms a system with two agents (one primal model and one dual model) to utilize such duality. In this paper, we extend this framework by introducing multiple primal and dual models, and propose the multi-agent dual learning framework. Experiments on neural machine translation and image translation tasks demonstrate the effectiveness of the new framework. In particular, we set a new record on IWSLT 2014 German-to-English translation with a 35.44 BLEU score, achieve a 31.03 BLEU score on WMT 2014 English-to-German translation with over 2.6 BLEU improvement over the strong Transformer baseline, and set a new record of 49.61 BLEU score on the recent WMT 2018 English-to-German translation.

1. INTRODUCTION:

Motivated by the dual nature of many tasks, e.g., English-to-German vs. German-to-English in machine translation, photo-to-Monet vs. Monet-to-photo in image translation, and speech recognition vs. speech synthesis, dual learning is proposed (He et al., 2016) and applied to many applications including neural machine translation (NMT) (He et al., 2016; Xia et al., 2017a; b) , image-to-image translation (Zhu et al., 2017; Yi et al., 2017; Luo et al., 2017) , question answering (Duan et al., 2017; Sun et al., 2019) and image captioning (Huang et al., 2018) . The basic idea of dual learning is to leverage the duality between the two tasks as the feedback signal to regularize training. The regularization has been implemented with different ways in existing works, such as maximizing the likelihood of data reconstruction (He et al., 2016) , constraining joint probabilistic consistency (Xia et al., 2017b) , and encouraging the model-level information sharing (Xia et al., 2018) . Among them, the principle of likelihood maximization of data reconstruction has been mostly adopted (Zhu et al., 2017; He et al., 2016) , in which dual learning is formulated as a two-agent game: One agent targets at learning the mapping f : X → Y, while the other learning the mapping of opposite direction g : Y → X . Here X and Y are two domains. In dual learning, x ∈ X is first mapped to ŷ ∈ Y through f and then reconstructed to x ∈ X by g(ŷ). The distortion between x and x, denoted as ∆ x (x, x), is used as the feedback signal to regularize training. Similar reconstruction error can also be constructed in Y and further used for training.

In such a two-agent game, g and f can be viewed as an evaluator for each other. g is used to evaluate the quality of ŷ generated by f and return the feedback signal ∆ x (x, g(ŷ)) back to f , and vice versa. The quality of such evaluation plays an important role to improve the training of the mapping functions. In the current dual learning framework, only one agent g is used to evaluate and provide feedback signals to the mapping function f in the other direction. In this work, we introduce multiple agents into the learning system to further exploit the potential of dual learning. The agents in the same direction have similar capability and certain level of diversity to map one domain to the other domain, i.e., X → Y or Y → X . Different agents can be obtained by training multiple f 's and g's independently with different random seeds for initialization and data access order. Then for the output of each f (or g), multiple g's (or f 's) will provide feedback signals. Intuitively, more agents can lead to more reliable and robust feedback, just like the majority voting of multiple experts, and it is expected to achieve better final performance. We name this new dual learning framework with multiple agents as multi-agent dual learning1 .

Although multiple agents are involved in multi-agent dual learning, we still focus on training two mappings f 0 : X → Y and g 0 : Y → X , similar to the traditional two-agent dual learning. The additionally introduced agents play the role of facilitating the training of f 0 and g 0 . The parameters of the additional agents are fixed during training and have no part to play during testing/inference. Therefore, the inference of multi-agent dual learning is of the same complexity as it is of the standard two-agent dual learning. More precisely, apart from the two agents f 0 and g 0 , we further employ 2(N -1) additional agents, N ≥ 2, which are pre-trained before optimizing f 0 and g 0 and subsequently held constant. Among them, N -1 agents are leveraged to model the mapping f i : X → Y and the others N -1 agents model g

i : Y → X , i ∈ {1, 2, • • • , N -1}.

2 All these 2(N -1) agents are used to facilitate the training of f 0 and g 0 in a similar manner as the standard dual learning with minor revisions: To help the training of f 0 , for any x ∈ X , we first map x to ŷ by f 0 , build the feedback signals by adaptively summing ∆ x (x, g i (ŷ)) for any i ∈ {0, 1, • • • , N -1}, and then regularize training with the feedback signals. The training of g 0 is performed similarly.

We conduct experiments on two benchmark tasks of dual learning, NMT and image-to-image translation, to verify our proposed framework. For NMT, we work on the English↔German translation task with different scales of bilingual data. On IWSLT 2014 translation task with 153K bilingual data and no monolingual data, we achieve 29.52 and 35.44 BLEU scores for English→German and German→English translations respectively, setting a new record on this task. On WMT 2014 translation task with 4.5M bilingual sentence, we achieve 30.05/31.03 and 33.32/35.64 BLEU scores without/with monolingual data, which are the best results of the same settings. On the recent WMT 2018 English→German translation task, we set a new record of 49.61 BLEU score, improving the previous best system by over 1.3 BLEU score. On WMT 2016 unsupervised NMT task, where no bilingual data is available and only 50M monolingual data is provided, we achieve 19.26/23.85 BLEU scores, outperforming all the previous pure NMT based systems. We work on two groups of tasks for image-to-image translation, paint↔photo translations (paint including Van Gogh, Monet, Ukiyo-e and Cezanne), and cityscapes label↔photo translation. Experimental results show that we can generate better images than CycleGAN.

The remaining part of this paper is organized as follows. We introduce the basic framework in Section 2. The applications to NMT and image translation are presented in Section 3 and Section 4. We conclude this paper and discuss the future directions in the last section.

2. FRAMEWORK:

In this section, we mathematically formulate the multi-agent dual learning framework, and compare it with previous learning settings.

Consider two domains of interests, X and Y. Let D x and D y denote the collection of training data from X and Y respectively, D x ⊂ X , D y ⊂ Y. We aim to learn two agents f 0 : X → Y, and g 0 : Y → X . Without loss of generality, we refer to learning f 0 and g 0 as the primal task and the dual task respectively. Let ∆ x (x, x ) be a mapping X × X → R, representing the dissimilarity/distance/error between x and x where x, x ∈ X , and ∆ y (y, y ) denote the dissimilarity between the two elements y and y in space Y. The standard dual learning loss (He et al., 2016) is formulated as

EQUATION

where |D x | and |D y | denote the number of elements in D x and D y . In a multi-agent dual learning framework, as a prerequisite there are N -1 pretrained primal models f i : X → Y and N -1 dual models g

i : Y → X , i = {1, 2, • • • , N -1}.

Each f i is pretrained by minimizing the cross entropy loss -(x,y) log P (y|x; f i ) via stochastic gradient descent where (x, y) is the paired data, or using unsupervised learning techniques to obtain like unsupervised NMT (Lample et al., 2018) or unsupervised image translation (Zhu et al., 2017) . Different methods can be used to obtain diverse agents 

f i , i ∈ {1, 2, • • • , N -1},

, i ∈ {1, 2, • • • , N -1}.

In the training process of f 0 and g 0 , all these pretrained 2(N -1) models remain fixed, and are linearly aggregated with f 0 , g 0 into two mixed models F α and G β . Specifically, given any α i ≥ 0 and

EQUATION

For multi-agent dual learning, the duality feedback signal is built upon F α and G β . Following the basic framework of dual learning (He et al., 2016) , for any x ∈ X , all agents first cooperate to generate a ŷ ∈ Y by ŷ = F α (x), and then jointly reconstruct the x ∈ X through x = G β (ŷ). The reconstruction error between y ∈ Y and ŷ = F α (G β (y)) is similarly constructed. The out-coming dual learning loss is defined as

dual (D x , D y ; F α , G β ) = 1 |D x | x∈Dx ∆ x (x, G β (F α (x))) + 1 |D y | y∈Dy ∆ y (y, F α (G β (y))). (3)

In this way, N agent pairs (prima-dual model pairs) are involved to jointly train and improve f 0 and g 0 . When N = 1, Our algorithm degenerates to standard dual learning. No labeled information is required for building dual . We can apply it on either labeled data or unlabeled data.

Note that Eqn. (3) is the loss about duality. Other training objectives could also be included. For example, in NMT, if bilingual data is available, the cross entropy loss can be included to guide the training; in image-to-image translation, the GAN loss could also be included to enforce the generated images into the correct categories. In the next two sections, we will discuss how to adapt Eqn. (3) to different applications.

Discussion. While there are existing works using multiple agents to boost the model performance, none of them has leveraged the duality. We use the X → Y mapping task to compare the previous work with our proposed framework, following the notations defined in Section 2.

(1) Ensemble learning (Zhou, 2012) is a straightforward way to combine multiple models during inference. To predict the label of x ∈ X , all agents vote together and the final label would be arg min y∈Y N -1 i=0 α i (f i (x), y), where is the loss function over space X × Y. The α i 's can be simply set as 1/N or adaptively set according to the quality of each agent. There are several differences between ensemble learning and our work: 1) ensemble learning does not use multiple agents in training as we do; 2) our multi-agent dual learning uses only one model f 0 in inference, which is more efficient than ensemble learning that uses multiple agents; and 3) duality is not considered in conventional ensemble learning.

(2) Knowledge distillation with multiple agents (Hinton et al., 2015; Kim & Rush, 2016) . Knowledge distillation consists of two steps: first, all f i 's generate soft labels for x ∈ X , e.g., ŷ = arg min y∈Y N -1 i=0 α i (f i (x), y); the generated pairs (x, ŷ)'s are together used to train a new model. Each (x, ŷ) is regarded as labeled data without evaluating the quality of ŷ or considering whether it is good enough for model training. In our proposed framework, we leverage the duality to build a feedback loop so as to evaluate the quality of generated pairs.

3. APPLICATION TO NEURAL MACHINE TRANSLATION:

In this section, we introduce how to adapt the proposed multi-agent dual learning framework to Neural Machine Translation (NMT), and present evaluation on several public translation datasets.

3.1. ADAPTION:

Note that for NMT, X and Y stands for the collection of all possible sentences of two given languages, while D x and D y is the dataset. Denote the parameters of mapping f 0 and g 0 as θ f 0 and θ g 0 respectively. Following the common practice in NMT, the ∆ x and ∆ y is specified as the negative log-likelihood, that is, for any x ∈ D x or x ∈ X ,

EQUATION

For ease of reference, we briefly denote Eqn. (4) as

EQUATION

Similarly, for any y ∈ Y, we have

EQUATION

Following Hassan et al. (2018) , we minimize the upper bounds of ∆ x and ∆ y for ease of optimization, which are respectively denoted as ∆x and ∆y :

EQUATION

where two ≥'s hold due to Jensen's inequality. Then we turn to minimize

˜ dual (D x , D y ; F α , G β ) = 1 |D x | x∈Dx ∆x (x, G β (F α (x))) + 1 |D y | y∈Dy ∆y (y, F α (G β (y))). (8)

Directly calculating the gradients for ∆x w.r.t θ f 0 and θ g 0 is difficult in that: i) Y is exponentially large since it represents all possible sentences of a language, which makes the computation intractable . ii) All f i 's and g i 's (i ≥ 1) need to be loaded into the memory, although they will not be updated, which makes the computation inefficient. Similar issues exist for computing the gradients of ∆y w.r.t θ f 0 and θ g 0 . We use importance sampling to address the above two difficulties. Define γ = (0

, 1 N -1 , • • • , 1 N -1

), thus F γ and G γ represent the combined model from all the pre-trained agents without the target models f 0 and g 0 (see Eqn. ( 2)). Define

δ(x, ŷ) = P (ŷ|x; F α ) P (ŷ|x; F γ ) log P (x|ŷ; G β ), φ(y, x) = P (x|y; G β ) P (x|y; G γ ) log P (y|x; F α ). (9)

Then the gradients for ∆x w.r.t

θ f 0 is: ∂ ∆x ∂θ f 0 = -ŷ∈Y P (ŷ|x; F γ ) ∂δ(x,ŷ) ∂θ f 0

, and can be estimated by:

(i) Given an x ∈ D x and y ∈ D y , sample a ŷ according to the distribution P (•|x; F γ ); sample an x according to the distribution P (•|y; G γ );

(ii) Estimate the gradients of ∆x w.r.t θ f 0 as follows (and similarly for θ g 0 without loss of generality):

EQUATION

)

Our method is summarized in Algorithm 1. We do the offline sampling by sampling x's and ŷ's in advance (step 3); and computing the probabilities related to f i 's and g i 's i ≥ 1 in advance (step 4). When bilingual data is available, denoted as B = {(x k , y k )} M k=1 where M is the number of training data, we can also apply the negative log-likelihood loss on bilingual data.

Algorithm 1: Algorithm for multi-agent dual learning.

Input: Data D x and D y ; learning rate η; f i and Grad f0 ← -1

g i i ∈ {0, 1, • • • , N -1}; mini-batch size K; bilingual data B if possible; Define γ = (0, 1 N -1 , • • • , 1 N -1 ) and α = β = ( 1 N , 1 N , • • • , 1 N ); Sample the datasets D x and D y ; obtain Dy = {ŷ ∼ F γ (x)|x ∈ D x }, Dx = {x ∼ G γ (y)

|B x ŷ | (x,ŷ)∈B x ŷ ∂δ(x,ŷ) ∂θ f 0 -1 |B xy | (x,y)∈B xy ∂φ(y,x) ∂θ f 0 and Grad g0 ← -1 |B x ŷ | (x,ŷ)∈B x ŷ ∂δ(x,ŷ) ∂θ g 0 -1 |B xy | (x,y)∈B xy ∂φ(y,x) ∂θ g 0 ; If bilingual data is available, sample a batch B xy ⊂ B of size K, calculate Grad f0 ← Grad f0 -1 K ∇ θ f 0 (x,y)∈Bxy log P (y|x; f 0 ) and Grad g0 ← Grad g0 -1 K ∇ θ g 0 (x,y)∈Bxy log P (x|y; g 0 );

Update the parameters:

θ f 0 ← θ f 0 -η Grad f0 , θ g 0 ← θ g 0 -η Grad g0 ; end 3.2 EXPERIMENT SETTINGS

Dataset We use multiple benchmark NMT datasets to evaluate the effectiveness of the proposed framework, including IWSLT 2014 English↔German translation, WMT 2014 English↔German translation and WMT 2016 unsupervised English↔German translation. English and German is denoted as "En" and "De" respectively for ease of reference. For IWSLT 2014 En↔De translation, following Edunov et al. (2018b) , we lowercase all the sentences, and split them into training/validation/test set with 153k/7k/7k sentences respectively. For WMT 2014 En↔De translation, we choose WMT 2014 training set and filter out 4.5M sentences pairs following Gehring et al. (2017) and Vaswani et al. (2017) . We concatenate newstest 2012 and newstest 2013 as the validation set and use newstest 2014 as the test set. We also select 8M English and 8M German monolingual sentences from newscrawl 2013 as monolingual dataset. For unsupervised En↔De, following Lample et al. (2018) , we choose 50M monolingual English and German sentences and use newstest 2016 as the test set for a fair comparison with the previous work. We preprocess the words into word-pieces in the same way as Wu et al. (2016) .

Model Architecture We use Transformer (Vaswani et al., 2017) as the basic model structure. For IWSLT En↔De, we use the transformer small configuration with 4 and 8 blocks to verify the generality of our model with different structures (a 4-block setting refer to transformer with 4 encoder blocks and 4 decoder blocks), where the word embedding dimension, hidden state dimension and non-linear layer dimension is set to be 256, 256 and 1024 respectively. For the WMT task, following Vaswani et al. (2017) , we use the transformer big setting with 6 blocks, where the word embedding dimension, hidden state dimension and non-linear layer dimension is 1024, 1024 and 4096 respectively. For the unsupervised NMT task, we choose transformer base setting following Lample et al. (2018) , where the above three dimensions are 512, 512 and 2048. The dropout rates for the three settings are 0.2, 0.1 and 0.1 respectively.

Optimization and Evaluation:

We choose Adam (Kingma & Ba, 2015) to optimize the network. The η in Algorithm 1 is set as 2 × 10 -4 . The learning rate decay rule and the two β's of Adam are the same as Vaswani et al. (2017) . For the three tasks, we use one, eight and four M40 GPUs to train those networks for three, five and six days respectively. Beam search is applied to generate translations for all the models, where the beam sizes for the three tasks are 6, 4 and 4 respectively.

The evaluation metric is BLEU score (Papineni et al., 2002) , which is the geometric mean of n-gram precisions, n ∈ {1, 2, 3, 4}.

Baselines We implement three types of important baselines3 , including back translation (Sennrich et al., 2015; Edunov et al., 2018a) , knowledge distillation (Kim & Rush, 2016) and the two-agent dual learning (He et al., 2016) . Take the f : X → Y translation as an example: 1) To use back translation, a reversed translation model g : Y → X is first trained on the original bilingual dataset, then the dataset {(g(y), y)|y ∈ D y } is constructed, which is then concatenated with the original dataset and train another model f . 2) To use knowledge distillation, we pre-train a teacher model f T : X → Y, and use it to generate the corresponding dataset {(x, f T (x))|x ∈ D x }, which is then mixed with the ground truth data and used to train a new model4 . For these two baselines, we generate sentences using both a single model and multiple models. 3) Two-agent dual learning, which uses f together with g to build the feedback signal to regularize the training (see Eqn. ( 1)), which we denote as "Dual-1". Our multi-agent models are denoted as "Dual-i" (i > 1)

3.3. RESULTS OF NMT ON IWSLT DATASET:

We regard the English and German sentences in the IWSLT14 bilingual corpus as D x and D y in Algorithm 1. We use 4-block networks (i.e. F γ in Algorithm 1) to generate translations, which are then used to train both 4-block (4B) and 8-block (8B) networks. We compare dual learning (Duali) with the standard baseline (Standard), knowledge distillation (KD-i) and back translation (BT-i) with different number of primal-dual model pairs i ∈ {1, 5}. Specifially, Dual-1 is the two-agent dual learning baseline, and Dual-5 is our multi-agent model with 4 additional pairs of agents.

The results of IWSLT En↔De are presented in Table 1 . We can observe that: 1) Dual learning has brought significant improvement over all the baselines (Standard, KD and BT) in both single-agent and multi-agent settings, demonstrating the effectiveness of dual learning; 2) Involving multiple agents into the learning system leads to better performances, which shows the importance of additional feedback signals from other agents. In particular, Dual-5 outperforms Dual-1 by around 0.5 BLEU in 4B setting and around 0.8 BLEU in 8B setting, which proves that dual learning can benefit from cooperating with more agents and demonstrates the effectiveness of our proposed algorithm. Particularly, dual learning with N = 5 agent pairs under 8B setting sets a new record of 35.44 BLEU in De→En and 29.52 BLEU in En→De translations. Study on different number of agents. We study the performances of our proposed algorithm with respect to different numbers of agent pairs, i.e., N and explore the optimal value of N . We enumerate within N = {1, • • • , 5} and did not try larger N values due to the limitation of computational resources. As we can see from Figure 1 , more agents can bring better performance for all settings. While it appears that the performance becomes better with more pair of agents involved into the learning system, the gain becomes more and more marginal (e.g., the maximum difference for N = 4 and N = 5 across all settings is around 0.1 BLEU). Furthermore, larger N leads to higher computational costs such as GPU resources. Thus, it is the most practical to leverage N = 3 agent pairs, where we can benefit from the substantial gain over baselines without too much computational costs. We employ N = 3 for the rest of our experiments.

Study on diversity of agents. We further study the influence of diversity among the agents. We show that diversity is important but not the focus of our algorithm. Detailed results and discussion is presented in Appendix A due to the space limitation. Study on translation tasks for other language pairs. We carry out another three groups of IWSLT14 experiments, English↔Spanish (Es), English↔Russian (Ru) and English↔Hebrew (He) translations to verify the generality of our framework on different languages. As is illustrated in Table 2, we achieve consistent improvements across different translation tasks (for all languages and all translation directions), further demonstrating the effectiveness and robustness of our algorithm. (2) additional monolingual data is provided. The results are summarized in Table 3 . From this table, we can see that the proposed algorithm achieves the best performances for all settings, and we observe that: 1) With WMT 2014 bilingual data only, the proposed multi-agent dual learning (Dual-3) achieves the state-of-the-art results to our knowledge, with 30.05 BLEU for En→De and 33.32 BLEU for De→En5 . 2) Compared with the traditional two-agent dual learning in He et al. (2016) , we can achieve 0.61 BLEU score improvement on En→De and 0.33 on De→En with bilingual data only. For the setting of using monolingual data, the improvements are 1.10 and 0.66 respectively. 3) Although KD and BT can bring improvements under different settings, the results are not as consistent as dual learning. For example, the performance of BT is not as good as KD under bilingual setting, but is better than KD under monolingual setting. We conjecture the reason is that with feedback signals to a generated sentence in the dual learning framework, the dataset is not only enlarged through sampling, but also guaranteed with good quality of generation.

Study on generality of the algorithm. We apply our method to the state-of-the-art systems on WMT 2016, 2017 and 2018 English-to-German translation challenge to verify the generality of our method. We work with the degenerated case where f 0 is warm started with a state-of-the-art model provided by FAIR6 and g 0 is fixed. We adopt the commercial system Google Translator (crawled on Oct 16, 2018), the champion system of WMT18 En→De: MS-Marian (Junczys-Dowmunt, 2018), and FAIR's single and ensemble models (Edunov et al., 2018a) as our baselines. The models are evaluated by sacreBLEU 7 . More details are left in Appendix B.

As can be seen from Table 4 , our method is capable of further improving the strong model by a large margin. Both single and ensemble models with our algorithm outperform the previous best systems and set state-of-the-art results on these tasks, demonstrating the effectiveness of our method. As is shown in Table 5 , with one pair of agents, KD-1 and BT-1 obtains almost the same results as the standard baseline, while Dual-1 achieves slightly better results on En→De translation. When we increase the number of agents, there are substantial improvements for KD-3, BT-3 and Dual-3. Especially, multi-agent dual learning (Dual-3) achieves 19.26 and 23.85 BLEU scores, setting new records for unsupervised NMT with pure NMT models.

4. APPLICATION TO IMAGE TRANSLATION:

In this section, We apply the multi-agent dual learning framework to image translation tasks. Different from Ciregan et al. which also uses multiple agents (several columns of deep neural networks), we leverage the duality of translation tasks and only use one model during inference. We follow the setting of CycleGAN (Zhu et al., 2017) , the most popular implementation of image translation that combines the ideas of GAN and dual learning.

Following the common practice in image translation (Zhu et al., 2017) , for each term in Eqn. (3), the ∆ x and ∆ y are specified as the L1 norm of pixel-level difference between two images. We set

α = β = ( 1 N , 1 N , • • • , 1 N ).

Since there is few literature focusing on generating consensus images using multiple models, we switch Eqn. (3) to a simpler form:

dual (D x , D y ; F α , G β ) = 1 |D x | x∈Dx x -G β (f 0 (x)) 1 + 1 |D y | y∈Dy y -F α (g 0 (y)) 1 , (11)

where D x and D y are the datasets of domain X and domain Y. The remaining parts of our model, including the model components, the objective functions, are exactly the same as the standard Cy-cleGAN. There are two discriminators d X : X → [0, 1] and d Y : Y → [0, 1] used to differentiate the generated images from natural images. The outputs of d X and d Y represent the probability that the input image is a natural one. The GAN loss (Goodfellow et al., 2014) is defined as

GAN = 1 |D x | x∈X [log d X (x) + log 1 -d Y (f 0 (x)) ] + 1 |D y | y∈Y [log d Y (y) + log 1 -d X (g 0 (y)) ].

During the optimization process, the two discriminators will try to maximize GAN , while the f 0 and g 0 will minimize dual (D x , D y ; F α , G β ) + λ GAN in cooperation with other provided agents, where λ is a hyper-parameter to be tuned. In the experiments, we follow Zhu et al. (2017) to set λ as 10.

4.2. SETTINGS:

We use the same datasets, model configuration and optimization algorithm as Zhu et al. (2017) . We implement our method based on CycleGAN8 . We choose two tasks, the photo↔painting translations where the art styles include Monet, Van Gogh, Ukiyo-e, Cezanne, and cityscapes label↔photo translation. All the images are cropped to 256 × 256. The configuration details can be found at Appendix D.1, which follows the same settings as Zhu et al. (2017) .

4.3. RESULTS OF PAINTING-TO-PHOTO TRANSLATION:

The task of painting→photo translation is to translate an input painting with specific art style to the original photo captured by a camera. We present case studies on painting→photo translations in Figure 2 . Indeed, there are several cases that our algorithm is clearly better than CycleGAN: For the images in the top-left corner, CycleGAN fails to recover the tree while our algorithm succeeds, with many details correctly included like the branches and leaves. In the bottom right corner, the texture of the ground is kept by our results, which is missing from the images generated by CycleGAN. We further present quantitative evaluation with Fréchet Inception Distance (briefly, FID), which is introduced in Heusel et al. (2017) and verified to be a reasonable metric by Lucic et al. (2018) , to evaluate the quality of painting→photo translations. FID captures the distance between generated images and the real ones, and a smaller FID score indicates the better model quality. As is shown in Table 6 , our method yields consistent improvements across all datasets. More results in photo→painting are left in Appendix D.2 due to space limitation. We present case studies and quantitative evaluation of label→photo translation on cityscapes images in Figure 3 and Table 7 . As can be seen from Figure 3 , both CycleGAN and our method can transfer the input label to a matching natural scene, while results of ours are clearer with less noise. For example, in the top left corner, our method can generate flatten roads, clearer cars and buildings and the baseline model fails. In Table 7 , we present quantitative evaluation with "FCN-scores" following Isola et al. (2017) . Our method achieves better results on the three evaluation metrics.

More details about FCN-score and results on photo→label translation are left in Appendix D.3. 

5. CONCLUSION:

In this paper, we proposed a new framework, multi-agent dual learning, in which multiple primal models and dual models are involved in the learning process. We empirically verified the effectiveness of the proposed framework on multiple machine translation tasks and image translation tasks.

Extending the multi-agent dual learning framework to more applications would be an interesting direction for future research. For example, we can apply our framework to improve the recent non-autoregressive machine translation, which is known to have high inference efficiency yet poor translation quality (Gu et al., 2018; Wang et al., 2019; Guo et al., 2019) . It is also worth studying how to apply the framework to other dual learning paradigms, including dual supervised learning (Xia et al., 2017b) and dual transfer learning (Wang et al., 2018) . Furthermore, it is challenging yet important to improve the training efficiency while maintaining the substantial improvements.

ACKNOWLEDGEMENT:

This work is partially supported by the National Science Foundation under Grant No. 1801652.

A STUDY ON DIVERSITY OF AGENTS:

We further study how the diversity among agents influences the performances of our algorithm on the IWSLT De↔En dataset. We compare three group of agents: (1) Agents with independent runs ("Independent Run"): We use the same agents as Section 3. We train 8-block (8B) networks as Section 3.3 with N = {1, ..., 5} pairs of agents from the above three groups respectively. The same training and inference procedures as Section 3.3 are applied here. As is shown from results in Figure 4 , we have the following observations and conclusions:

1. Diversity among agents plays an important role in our method. The Homogeneous agents with poor diversity has the worst performances among the three groups. That is, introducing more homogeneous agents brings very marginal gain.

2. A more diverse group of agents can potentially bring further improvements, yet diversity is not the key focus of our algorithm. Independent Run achieves comparable performances with Different Arch, suggesting that our framework can achieve substantial improvements with a reasonable level of diversity among agents.

B DETAILS ON WMT18 EN→DE CHALLENGES:

For the study on generality of the algorithm on WMT18 En→De challenges, the training data, F γ (•) and G γ (•) are the same as those used in Section 3.4. We work with a degenerated case where g 0 (i.e., the De→En model) is pre-trained before optimizing f 0 and subsequently hold constant. The f 0 is warm started by the models from https://dl.fbaipublicfiles.com/fairseq/ models/wmt18.en-de.ensemble.tar.bz2, which are well trained with tremendous resources. We degenerate our algorithm to optimize the single direction due to the limitation that only En→De model checkpoints are released.

We train the model on 8 M40 GPUs for 2 days with PyTorch implementation of our algorithm based on FairSeq Toolkit9 . The batch size is 4096 tokens. We use Adam optimizer and the learning rate is automatically calculated by the system based on the reloaded checkpoint. The dropout ratio is 0.3. We use beam search with beam size 5 to generate candidates.

C INTRODUCTION TO UNSUPERVISED NMT:

We briefly introduce the recent state-of-the-art unsupervised NMT algorithm in (Lample et al., 2018) . Similar to the standard NMT model with bilingual training data, an unsupervised NMT model is also based on the encoder-decoder architecture. The source sentences and target sentences are mapped into a same vocabulary using BPE techniques. The shared embedding is pretrained with fastText Bojanowski et al. (2017) to get good initial values. The proposed model can handle both source-to-target translation and target-to-source translation like Johnson et al. (2017) .

The training loss usually consists of two parts, a language model loss and a back-translation loss.

Let P X →Y denote the translation from space X to space Y, and so for the other similar notations.

( We follow Zhu et al. (2017) to configure the model architectures for all tasks on image-to-image translation. For the generator, the network contains 2 stride-2 convolutions, 9 residual blocks, and 2 convolutional layers with stride 1 2 , (i.e., the transposed convolutional layers). Instance normalization is applied to the network. For the discriminator, the 70 × 70 PatchGANs (Isola et al., 2017) , which aim to classify whether 70 × 70 overlapping image patches are real or fake. Such a patch-level discriminator architecture has fewer parameters than a full-image discriminator and can work on arbitrarily-sized images in a fully convolutional fashion.

D.2 RESULTS OF PHOTO-TO-PAINTING TRANSLATION:

We present results on photo→painting translation in Figure 5 . As we can see, both standard Cycle-GAN and our algorithm successfully translates the inputs to the corresponding target domains. Our model outperforms the baseline in two aspects: 1) multi-agent dual learning can generate clearer images with less block artifact (the Monet column and Ukiyo-e column); and 2) the images generated by our algorithm are faithful to the original images in semantics. For example, the sun is missing from CycleGAN generated images in the top Ukiyo-e column and the bottom Cezanne column, while our algorithm is able to keep the sun.

D.3 RESULTS OF CITYSCAPES PHOTO-TO-LABEL TRANSLATION:

FCN Evaluation FCN-scores are used in Zhu et al. (2017) and Isola et al. (2017) to quantitatively evaluate Cityscape photo to semantic label translation. A pre-trained fully convolutional network (Long et al., 2015) , briefly, FCN, is used to convert the photo to the corresponding semantic label, and then calculate the per-pixel accuracy, per-class accuracy and mean class Intersection-Over-Union(Class IOU) (Cordts et al., 2016) against the ground truth labels.

Results on photo→label translation:

The example cases and quantitative evaluation with FCNscores of photo→labels on cityscapes images are shown in Figure 6 and Table 8 10 respectively. We

:

Although with the same term, the "multi-agent" or "agent" in this paper has no relationship with multiagent reinforcement learning

2 The numbers of agents served for two mapping directions are not necessarily the same. For the tasks where the two domains X and Y are fully symmetric, like the language spaces in NMT and image collections in image-to-image translation, learning the two mappings are of similar difficulty. Therefore, we use the same number of agents to assist training in this paper, and leave the study of asymmetric setting to the future work.

All the implementations are based on the official tensor2tensor release: https://github.com/ tensorflow/tensor2tensor.

We have tried both mixing with ground truth and not and found that mixing is better. On IWSLT 2014 De→En, with/without mixing can achieve 33.89/33.14 BLEU scores.

Although Edunov et al. (2018a) provides a higher BLEU score, they use WMT'18 dataset, which is different from ours and currently not widely used for WMT14 En↔De tasks in NMT literature.

https://s3.amazonaws.com/fairseq-py/models/wmt18.en-de.ensemble.tar.bz2

17,18}+tok.13a+version.1.2.11

https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix

https://github.com/pytorch/fairseq

The baseline performance is different fromZhu et al. (2017) since it is unclear how the RGB based labels is converted to integer labels without the official evaluation code released. This is a common issue at https://github.com/phillipi/pix2pix/issues/115.

Figures and Tables:

|y ∈ D y }; Compute the probabilities P (ŷ|x; F γ ) and P (x|y; G γ ); while not converged do Randomly sample two batches of B xŷ ⊂ D x × Dy and B xy ⊂ Dx × D y , each of size K; Following Eqn. (9) and Eqn. (10), calculate Grad f0 and Grad g0 as follows:

Figure 1: BLEU scores on IWSLT De↔En w.r.t the number of agents pairs.

Figure 2: Four groups of experiments on painting→photo translations.

Figure 3: Cityscapes labels → photos translation.

3, which have the same model architecture and hyperparameters, and are trained through independent runs with different random seeds. (2) Agents with different architectures ("Different Arch"): The agents have different hidden dimensions (256, 512) with different numbers of heads (2, 4). They are also trained through independent runs with different random seeds. Intuitively, this group of agents with different architectures could be more diverse than the first group. (3) Agents from a same run ("Homogeneous"): In comparison to the first two groups where different methods (i.e., different random seeds, different model architecture) are used to encourage diversity among agents, in this group, we use model checkpoints at different (but very close) iterations from a same run. At the late stage of training, the learning rate becomes smaller. Thus the diversity among agents would naturally be very poor.

Figure 4: BLEU scores on IWSLT De↔En with different pools of agents.

including using various random seeds to affect weight initialization and input order of the training samples, different model architectures, or training over different subsets. The same holds for g i

Table content: <table/>

BLEU scores on IWSLT De↔En translation. "KD", "BT" and "Dual" stands for knowledge distillation, back translation and dual learning respectively.

Table content: <table><tr><td/><td colspan="3">Standard KD-1 KD-5 BT-1 BT-5 Dual-1 Dual-5</td></tr><tr><td>De→En (4B)</td><td>33.42</td><td>33.89 34.20 33.71 33.61 34.25</td><td>34.70</td></tr><tr><td>En→De (4B)</td><td>27.89</td><td>28.45 28.65 28.35 28.22 28.63</td><td>28.99</td></tr><tr><td>De→En (8B)</td><td>34.01</td><td>34.36 34.85 33.87 33.77 34.57</td><td>35.44</td></tr><tr><td>En→De (8B)</td><td>27.95</td><td>28.74 29.18 28.28 28.25 29.07</td><td>29.52</td></tr></table>

BLEU scores on the translations between {Es, Ru, He} and En.

Table content: <table><tr><td/><td colspan="6">Es→En En→Es Ru→En En→Ru He→En En→He</td></tr><tr><td>Standard</td><td>40.50</td><td>37.95</td><td>18.94</td><td>16.06</td><td>33.25</td><td>22.20</td></tr><tr><td>Dual-1</td><td>42.14</td><td>38.09</td><td>21.14</td><td>16.31</td><td>35.42</td><td>22.75</td></tr><tr><td>Dual-3</td><td>42.41</td><td>38.62</td><td>21.56</td><td>16.71</td><td>36.08</td><td>23.80</td></tr><tr><td>3.4 RESULTS</td><td/><td/><td/><td/><td/><td/></tr></table>

BLEU scores on WMT14 En↔De translation. "Bitext" and "Mono" respectively represents using bilingual data only and using the mix of bilingual&monolingual data.

Table content: <table><tr><td/><td colspan="3">Standard KD-1 KD-3 BT-1 BT-3 Dual-1 Dual-3</td></tr><tr><td>En→De (Bitext)</td><td>28.4</td><td>29.17 29.28 28.26 27.79 29.44</td><td>30.05</td></tr><tr><td>De→En (Bitext)</td><td>32.15</td><td>32.43 31.38 32.41 32.68 32.99</td><td>33.32</td></tr><tr><td>En→De (Mono)</td><td>28.4</td><td>29.20 29.36 29.42 29.68 29.93</td><td>31.03</td></tr><tr><td>De→En (Mono)</td><td>32.15</td><td>32.28 32.43 34.58 34.77 34.98</td><td>35.64</td></tr></table>

BLEU scores on WMT {16, 17, 18} En→De. The single models are independently trained for six times, with mean and standard derivation values reported; the ensemble models are the ensemble results of all different runs.

Table content: <table><tr><td/><td/><td>2016</td><td>2017</td><td>2018</td></tr><tr><td colspan="2">Google Translator</td><td>38.03</td><td>31.41</td><td>47.67</td></tr><tr><td colspan="2">FAIR (Single)</td><td>37.04 ± 0.16</td><td>31.86 ± 0.21</td><td>44.63 ± 0.12</td></tr><tr><td colspan="2">Ours (Single)</td><td colspan="3">40.68 ± 0.11 33.47 ± 0.16 48.89 ± 0.13</td></tr><tr><td colspan="2">MS-Marian (Ensemble)</td><td>39.6</td><td>31.9</td><td>48.3</td></tr><tr><td colspan="2">FAIR (Ensemble)</td><td>37.99</td><td>32.80</td><td>46.05</td></tr><tr><td colspan="2">Ours (Ensemble)</td><td>41.23</td><td>34.01</td><td>49.61</td></tr><tr><td colspan="4">3.5 RESULTS OF NMT WITH MONOLINGUAL DATA ONLY</td></tr><tr><td colspan="5">Unsupervised NMT is studied recently to learn two translation models without bilingual data (details</td></tr><tr><td colspan="5">in Appendix C). We pre-train two unsupervised NMT models with different initialization, use them</td></tr><tr><td colspan="5">to translate the 50M monolingual sentences, and apply KD, BT and dual learning algorithms.</td></tr><tr><td colspan="5">Table 5: BLEU scores on WMT 2016 unsupvised NMT En↔De translation.</td></tr><tr><td/><td colspan="4">Standard KD-1 KD-3 BT-1 BT-3 Dual-1 Dual-3</td></tr><tr><td>En→De</td><td>17.52</td><td colspan="3">17.33 18.27 17.50 18.15 18.01</td><td>19.26</td></tr><tr><td>De→En</td><td>22.12</td><td colspan="3">22.10 22.71 22.51 22.91 22.17</td><td>23.85</td></tr></table>

FID scores on painting→photos translation.

Table content: <table><tr><td/><td colspan="3">Cezanne Monet Ukiyo-e Vangogh</td></tr><tr><td>CycleGAN</td><td>189.40</td><td>134.25 197.30</td><td>91.19</td></tr><tr><td>Ours</td><td>173.92</td><td>129.86 184.69</td><td>89.32</td></tr><tr><td colspan="3">4.4 RESULTS OF CITYSCAPES LABEL-TO-PHOTO TRANSLATION</td><td/></tr></table>

FCN-scores on cityscapes labels→photos translation. Per-pixel Acc. Per-class Acc. Class IOU

Table content: <table><tr><td>CycleGAN</td><td>0.52</td><td>0.17</td><td>0.11</td></tr><tr><td>Ours</td><td>0.57</td><td>0.19</td><td>0.14</td></tr></table>

1) Language model loss, implemented by a denoising autoencoder. Mathematically,L 1 =E x∼X [-log P X →X (x|C(x))] + E y∼Y [-log P Y→Y (y|C(y))],(12)where C(•) is a noise model with randomly dropping several words, swapping words, etc.(2) Back translation loss, implemented by back-translating the monolingual data and feeding into the reversed models. Mathematically,L 2 =E x∼X [-log P Y→X (x|ŷ(x))] + E y∼Y [-log P X →Y (y|x(y))],(13)in which ŷ(x) = arg max u∈Y P X →Y (u|x) and x(y) = arg max v∈X P Y→X (v|y). Note that the four P ••• 's are implemented in a single encoder-decoder based model, where each translation task has a different "task embedding", i.e., a learnable vector indicating the translating directions.

Table content: <table/>

Peer review (iclr2019):
 Summary The paper proposes to modify the "Dual Learning" approach to supervised (and unsupervised) translation problems by making use of additional pretrained mappings for both directions (i.e. primal and dual). These pre-trained mappings ("agents") generate targets from the primal to the dual domain, which need to be mapped back to the original input. It is shown that having >=1 additional agents improves training of the BLEU score in standard MT and unsupervised MT tasks. The method is also applied to unsupervised image-to-image "translation" tasks. Positives and Negatives +1 Simple and straightforward method with pretty good results on language translation. +2 Does not require additional computation during inference, unlike ensembling. -1 The mathematics in section 3.1 is unclear and potentially flawed (more below). -2 Diversity of additional "agents" not analyzed (more below). -3 For image-to-image translation experiments, no quantitative analysis whatsoever is offered so the reader can*t really conclude anything about the effect of the proposed method in this domain. -4 Talking about "agents" and "Multi-Agent" is a somewhat confusing given the slightly different use of the same term in the reinforcement literature. Why not just "mapping" or "network"? -1: Potential Issues with the Maths. The maths is not clear, in particular the gradient derivation in equation (8). Let*s just consider the distortion objective on x (of course it also applies to y without loss of generality). At the very least we need another "partial" sign in front of the ".delta" function in the numerator. But again, it*s not super clear how the paper estimates this derivative. Intuitively the objective wants f_0 to generate samples which, when mapped back to the X domain, have high log-probability under G, but its samples cannot be differentiated in the case of discrete data. So is the REINFORCE estimator used or something? Not that the importance sampling matter is orthogonal. In the case of continuous data x, is the reparameterization trick used? This should at the very least be explained more clearly. Note that the importance sampling does not affect this issue. -2: Diversity of Agents. As with ensembles, clearly it only helps to have multiple agents (N>2) if the additional agents are distinct from f_1 (again without loss of generality this applies to g as well). The paper proposes to use different random seeds and iterate over the dataset in a different order for distinct pretrained f_i. The paper should quantify that this leads to diverse "agents". I suppose the proof is in the pudding; as we have argued, multiple agents can only improve performance if they are distinct, and Figure 1 shows some improvement as the number of agents are increase (no error bars though). The biggest jump seems to come from N=1 -> N=2 (although N=4 -> N=5 does see a jump as well). Presumably if you get a more diverse pool of agents, that should improve things. Have you considered training different agents on different subsets of the data, or trying different learning algorithms/architectures to learn them? More experiments on the diversity would help make the paper more convincing.