Title: BILINGUAL-GAN: NEURAL TEXT GENERATION AND NEURAL MACHINE TRANSLATION AS TWO SIDES OF THE SAME COIN

Abstract: Latent space based GAN methods and attention based encoder-decoder architectures have achieved impressive results in text generation and Unsupervised NMT respectively. Leveraging the two domains, we propose an adversarial latent space based architecture capable of generating parallel sentences in two languages concurrently and translating bidirectionally. The bilingual generation goal is achieved by sampling from the latent space that is adversarially constrained to be shared between both languages. First an NMT model is trained, with back-translation and an adversarial setup, to enforce a latent state between the two languages. The encoder and decoder are shared for the two translation directions. Next, a GAN is trained to generate 'synthetic' code mimicking the languages' shared latent space. This code is then fed into the decoder to generate text in either language. We perform our experiments on Europarl and Multi30k datasets, on the English-French language pair, and document our performance using both Supervised and Unsupervised NMT.

1. INTRODUCTION:

Neural machine translation (NMT) and neural text generation (NTG) are among the pool of successful NLP tasks handled by neural approaches. For example, NMT has acheived close to human-level performance using sequence to sequence models, which tries to solve the translation problem endto-end. NTG techniques can be categorized into three classes: Maximum Likelihood Estimation based, GAN-based and reinforcement learning (RL)-based. Recently, researchers have extensively used GANs (Goodfellow et al., 2014) as a potentially powerful generative model for text (Yu et al., 2017) , because of their great success in the field of image generation.

Inspired by human bilingualism, this work proposes a Bilingual-GAN agent, capable of deriving a shared latent space between two languages, and then leveraging that shared space in translation and text generation in both languages. Currently, in the literature, neural text generation (NTG) and NMT are treated as two independent problems; however, we believe that they are two sides of the same coin and could be studied jointly. Emerging latent variable-based techniques can facilitate unifying NTG and NMT and the proposed Bilingual-GAN will be a pioneering attempt in this direction.

Learning latent space manifold via adversarial training has gained a lot of attention recently (Schwenk & Douze, 2017) ; text generation (Zhao et al., 2017) and unsupervised NMT (Lample et al., 2017) are among these examples where autoencoder (AE) latent space manifolds are learned adversarially. For NTG, in Adversarially Regularized Autoencoders (ARAE) work (Zhao et al., 2017) , a critic-generator-autoencoder combo is proposed to tackle the non-differentiability problem rising due to the discrete nature of text. The ARAE approach is to learn the continuous manifold of the autoencoder latent space and generate samples from it instead of direct synthesis of discrete (text) outputs. Output text is then reconstructed by the decoder from the generated latent samples, similarly to the autoencoding process.

Adversarial learning of autoencoders' latent manifold has also been used for unsupervised NMT (Lample et al., 2017; 2018b; Yang et al., 2018; Artetxe et al., 2017b) . In Lample et al. (2017) , a single denoising autoencoder is trained to derive a shared latent space between two languages using different loss functions. One of their objectives adversarially enforces the latent space generated by the encoders of the different languages to become shared and difficult to tell apart. Other objectives are autoencoder reconstruction measures and a cross-domain cost closely related to backtranslation (Sennrich et al., 2015b) terms.

The contribution of this paper is to propose a latent space based architecture as a bilingual agent handling text generation and machine translation simultaneously. We demonstrate that our method even works when using complex multi-dimensional latent representations with attention based decoders, which weren't used in Zhao et al. (2017) .

2. RELATED WORK:



2.1. LATENT SPACE BASED UNMT:

Neural Machine Translation (Kalchbrenner & Blunsom, 2013; Sutskever et al., 2014; Cho et al., 2014; Bahdanau et al., 2014; Vaswani et al., 2017) constitutes the state-of-the-art in translation tasks for the majority of language pairs. On the unsupervised side, a few works Lample et al. (2017) ; Artetxe et al. (2017a) ; Lample et al. (2018a) have emerged recently to deal with neural machine translation without using parallel corpora, i.e sentences in one language have no matching translation in the other language. They all have a similar approach to unsupervised neural machine translation (UNMT) that uses an encoder-decoder pair sequence-to-sequence model that is shared between the languages while trying to find a latent space common to both languages. They all make use of back-translation (Sennrich et al., 2015b) needed for the unsupervised part of the training. Lample et al. (2017) use a word by word translation dictionary learned in an unsupervised way (Conneau et al., 2017a) as part of their back-translation along with an adversarial loss to enforce language Independence in the latent code space. They later improve their model (Lample et al., 2018a) by removing these two elements and instead using a BPE sub-word tokenization (Sennrich et al., 2015a) with embeddings learned using FastText (Bojanowski et al., 2017) so that the sentences are embedded in a common space. Artetxe et al. (2017a) have a similar flavour but uses some crosslingual embeddings to embed sentences in a shared space. They also decouple the decoder so that one is used per language.

2.2. LATENT SPACE-BASED NTG:

Researchers have conventionally utilized GAN framework in image applications (Salimans et al., 2016) with great success. Inspired by their success, a number of works have used GANs in various NLP applications such as machine translation (Wu et al., 2017; Yang et al., 2017a) , dialogue models (Li et al., 2017) , question answering (Yang et al., 2017b) , and natural language generation (Gulrajani et al., 2017; Kim et al., 2017) . However, applying GAN in NLP is challenging due to the discrete nature of text. Consequently, back-propagation would not be feasible for discrete outputs and it is not straightforward to pass the gradients through the discrete output words of the generator. A latent code-based solution for this problem was proposed in Kim et al. (2017) , where a latent representation of the text is derived using an AE and the manifold of this representation is learned via adversarial training of a generator. Another version of the ARAE method with updating encoder, based on discriminator loss function was also introduced in (Spinks & Moens, 2018) .

3. METHODOLOGY:

The Bilingual-GAN comprises of two main components: a translation unit and a text generation unit. The complete architecture is described in Figure 1 . The middle left rectangle unit represents the text generation unit and the remaining part represents the translation unit.

3.1. TRANSLATION UNIT:

The translation system is a sequence-to-sequence model with an encoder and a decoder extended to support two languages. This first translation component is inspired by the unsupervised neural machine translation system by Lample et al. (2017) . We have one corpus in language 0 and another Figure 1 : The complete architecture for our Bilingual GAN in language 1 (they need not be translations of each other), an encoder and a decoder shared between the two languages.

The loss function which is used to compare two sentences is the same as the standard sequenceto-sequence loss: the token wise cross-entropy loss between the sentences, that we denote by ∆(sentence 1, sentence 2). For our purpose, let s li be a sentence in language i with i ∈ {0, 1}. The encoding of sentence s li is denoted by enc (s li ) in language i which is used as the word embeddings of language i to convert the input sentence s li . Similarly, denote by dec (x, l i ) the decoding of the code x (typically an output of the encoder) into language l i using the word embeddings of target language i to convert into words. Then, the system is trained with three losses aimed to allow the encoder-decoder pair to reconstruct inputs (reconstruction loss), to translate correctly (cross-domain loss) and for the encoder to encode language independent codes (adversarial loss). The losses are applied for every batch for both languages.

Reconstruction Loss This is the standard auto-encoder loss which aims to reconstruct the input:

L recon = ∆   sl i , ŝl i := dec (enc (s li ) , l i )   

This loss can be seen in figure 2 .

Cross-Domain Loss This loss aims to allow translation of inputs. It is similar to back-translation (Sennrich et al., 2015b) . For this loss, denote by transl (s li ) the translation of sentence s li from language i to language 1 -i. The implementation of the translation is explained in subsection 3.1.1 when we address supervision.

EQUATION

In this loss, we first translate the original sentence s li into the other language and then check if we can recreate the original sentence in its original language. This loss can be seen in figure 2 .

Adversarial Loss This loss is to enforce the encoder to produce language independent code which is believed to help in decoding into either language. This loss has been defined adversarially. Let D be a discriminator where D(c) is a prediction for the language of the sentence that was used to create code c (typically the output of an encoder), 0 if the sentence is in language 0 and 1 if the sentence is in language 1. We thus have for the discriminator D the following

L D = max{D(enc (s li )) -D(enc s lj )}

and for its adversary, the encoder, the opposite:

L enc = min{D(enc (s li )) -D(enc s lj )}

Input Noise In order to prevent the encoder-decoder pair to learn the identity function and to make the pair more robust to noise, noise is added to the input of the encoder. This is illustrated in figure 2 where you see the + noise atop the arrows feeding into the encoder. On the input sentences, the noise comes in the form of random word drops (we use a probability of 0.1) and of random shuffling but only moving each word by at most 3 positions. This is also the noise scheme that Lample et al.

(2017) use in their work. We also add a Gaussian nose of mean 0 and standard deviation of 0.3 to the input of the decoder.

Figure 2 : The translation unit of the Bilingual-GAN.

3.1.1. SUPERVISION:

Recall that in the cross-domain loss above, equation 1, the translation function transl (s li ) was used to translate the sentence s li from language i to language 1 -i. In fact, the choice of this function directly affects the amount of supervision in the trained model. Indeed, notice that only s li and transl (s li ) are used in the losses.

If the translation function transl () is a lookup of a word-by-word translation dictionary learned in an unsupervised fashion as in Conneau et al. (2017a) , then the whole system is trained in an unsupervised manner since we have no groundtruth information about s li . After a couple of epochs, the encoder-decoder model should be good enough to move beyond simple word-by-word translation so then the translation function can be changed to using the model itself to translate input sentences. This is what's done in Lample et al. (2017) where they change the translation function from wordby-word to model prediction after 1 epoch. In our case, we get the word-by-word translation lookup table by taking each word in the vocabulary and looking up the closest word in the other language in the multilingual embedding space created by Conneau et al. (2017b) .

If the translation function transl () is able to get the groundtruth translation of the sentence, for example if we have an aligned dataset, then transl (s li ) = s lj which is encoded and decoded into the original language i and compared with s li getting the usual supervised neural machine translation loss. However, note that this supervision is only one way since you learn to predict in language i given a sentence in language j. We refer to this level of supervision as Half-Supervised in our results section later. In order to have supervision both ways, one would need to have both s li and s lj in the training corpus, this is what we refer to as the Supervised level.

3.1.2. EMBEDDINGS:

There are a few choices for embedding the sentence words before feeding into the encoder. We experiment with a few and show the results in section 4.3. In particular, we use randomly initialized embeddings, embeddings trained with FastText (Bojanowski et al., 2017) and both pretrained and self-trained cross-lingual embeddings (Conneau et al., 2017b) .

3.1.3. SYSTEM SPECIFICATIONS:

Here we show the exact specifications and training optimizers for the translation part of the Bilingual-GAN. The embeddings have size 300, the encoder consists of either 1 or 2 layers of 256 bidirectional LSTM cells, the decoder is equipped with attention (Bahdanau et al., 2014) and consists of a single layer of 256 LSTM cells. The discriminator, when the adversarial loss is present, is a standard feed-forward neural network with 3 layers of 1024 cells with ReLU activation and one output layer of one cell with Sigmoid activation.

We used Adam with a β 1 of 0.5, a β 2 of 0.999, an of 10 -8 and a learning rate of 0.0003 to train the encoder and the decoder whereas we used RMSProp with a learning rate of 0.0005 to train the discriminator. Most of the specifications here were taken from Lample et al. (2017) .

3.2. TEXT GENERATION UNIT:

First, we pre-train our NMT system 3.1. The NMT system learns a shared latent space (c x , c y ) for the two language directions, and this shared latent space is enforced by a GAN setup between a critic and the encoders, and through back-translation (Sennrich et al., 2015b) . Then, a bilingual generator is trained adversarially to learn the manifold of the shared latent space (c x , c y ), which is learned in the NMT system. It is trained similar to a modified version of ARAE (Spinks & Moens, 2018) to generate codes ĉ which mimic the samples from the shared latent space. Once GAN training is finished, the decoders of the NMT system can be used to generate parallel bilingual sentences by decoding the generator output code, ĉ.

The proposed bilingual generator is a GAN (Goodfellow et al., 2014) trained to learn the hidden state manifold of the RNN-based encoder as in Zhao et al. (2017) .

We used Wasserstein GAN gradient penalty (WGAN-GP) (Gulrajani et al., 2017) approach in our experiments as:

(2)

L = E ĉ∼Pg [D(ĉ)] -E c∼Pr [D(c)] + λE c∼Pḡ [(||∇ cD(c)|| 2 -1) 2 ])

where

[c ∼ P ḡ (c)] ← α [c ∼ P r (c)] + (1 -α) [ĉ ∼ P g (ĉ)

] and it is a random latent code obtained by sampling uniformly along a line connecting pairs of the generated code and the encoder output. P r is the distribution of the encoder output data, c represents the latent 'code' or the latent space representations of the input text, P g is the distribution of the generated output data, ĉ represents the generated code representations, and λ is the gradient penalty term. We used λ = 10 (Gulrajani et al., 2017).

3.2.1. TRAINING:

In order to train the GAN, we used the encoder output of our NMT system as 'real' code. The encoder output is a latent state space matrix which captures all the hidden states of the LSTM encoder. We then generate noise which is fed into a generator neural network comprising 1 linear layer and 5 convolutional layers to produce a 'mimicked' or 'fake' code matrix. The 'real' code and the fake code are then fed into the discriminator neural network, which also consists of 5 convolutional and 1 linear layer. The discriminator output is used to calculate the generator and discriminator losses. The losses are optimized using Adam (Kingma & Ba, 2014) . Unlike the GAN update in Gulrajani et al. (2017) , we use 1 discriminator update per generator update. We have seen that by increasing the number of discriminator updates per generator update did not improve model training.

In one training iteration, we feed both an English and a French sentence to the encoder and produce two real codes. We generate one fake code by using the generator and calculate losses against both the real codes. We average out the two losses. Although, the NMT is trained to align the latent spaces and we can use just one language to train the GAN, we use both real codes to reduce any biases in our NMT system. We train our GAN on both the supervised and unsupervised NMT scenarios. In the supervised scenario, we feed English and French parallel sentences in each training iteration. In the unsupervised scenario, we ensure the sentences are not parallel.

Once the GAN is trained, the generator code can be decoded in either language using the pre-trained decoders of the NMT system.

3.2.2. MATRIX-BASED CODE REPRESENTATION:

In latent-space based text generation, where the LSTM based encoder-decoder architectures do not use attention, a single code vector is generally employed which summarizes the entire hidden se-quence (Zhao et al., 2017) . A variant of the approach is to employ global mean pooling to produce a representative encoding (Semeniuta et al., 2018) . We take advantage of our attention based architecture and our bi-directional encoder to concatenate the forward and backward latent states depth-wise and produce a code matrix which can be attended to by our decoder. The code matrix is obtained by concatenating the latent code of each time steps. Consequently, the generator tries to mimic the entire concatenated latent space. We found that this richer representation improves the quality of our sentence generation.

4. EXPERIMENTS:

This section presents the different experiments we did, on both translation and generation, and the datasets we worked on.

4.1. DATASETS:

The Europarl and the Multi30k datasets have been used for our experimentation. The Europarl dataset is part of the WMT 2014 aligned corpora (Koehn, 2005) while the Multi30k dataset is one used for a captioning task (Elliott et al., 2017) and consists of images and their captions. We only use the French and English pair.

As preprocessing steps on the Europarl dataset, we removed sentences longer than 20 words and those with a ratio of number of words between translations is bigger than 1.5. Then, we tokenize the sentence using the Moses tokenizer (Koehn et al., 2007) . For the Multi30k dataset, we use the supplied tokenized version of the dataset with no further processing. For the BPE experiments, we use the sentencepiece subword tokenizer by Google (Sennrich et al., 2015a) . BPE is a subword tokenization method used sentences. Consequentially, the decoder also predicts subword tokens. This results in a common embeddings table for both languages since English and French share the same subwords. The BPE was trained on the training corpora that we created.

For the training, validation and test splits, we used 200k randomly chosen sentences for the Europarl dataset for training and 40k sentences for testing. When creating the splits for unsupervised training, we make sure that the sentences taken in one language have no translations in the other language's training set by randomly choosing different sentences for each of them with no overlap. For the validation set in that case, we chose 80k sentences. In the supervised case, we randomly choose the same sentences in both languages with a validation set of 40k. For the Multi30k dataset, we use 12 850 and 449 sentences for training and validation respectively for the unsupervised case and the whole provided split of 29k and 1014 sentences for training and validation respectively. In both cases, the test set is the provided 1k sentences Flickr 2017 one. For the hyperparameter search phase, we chose a vocabulary size of 8k for the Europarl, the most common words appearing in the training corpora and for the final experiments with the best hyperparameters, we worked with a vocabulary size of 15k. For Multi30k, we used the 6800 most common words as vocabulary.

4.2. QUANTITATIVE EVALUATION METRICS:

Translation BLEU We calculate the BLEU-N score according to the following equation (Papineni et al., 2002) :

BLEU-N = BP • exp( N n=1 w n log(p n )) (3)

where p n is the probability of n-gram and w n = 1 n . The BP is set to 1 as we translated to the fixed length sentences in both directions. We report the results of BLEU-4 in Table 1 and 4 . Generation BLEU We also use the BLEU-N scores to evaluate the generated sentences. Here, we set BP to 1 as there is no reference lengths like in machine translation. The results is described in Table 2 . For the evaluations, we generated 40 000 sentences for the model trained on Europarl and 1 000 on the model trained on Multi30k.

Perplexity is used to evaluate the fluency of the generated sentences. For the perplexity evaluations, we generated 100 000 and 10 000 sentences for the Europarl and the Multi30k datasets respectively. The forward and reverse perplexities of the LMs trained with maximum sentence length of 20 and 15 using the Europarl and he Multi30k datasets respectively are described in Table 3 . The forward perplexities are calculated by training an RNN language model (Zaremba et al., 2015) on real training data and evaluated on the generated samples. This measure describe the fluency of the synthetic samples. We also calculated the reverse perplexities by training an RNNLM on the synthetic samples and evaluated on the real test data. The results are illustrated in Table 3 .

4.3. TRANSLATION:

A lot of hyperparameters were used in our experiments and to keep the restults table compact, we abbreviated a few. We first explain the shorthands before going to the discussion of the results.

The levels of supervision has been explained in the previous section 3.1.1. MTF stands for model translation from and is the epoch at which we stop using the transl () function and instead start using the model. NC stands for a New Concatenation method we used to combine the bidirectional encoder output: either we concatenate the forward and backward states lengthwise to get as many output vectors as twice the sentence length but each of them has dimension equal to the number of encoder cells (old) or depthwise to get the same number of output vectors as the sentence length but each vectors is twice the size of the number of encoder cells (new). FastText refers to the use of FastText (Bojanowski et al., 2017) to train our embeddings, Xlingual refers to the use of cross-lingual or multilingual embeddings using Conneau et al. (2017b) either trained on our own (Self-Trained) or using the pretrained (Pretrain.) ones and BPE refers to the use of subword tokenization (Sennrich et al., 2015a) with the tokens and the embeddings learned as in Sennrich et al. (2015a) . NoAdv refers to not using the adversarial loss to train, i.e. we do not enforce language independance in the code space through the adversarial loss, 2Enc refers to using a 2 layers of 256 cells each bidirectional LSTM encoder. This section of the results focuses on the scores we have obtained while training the neural machine translation system. The main results table will show the BLEU scores for translation on a held out test set for the WMT'14 Europarl corpus and for the official Flickr test set 2017 for the Multi30k dataset. From the results table, we notice first from Table 1 : The *'ed experiments use a vocabulary size of 15k words. The Multi30k experiments use a vocabulary size of 6800 words. lines 4 and 6 that removing the adversarial loss helps the model. This is probably what motivated the removal of the adversarial loss in Lample et al. (2018a) It's possible that the reconstruction and the cross-domain losses are enough to enforce a language independent code space. Lines 5 and 6 show that using 2 layers for the encoder is beneficial but that was to be expected. Lines 6 and 7 show that the new concatenation method improved upon the model. A small change for a small improvement that may be explained by the fact that both the forward and the backward states are combined and explicitly represent each word of the input sentence rather than having first only the forward states and then only the backward states. Surprisingly, BPE gave a bad score on English to French (line 8). We think that this is due to French being a harder language than English but the score difference is too big to explain that. Furter investigation is needed. Line 10 shows good results with trainable FastText embeddings trained on our training corpora. Perhaps using pre-trained ones might be better in a similar fashion as pretrained cross-lingual embeddings helped over the self-trained ones as in lines 11 and 13. Lines 11 and 14 also show the importance of letting the embeddings change during training instead of fixing them.

4.4. TEXT GENERATION:

We evaluated text generation on both the fluency of the sentences in English and French and also on the degree to which concurrently generated sentences are valid translations of each other. We fixed our generated sentence length to a maximum of length 20 while training on Europarl and to a maximum of length 15 while training on Multi30k. We measured our performance both on the supervised and unsupervised scenario. The supervised scenario uses a pre-trained NMT trained on parallel sentences and unsupervised uses a pre-trained NMT trained on monolingual corpora.

Generation BLEU scores are measured using the two test sets. The results are described in and French. We can note that the English sentences have a higher BLEU score which could be a bias from our NMT. We can also note that lower BLEU scores for the Multi30k because of the smaller test size.

Perplexity result is described in Table 3 . The perplexities of the LMs using real data are 140.22 (En), 136.09 (Fr) and 59.29 (En), 37.56 (Fr) for the Europarl and the Multi30k datasets respectively reported in F-PPL column. From the tables, we can note the models with lower forward perplexities (higher fluency) for the synthetic samples tend to have higher reverse perplexities. This is because the LMs are trained on synthetic sentences and they might have ungrammatical sentences, which give the higher reverse perplexities on real test data. Also, the lower forward perplexities for the Bilingual-GAN generated sentences than the real data might indicate that the generated sentences has less diversity.

Translation BLEU score is used to evaluate the ability of our GAN to generate parallel sentences. However, we need access to a reference set to measure BLEU score. We use Google Translate to translate English sentences to French and vice-versa. We used the sentences generated by our Bilingual-GAN as the candidate set and the Google translations are used as the reference set. We measure BLEU scores on 1000 sentences for each dataset and for the supervised and unsupervised models. The BLEU scores are shown in Table 4 . We perform well for the Multi30k dataset specially for the supervised scenario. Our BLEU scores are lower on the Europarl dataset. However, we get slightly higher scores for the unsupervised model compared to the supervised. If we compare our BLEU scores to conventional NMT systems, trained on these datasets, they are lower. However, generating parallel sentences by using the proposed Bilingual-GAN is a novel approach and these numbers can be a benchmark for future research. Multi30k Supervised a child in a floral pattern, mirrored necklaces, walking with trees in the background.

un enfant avec un mannequin, des lunettes de soleil, des cartons, avec des feuilles. two people are sitting on a bench with the other people.

deux personnes sont assises sur un banc et de la mer. a man is leaning on a rock wall.

un homme utilise un mur de pierre. Multi30k Unsupervised three people walking in a crowded city.

trois personnes marchant dans une rue animée.

a girl with a purple shirt and sunglasses are eating. un homme et une femme mange un plat dans un magasin local.

a woman sleeping in a chair with a graffiti lit street. une femme âgée assise dans une chaise avec une canne en nuit. The subjective judgments of the generated sentences of the models trained using the Europarl and the Multi30k datasets with maximum sentence length of size 20 and 15 is reported in Table 6 . We used 25 random generated sentences from each model and give them to a group of 4 people. We asked them to rate the sentences based on a 5-point Likert scale according to their fluency. The raters are asked to score 1 which corresponds to gibberish, 3 corresponds to understandable but ungrammatical, and 5 correspond to naturally constructed and understandable sentences (Semeniuta et al., 2018) . From Table 6 , we can note that the proposed Bilingual-GAN approach gets good rate.

The supervised approach get better rate compare to the unsupervised approach. Some examples of aligned generated sentences are describe in Table 5 . Table 6 : Human evaluation on the generated sentences by Bilingual-GAN using the Europarl and the Multi30k dataset.

5. CONCLUSION:

Our work proposed a novel method combining neural machine translation with word-based adversarial language generation to generate bilingual, aligned sentences. This work demonstrates the deep common grounds between language (text) generation and translation, which not been studied before. We also explored learning a large code space comprising of the hidden states of an RNN over the entire sequence length. The results are promising and motivate a few improvements such as improving the quality of the generated sentences and eliminating language specific performance degradation. Finally, various generation methods including reinforcement learning-based, codebased, text-based and mixed methods can be incorporated into the proposed framework to improve the performance of bilingual text generation. Since during language generation our learned code space favors English sentences over French sentences, we need to remove language specific biases or explore disentangling the code space into language specific and language agnostic subspaces.

:



Figures and Tables:

The higher BLEU scores demonstrate that the GAN can generate fluent sentences both in English

Table content: <table><tr><td/><td>Europarl</td><td/><td/></tr><tr><td/><td>English</td><td/><td>French</td></tr><tr><td colspan="4">Supervised Unsupervised Supervised Unsupervised</td></tr><tr><td>B-2 89.34</td><td>86.06</td><td>82.86</td><td>77.40</td></tr><tr><td>B-3 73.37</td><td>70.52</td><td>65.03</td><td>58.32</td></tr><tr><td>B-4 52.94</td><td>50.22</td><td>44.87</td><td>38.70</td></tr><tr><td>B-5 34.26</td><td>31.63</td><td>28.10</td><td>23.63</td></tr><tr><td/><td>Multi30k</td><td/><td/></tr><tr><td>B-2 68.41</td><td>68.36</td><td>60.23</td><td>61.94</td></tr><tr><td>B-3 47.60</td><td>47.69</td><td>41.31</td><td>41.76</td></tr><tr><td>B-4 29.89</td><td>30.38</td><td>25.24</td><td>25.60</td></tr><tr><td>B-5 17.38</td><td>18.18</td><td>14.21</td><td>14.52</td></tr></table>

Generation BLEU scores for Text Generation on Europarl and Multi30k Datasets

Table content: <table/>

Forward (F)  and Reverse (R) perplexity (PPL) results for the Europarl and Multi30k datasets using synthetic sentences of maximum length 20 and 15 respectively. F-PPL: Perplexity of a language model trained on real data and evaluated on synthetic samples. R-PPL: Perplexity of a language model trained on the synthetic samples from Bilingual-GAN and evaluated on the real test data.

Table content: <table><tr><td/><td/><td>English</td><td/><td>French</td></tr><tr><td>Dataset</td><td colspan="4">Supervised Unsupervised Supervised Unsupervised</td></tr><tr><td>Europarl</td><td>8.24</td><td>8.39</td><td>8.19</td><td>8.65</td></tr><tr><td colspan="2">Multi30k 17.65</td><td>10.08</td><td>13.86</td><td>7.13</td></tr></table>

Translation BLEU score for translation quality on Europarl and Multi30k datasets measured on 1000 generated sentences each. The reference set is approximated using Google Translate

Table content: <table><tr><td>English</td><td>French</td></tr><tr><td colspan="2">Europarl Supervised</td></tr><tr><td>the vote will take place tomorrow at 12 noon tomorrow.</td><td>le vote aura lieu demain à 12 heures.</td></tr><tr><td>mr president, i should like to thank mr unk for the</td><td>monsieur le président, je tiens à remercier tout</td></tr><tr><td>report.</td><td>particulièrement le rapporteur.</td></tr><tr><td>the debate is closed.</td><td>le débat est clos.</td></tr><tr><td colspan="2">Europarl Unsupervised</td></tr><tr><td>i have no need to know that it has been adopted in a democratic dialogue.</td><td>je n'ai pas besoin de ce qu'il a été fait en justice.</td></tr><tr><td>written statements ( amendment)</td><td>explications de vote: voir procès-verbal</td></tr><tr><td>that is what is the case of the european commission's unk.</td><td>c'est le cas qui suppose de la unk de la commission.</td></tr></table>

Examples of aligned generated sentences 4.5 HUMAN EVALUATION

Table content: <table/>

Peer review (iclr2019):
 Bilingual GANs Paper Summary: The paper proposes to learn a generative model of sentences leveraging machine translation. A first task learns a model which auto-encodes and translates sequences in a language agnostic continuous space (a first GAN objective penalizes including language information in the code). The second GAN task learns a generator that allows sampling could that cannot be distinguished with genuine sentence codes from the encoder of the first task. The first model is evaluated in a supervised and unsupervised translation setup. The second model is evaluated by generating text, measuring language model perplexity. Review: Readability: This paper could be more readable. In particular, it is not easy to understand how the two models interact: do you share the encoder and decoder parameters between the two tasks? Does the generation task fine tune these parameters? The evaluation protocol is unclear: what is the difference between supervised and half supervised MT? I do not understand what BLEU means for unconditional generation. E.g. when you generate 40k sentence sentences from the Europarl model, you compute the precision of your prediction with respect to which reference? Contribution: The contribution in terms of (un)-supervised MT is limited at best compared to (Artetxe et al 2017,2018) or (Lample et al, 2017, 2018). In particular, I do not see any reason not to use the empirical setup used in these papers and compare your results with them. The experimental setup seems weak with only short sentences < 20 words, with a small vocabulary and no attempt to deal with unknown replacements or subword units. The contribution for unconditional generation is more interesting as relying on a space defined by translation seems a good idea. But it is hard to assess. Again, the paper lacks comparison with baselines. In particular, sampling from a large language model to generate text seems an obvious baseline. It is also necessary to compare with a monolingual GAN (Zhao et al 2017), since it would quantify the benefit of the bilingual latent space. Related Work: Previous work is mischaracterized: Schwenk & Douze 2017, Lample et al 2018, Artexte et al 2017b are not relying on adversarial training. Review Summary: The paper could be more readable, the contribution is limited. The machine translation experiments should be compared with previous work on a benchmark setup. The generation experiments should consider obvious baseline such as sampling from a language model.