Multichannel Generative Language Models
Authors: Harris Chan
Jamie Kiros
William Chan
Venue:  (2020)
Keywords: text generation
generative language models
natural language processing

Abstract: A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning but through two separate channels corresponding to their languages. In this work, we present Multichannel Generative Language Models (MGLM), which models the joint distribution over multiple channels, and all its decompositions using a single neural network. MGLM can be trained by feeding it k way parallel-data, bilingual data, or monolingual data across pre-determined channels. MGLM is capable of both conditional generation and unconditional sampling. For conditional generation, the model is given a fully observed channel, and generates the k-1 channels in parallel. In the case of machine translation, this is akin to giving it one source, and the model generates k-1 targets. MGLM can also do partial conditional sampling, where the channels are seeded with prespecified words, and the model is asked to infill the rest. Finally, we can sample from MGLM unconditionally over all k channels. Our experiments on the Multi30K dataset containing English, French, Czech, and German languages suggest that the multitask training with the joint objective leads to improvements in bilingual translations. We provide a quantitative analysis of the quality-diversity trade-offs for different variants of the multichannel model for conditional generation, and a measurement of self-consistency during unconditional generation. We provide qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.

Under review as a conference paper at ICLR 2020
MULTICHANNEL GENERATIVE LANGUAGE MODELS
Anonymous authors
Paper under double-blind review
ABSTRACT
A channel corresponds to a viewpoint or transformation of underlying mean- ing. A pair of parallel sentences in English and French express the same underly- ing meaning but through two separate channels corresponding to their languages. In this work, we present Multichannel Generative Language Models (MGLM), which models the joint distribution over multiple channels, and all its decomposi- tions using a single neural network. MGLM can be trained by feeding it & way parallel-data, bilingual data, or monolingual data across pre-determined channels. MGLM is capable of both conditional generation and unconditional sampling. For conditional generation, the model is given a fully observed channel, and generates the k 1 channels in parallel. In the case of machine translation, this is akin to giving it one source, and the model generates A - targets. MGLM can also do partial conditional sampling, where the channels are seeded with prespecified words, and the model is asked to infill the rest. Finally, we can sample from MGLM unconditionally over all k channels. Our experiments on the Multi30K dataset containing English, French, Czech, and German languages suggest that the multitask training with the joint objective leads improvements in bilingual translations. We provide quantitative analysis of the quality-diversity trade-off's for different variants of the multichannel model for conditional generation, and a measurement of self-consistency during unconditional generation. We provide qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.
INTRODUCTION
natural way to consider two parallel sentences different languages that each language ex- pressing the same underlying meaning under different viewpoint. Each language can be thought of as a transformation that maps an underlying concept into a view that we collectively agree is determined as 'English' or 'French'. Similarly, an image of a cat and the word 'cat' are expressing two views of the same underlying concept. In this case, the image corresponds to a high bandwidth channel and the word 'cat" to low bandwidth channel. This way of conceptualizing parallel view- points naturally leads to the formulation of a fully generative model over each instance, where the transformation corresponds
particular generation of the underlying view. We define each of these views as a channel. As concrete example, given parallel corpus of English and French sen- tences. English and French become two channels and the corresponding generative model becomes p( (English, French). One key advantage to this formulation is that single model can be trained that can capture the full expressivity of the underlying concept, allowing us to compute conditionals and marginals along with the joint. In the case of parallel sentences, the conditionals correspond to translations from one channel to another while the marginals correspond to standard monolingual language models.
In this work, we present general framework for modeling the joint distribution p(x), x.) over channels. Our framework marginalizes over all possible factorizations of the joint distribution. Sub- sequently, this allows our framework to perform, 1) unconditional generation and 2) conditional gen- eration. We harness existing recent work insertion- methods that utilize cemi-autoregressive models that are permutation- -nntuuattiioonvarianttt to the joint factorization.
Specifically, show proof-of-concep multichannel modeling by extending KERMIT (Chan et al., 2019) to model the joint distribution over multiple sequence channels. Specifically, we train KERMIT the Multi30K (Elliott et al. 2016) machine translation task, consisting of four lan-

---

Under review as a conference paper at ICLR 2020
guages: English (EN), French (FR), Czech (CS), and German (DE). One advantage of multilingual KERMIT is during inference, we can generate translation for a single target language, or generate translations for 1 languages in parallel in logarithmic time in the token length per language. We illustrate qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.
The key contributions in this work are:
1. We present MGLM, a multichannel generative modeling framework. MGLM models the joint distribution p(x1..... ,.,,,,,, X1.) over & channels.
2. We demonstrate both conditional generation (i.e., machine translation) and unconditional sampling from MGLM.
3. In the case of conditional generation over multiple languages, we show that not only we are competitive in BLEU, but also with significant advantages in inference time and model memory savings.
4. We analyze the Quality-Diversii tradeoff from sampling MGLM and prior work.
We highlight that while focus languages specific instantiation of channel, framework can generalize to any arbitrary specification, such as other types of languages or other modalities.
2 BACKGROUND
Traditional autoregressive sequence frameworks (Sutskever et al. 2014 Cho et al. 2014) model the conditional probability p(y x) of an output sequence y conditioned on the input sequence æ with left-to-right factorization. The model decomposes pvy x)) as predicting one output token at time, conditioning on the previously generated output tokens Yez and the input sequence x:
ply x) = II plye (.yee)
(1)
Recent encoder-decoder models with attention such Transformer (Vaswani et al. 2017) have been successfully applied to various domains, including machine translation. we were to apply this left-to-right autoregressive approach towards multichannel modeling, would require to choose particular factorization order, such as p( w, x, y) = p(w)(((())))) httpp///(p(p)))) w).
Instead of assuming fixed left-to-right decomposition, recent autoregressive insertion-based con- ditional modeling frameworks (Stern et al., 2019; Welleck et al., 2019] Gu et al.) 2019) consider arbitrary factorization of the output sequence using insertion operation, which predicts both (1) content token e E from the vocabulary, and (2) location insert, relative to the current partial output y::
plc,1|x, 81) InsertionnTransfommrr// 81)
Subsequent work, KERMIT CChan et al. 2019). simplified the Insertion Transformer model by removing the encoder and only having decoder, and the trick is to concatenate the original input and output sequence as one single sequence and optimize over all possible factorizations. Consequently, KERMIT'S is able to model the joint p(x,)) y). conditionals p(x y), p(y x), as well as the marginals p(x)
Unlike with the left-to-right autoregressive approach, the exact computation of the log- log-likelihood equation 33 not possible due to the intractable marginalization over the generation order z, where S,, denotes the set of all possible permutations elements. However, we can lower bound the ligeliiel using Jensen's 's inequality:
log p(z) = loog Σ p(z)p(x
(3)
zESm
2
Σ
p(z) log p(x 12) =: L(x)
(4)
SES.
2

---

Under review as a conference paper at ICLR 2020
The loss term be simplified by changing the summation and careful decomposition of the per- mutation, leading to:
[(x) = Σ p(2) log II (( x
=ESn
Σ
Σ p(z15-1) Σ E(ss) 21:111 logp((c;;1)) x
Inference can be autoregressive via greedy decoding:
(é, 1) argmax p(c, (),,)
(5)
ett
or partially autoregressive via parallel decoding:
è argmax p(c 8c),
which is achieved by inserting at all non-finished slots. Stern et al. (2019) has shown that using a binary tree prior for p(z) led to loga iterations for token generation.
MULTICHANNEL GENERATIVE LANGUAGE MODELS
In multichannel generative language modeling, our goal is to learn generative model given
dataset consisting of a set of sequences {x(*) ,...., x (3) M from up to channels, where x (1)
(1) represents sequence of tokens from the j-th channel for the i-th example. The sulting MGLM models a joint generative distribution over multiple channels. While there are many possible implementation of Multichannel Generative Language Models, chose extended the work of Chan et al. (2019) to investigate applying the KERMIT objective on tasks with more than sequences, in order learn the joint distribution p(x1, ..... Xx) over channel sequences. For exam- ple, these channel sequences can denote different languages, such as learning pl EN, FR, CS, DE).
[x:]]] j.1......
Multichannel Model: Training
Siet Tagett
D.E
Decouinn lteration
FNI
[SFF]
BEP
Softmax
[SEP]
Sell-Atenton
Decoding Ferratio
Babed
[BEP)
SEP1
Febel
EN
[SEP)
SEP
Figure 1: (Left) An example multichannel modeling over languages (English, French, Czech), where the model predicts the missing tokens at each location across multiple channels. (Right) During inference, MGLM can generate output sequence for single target language channel (top), or for multiple language channels in parallel (bottom), conditioning on source channel sentence and partial translations of multiple language channels.
We illustrate an example data input consisting of 3 channels in Figure I (left.. We concatenate the sequences together from all channels for each example, separate by a SEP token. Even with shared vocabulary, each channel results in different token embedding, via an addition of channel-specific (learnable) embedding, or simply having a separately learned token embedding per channel. After passing through the dense self-attention layers in per Transformer architecture, the contextualized representation at each output time step predicts the possible tokens to be inserted to the left of the current input token.
At inference (generation) time, we can generate unconditionally by seeding the canvas with the [SEP] ] token and predicting the first actual token, provide much, little, partial/complete sequence in each channel. Figure [ (right) shows two possible decoding inference modes: a single target language channel (top), or multiple target language channels in parallel (bottom).
3

---

Under review as a conference paper at ICLR 2020
4 EXPERIMENTS
We experiment on multilingual dataset to demonstrate that we can learn MGLM. We perform both qualitative and quantitative experiments. We highlight the model's capabilities ranging from conditional generation (i.e., machine translation) to unconditional sampling the joint distribution over multiple languages.
We experiment on the Multi30k (Elliott et al., 2016 2017 Barrault et al. 2018). multilingual
dataset which consists of 29000 parallel training sentences in English (EN), French (FR), Czech (CS), and German (DE) sentences. We use Multi30k because multiple high quality channels (mul- tilingual translations in this case) is readily available to highlight our framework. We implement MGLM base Transformer decoder, without any causal masking, with 6 hidden layers and 1024 dimensional hidden representation. We concatenate all 4 language raw text training examples and use SentencePiece (Kudo & Richardson 2018) learn an universal subword unigram (Kudo 2018) tokenizer with a shared 32K vocabulary size. We follow a similar training set up to BERT (Devlin ct al., 2019), using Adam (Kingma & Ba 2015) optimizer with learning rate of Ic-4, warmup over the first 10% of the total training iterations varying between 10k to 50k iterations. We can train 3 different variants of MGLM by altering the sampling ratio of training data seen by the model:
1. Bilinguaa (e.g. EN + FR). We give the model fully observed source (e.g., EN), and ask the model to infill the target (e.g., FR)..
2. Multi-target (e.g., any + Rest). We give the model fully observed source (e.g., EN), and ask the model to infill the rest of the targets (e.g.. DE, FR, CS).
3. Joint. We ask the model to infill all the targets, consequently we learn a joint distribution over all the languages p(en, fr, de, cs).
4.1 TRANSLATION PERFORMANCE
The goal of MGLM is not conditional generation (i.e., machine translation), but nevertheless, we demonstrate its ability to do conditional generation in this section. We report the BLEU scores on the three test sets: test 2016 Flickr, test 2017 Flickr, test 2017 MSCOCO, for different English- + (German French, Czech} translations. We use parallel greedy decoding (Stern et al. 2019: Chan et al., 2019). i.e. inserting to all incomplete slots. Table [ the results for English to German and vice versa, respectively. Additional results for English to French, English to Czech, and German to English are shown in Appendix A.. We observe that the Multi- target models performed similar to slightly better than the bilingual models trained only on single language pair. This is particularly useful when multiple machine translation targets are desired. We now only need one MGLM model which is competitive to the bidirectional expert models. This implies we only need model for inference over multiple languages, as opposed to N models (i.c., saving substantial memory).
We also observe the full generative joint model has a BLEU gap compared to the bilingual baseline, which is consistent with the findings in Chan al. (211)). We hypothesize this is due the joint distribution being a more challenging task. We further hypothesize that in particular, during training the Joint model needs to fantasize additional details when conditioning on partial sequence in each of the channels. This results in fantasizing additional details not present in the original source sentence during translation tasks.
4.2 PARALLEL GREEDY DECODING: PARALLEL IN TARGET LANGUAGES
As alluded conceptually Figure T and in the previous section, our KERMIT-based MGLM also able to perform parallel greedy decoding that is also parallel in number of target languages. We illustrate this process in Figure 2 By starting with K initial [SEP] tokens for K target output languages, MGLM can decode K target languages that has most output tokens per language in Olog n). i.e. constant in number of target languages.
We investigate the relative speed up in generating multiple target language outputs in parallel versus generating the targets in series, in terms of wall-clock time and number of decoding iterations. In Figure 3a we plot the number of decoding iterations taken versus the total output length N for each
4

---

Under review as a conference paper at ICLR 2020
Model
Inference
Test2016
Test2017
MSCOCO
Bilingual (EN DE)
EN DE
36.14
28.32
24.15
Bilingual (EN ++ DE)
EN + DE
37.08
28.69
26.11
Multi-target (EN Rest)
EN DE
36.83
28.35
25.14
EN - FR,CS,DE
35.41
29.69
25.64
Multi-target (Any Rest)
EN DE 36.63
28.37
26.98
EN - FR,CS,DE
36.51
28.53
25.84
Joint (,(EN, CS, DE))
EN + DE EN DE
33.06
23,42
21.39
32.53
23.78
20.97
Table 1: Multi30k English + German test BLEU.
Input: A man sits bench holding his dog and looking at the water.
Parallel Decode:
FR: _Un homme _est Lassis .sur Jun _aanc. .ten ant _snn _chien ,ct regarddnt _1' cau [SEP)
CS: _Muž _eed _na lavvčce -a _drží svvé ho -psa _dívá _se _na _vodu. [SEP]
DE: _Ein _Mann _sitzt _aff _einer _Bank _und hält _seine _Hund _und _schaut _auf _das WWssse [SEP]
FR: Un homme est assis sur un banc, ten ant son chien .et regardant _ cau [SEP]
CS: Muž sedí na lavičce a drží své ho psa a dívá se _na vodu.. [SEP]
DE: _Ein _Mann sitzt auf _einer Bank und hält seine .Hund Lund sccaut Lauf _das WWasser [SEP]
FR: Un homme eest assis
banc. ten ant son chien et regardant 1'eau. [SEP]
CS: _Mu seeíí .na Lavičce -a _drží _své ho pssa a .divá _na _vodu. [SEP]
DE: _Ein _Mann sitzt Lauf _einer _Bank _und hält sseine _Hund Lund _schaut Lauf _das _Wasser.. [SEP]
FR: _Un homme _est aasss _sur _banc, ,ten ant son .chien ,ct regardant JI cau, (SEP)
CS: _Mž _edií _na Javvčce la _drží _své ho _psa aa _divá _se _na _vodu [SEP]
DE: _Fin _Mann _sitzt aaff _einer Bank _und hält seine _Hund _und _schaut _auf _das _Wasser, (SEP)
FR: _Un _homme _est Lassis
_banc _ten ant ssnn .chien regarddnt 1 ''eau. [SEP]
CS: _Mž ssedí _na lavičce -a _drží ho ppsa -a _divá _se _na _vodu. [SEP]
DE: _iin MMnn _sitzt _auf _einer _Bank _und Jält _seine _Hund _und _schaut _auf _das Wasser. [SEP]
Figure 2: Example parallel greedy decode using the Multi-target (Any - Rest) KERMIT model, starting with an English sentence. Blue underlined tokens are the inserted tokens at each iteration, and the gray tokens are the final output that have not been generated yet. Note that the three target languages are generated together in parallel.
sentence in the test 2015 Flickr test set, using the Joint KERMIT model when decoding from single source language to 3 target languages: English - (French, German, Czech). When performing serial target decoding, we only output the target conditioned on English, i.e. English > French, English- German, English Czech. We also plot several theoretical bounds: (1) upper bound (N) when decoding entirely serially, (2) lower bound 3( [logg + 2) when decoding languages serially but parallel within each language, (3) lower bound (IV
2, when
decoding the target languages in parallel and parallel within each language, and (4) [log,(N) we decode the entire output in parallel as single sequence. We observe that our model is able to meet the lower bound several times and in many cases decode below the fourth [No 2 bound, Figure compares the wall-clock speed up when decoding targets parallel vs. in series, with a linear regression line plotted. Our model achieving almost 3 times speed up in wall-clock speed. The parallel targets decoding is bottlenecked by the target language with the longest output sequence. Figure 3c compares the total output length when decoding the targets in series versus in parallel. We observe that there is a linear relationship between the output lengths using the two modes.
4.3 CONDITIONAL BILINGUAL GENERATION: QUALITY- SELLITY-DIVERSIYY TRADE-OFF
We first evaluated the models conditional generation task by sampling bilingual translations (1 source, target language) for each of the 12 language pair directions. We sample the token and location (c, plc, Ao. ÿ) from the partial canvas at each iteration, generating 100 hypothesis
5

---

Under review as a conference paper at ICLR 2020
Number of Diccoding itarstions
60
m
Liaer Bouns (1a.//31 13
Liny
50-
[i 12)
Decoding Iteratioon Speed
Upper Bouns
Output Ni (Srial
50-
panget
40-
20.
20-
10-
03
30 Output Length
10
d0 Output Length
70
10
20
50
Output Length (Paratiel Target)
(a)
(b)
(c)
Figure 3: (Left) Number of decoding iterations the output length when decoding each target language serially vs. in parallel, compared to various logarithmic bounds. We have shown that the model is able to achieve close to the theoretical lower bound (og)(NN/))] + where number of target languages k = 3. (Middle) Relative wall-clock speed up when using the parallel target languages decoding vs. serial, achieving slightly under 3 times the performance. (Right) Total output length for the 3 target languages when using serial vs parallel target language generation. While not identical, observe linear relationship between the output length using the two different modes
translations per source sentence, at softmax temperature : 0.1, 0.5, 1.0. At each temperature and model, computed the quality of the generated samples by computing the BLBU Papineni et al. (2002) score between the reference translation and the samples, and the diversity by computing the pairwise BLEU between the 100 samples per source, also known Self-BLEU [Zhu et al.) (2018). Lower Self-BLEU indicates the higher the diversity as there is less overlap between the samples.
Quality- BLEU (test 2016 flickr)
English/ / French
English / Germann
English Czech
10-2
80.
10-
50
50
Joine Bl
40-
42
40
Joint *>EEN
Muartge EN
>EN
30-
30.
30
Billngual
Bilingual RODEN
20-
20
20
10
18
10
BLEU
Figure 4: Quality- BLEU curve for several KERMIT models (bilingual, multitarget,.jing on the Multi30k text 2016 Flickr test set. Dotted diagonal line signifies BLEU equals Self- BLEU. Points indicate different temperatures, from 0.1 (low diversity, left in graph) to 1.0 (high diversity, right in graph)
Figure illustratte the Quality-Diversity trade-off for the three models for different translation pairs involving English as one of the language. The top right portion of the graph is the ideal area. We observed that the Multitarget model outperformed the Bilingual model at lower temperature (both higher quality and diversity), and at higher temperature slightly above or below in quality but still higher diversity. Note that only one single Multitarget model was used for all language pair inference time, while each bilingual model was different for each language pair curve. Therefore, single Multitarget KERMIT model could outperform specialized bilingual KERMIT models.
4.4 PARTIAL CONDITIONING MULTILINGUAL GENERATION
We demonstrate model's ability to generate infilling for partial conditioning over the multiple channels. To be explicit, we seed each channel with a few (different) words, and sample from the model. We ask the model what text completions would best fit under the model's posterior. Figure 5 highlights several examples for (English, French, German) sentence completion. We took an example from the test 2016 6 Flickr test and split into 3 chunks-beginning English,

---

Under review as a conference paper at ICLR 2020
middle in French, and ending in set of diverse, coherent examples.
sample completion. The model is able to generate
English Groundtruth: young boy, wearing chef's hat and apron, is cutting sausages kitchen.
French Groundtruth: Un jeune garçon, portant une toque et un tablier, coupe des saucisses dans une cuisine. German Groundtruth: Ein kleiner Junge mit Kochmütze und Schürzee schneidet in einer Küiche Würstchen.
English Seed: A young boy,
French Seed: portant une toque et un tablier,
German Seed: chneidet in ciner Ktcch Würstchen.
English: A young boy wearing hat and an apron grilling hotdogs in the kitchen.
French: Un jeune garçon portant toque et tablier, faisant cuire du citron des hotdogs dans cuisine.
German: Ein junger Mann trägt eine Mütze und schneidet einer Küiche Würstchen.
English: A young boy wearing hat and apron, is in kitchen. cutting with various foods it.
French: Un jeune garçon, portant une toque un tablier, dans une cuisine en projetant des poêles de nourriture.
German: Ein kleiner Junge mit Hut und Schürze schneidet in einer Küche Würstchen.
English: A young boy. wearing an orange hat apron, puts barbecue chicken kitchen.
French: Un jeune garçon, portant une toque et un tablier, coupant du poulet dans une cuisine.
German: Ein kleiner Junge in einer weißen Mütze und mit Schürze schneidet in einer Küiche Würstchen glas
English: young boy, wearing a blue and apron, is cooking meat in kitchen.
French: Un petit garçon, portant une toque et un tablier, fait la cuisine dans une cuisine. Fin kleiner Junge mit blauer Mütze und schneidet in einer Küche Würstchen.
German:
Figure 5: Partially conditional generation samples drawn from our model. The seed text is shown in gray, with several different in- in-filling samples from the model in black. The samples show reasonable consistency and diversity across samples.
4.5 UNCONDITIONAL MULTILINGUAL GENERATION
We then evaluated the models on unconditional multilingual generation task, to generate a sentence each all languages such that they correspond each other. For the Joint model, we perform types of sampling: (1) unrestricted, (2) chain, and (3) common cause. For unrestricted, we sampled one (token, location) each iteration starting from an empty canvas, allowing the model to insert token in any language, until all slots were marked completed. In the chain generation, we first restrict to generating English sentence one token at time, then sampled French, German, and Czech in order, conditioned the last sentence in the previous language. For common cause, reuse the same English and French sampled sentences, and generate the German and Czech conditioned on the English sentence (i.e. languages are all conditioned on English).
Given these sets of sentences in 4 languages, for each pair of language direction, we computed pseudo target by using separately trained (on Multi30k) vanilla Transformer (Vaswani et al. 2017) and performed beam search (size 5) to translate the chosen source language sample. Figure 6 visualizes the pseudo target BLEU score for different source-target language pairs when comparing the Joint model under different types of sampling. The shaded colour represents the difference between the current sampling scheme versus the unrestricted reference. We observe that letting the model sample in unrestricted order was better than either the chain or the common cause sampling.
5 RELATED WORK
While we have demonstrated a KERMIT implementation of a MGLM, many other variants of Trans- former models contain similar properties. Xia et al. (2019) and He et al. (0118) both consider shared encoder/decoders while KERMIT removes altogether the distinction between the encoder and de- coder. XLNet (Yang et al. 2019) also leams over all permutation of the factorization order, in addition to architectural modification for two-stream attention parameterization resolve ambigu- ity in the targets. The idea of concatenating pairs source and target sequences from different language channels have been explored by Lample & Conneau (2019). However, unlike the insertion

---

Under review as a conference paper at ICLR 2020
KERMIT jont (Utreptriceed)
KERMIT jaint (Chain)
(ERNIT joiht (Commpp Cause)
Source Language
Target Language
Target Language
Target Language
Figure 6: Unconditional multilingual generation Pseudo- Target BLEU for self-consistency when generating sentences in multiple languages. Colour shading indicates the difference compared to the Joint model (unrestricted) generation.
Model
Language
Generated Sentences
Joint
English
A young man in blue jacket walking mountain.
French
Un jeune homme en veste bleue descendant une paroi rocheuse horu.
German
Ein junger Mann in ciner blauen Jacke klettert eine Felswand hoch. Mladý muž v modré bundě stoupá po horách.
Czech
~J YYunng men in blue jackets ascend and climb mountains."
Biling.
English
Two small white dogs are holding the duck in fenced yard.
French
Deux petits chiens blancs tenant un canard dans une cour clóturée.
German
Zwei kleine weiße Hunde halten eine gelhe Ente in einem eingezäunten Hof.
Czech
Dva malí chlapei drží žlutou panou venku Alutého oploceném nádvoří.
"TTwo little boys holding a yellow gentleman outside by yellow fenced courtyard." x
Figure 7: Example unconditional text generation samples from the Joint (top) and chain of Bilingual model (bottom). Note that the Joint model generates one long sequence and we split them into the resulting four sentences in each language here, while Bilingual generate complete sentence in each language conditioned on previous sentence.
objective, their model trained through Masked Language Modeling as in BERT (Devlin et al.] 2019). and therefore was not readily able to be used for generation.
Evaluation of text generative models remain a challenge (Liu et al., 2016 Novikova et al. 2017). Quality versus diversity plots have been used to compare the trade-off at different output softmax temperatures, as such Stochastic Beam Search (Kool et al., 2019) which used simpler n- n-gram diversity instead of Self-BLEU (Zhu et al. 2018). However, are the first characterize the Q-D behaviour of insertion based models, versus cxisting left-to-right language models. Other metrics summarize the quality and diversity trade-off as single number, such as Frechet BERT Distance (Montahaei et al. 2019) inspired by the FID score (Heusel et al. 2017) used in computer vision, or take into account human evaluation (Hashimoto et 2019).
CONCLUSION
We have demonstrated that multichannel model implemented with KERMIT can learn joint dis- tribution over more than two sequences. Furthermore, our multichannel KERMIT model allows for efficient inference of multiple target languages in parallel using single model. Our work focused on specific instantiation of channels in the case of languages. However, there are no model lim- itations that inhibit further generalization to other notion of channels. In future work we aim to consider the addition of multimodal channels, such as images as well as other textual channels, such as paraphrases. premises and hypotheses, as well as questions and answers. Fully generative models still often lag behind purely discriminitive counterparts in terms of performance, but we believe it is crucial to make steps towards other model formulations that have high potential. We also intend to explore the limits on the number of channels that be considered, such building generative models over dozens or even hundereds of languages. We hope this initial line of work motivates future research on building generative models of the world.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Loic Barrault, Fethi Bougares, Lucia Specia, Chiraag Lala, Desmond Elliott, and Stella Frank. Find- ings of the third shared task on multimodal machine translation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 304-323, 2018.
William Chan, Nikita Kitaev, Kelvin Guu, Mitchell Stern, and Jakob Uszkoreit. KERMIT: Genera- tive Insertion-Baset Modeling for Sequences, 2019.
Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcchre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning Phrase Representations using RNN Encoder-Decodee for Statistical Machine Translation. In EMNLP, 2014.
Jacob Devlin, Ming-Wai Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL, 2019.
Desmond Elliott, Stella Frank, Khalil Sima'au an, and Lucia Specia. Multi30k: Multilingual english- german image descriptions. arXiv preprint arXiv: 1605.00599 2016.
Desmond Elliott, Stella Frank, Loic Barrault, Fethi Bougares, and Lucia Specia. Findings of the second shared task on multimodal machine translation and multilingual image description. In Proceedings of the Second Conference on Machine Translation, Volume 2: Shared Task Papers, pp. 215-233, Copenhagen, Denmark, September 2017. Association for Computational Linguis- tics. URL /tRt:ttp::/// wwwp:://wwwwwwwww..wrooo...... wwtp://eew..g//a.oooooo///////// 47788
Jiatao Gu, Qi Liu, and Kyunghyun Cho. Insertion-based Decoding with Automatically Inferred Generation Order. arXiv, 2019.
Tatsunori Hashimoto, Hugh Zhang, and Percy Liang. Unifying human and statistical evaluation for natural language generation. arXiv preprint arXiv:1904.02792. 2019.
Tianyu He, Xu Tan, Yingee Xia, Di He, Tao Qin, Zhibo Chen, and Tie-Yan Liu. Layer-wise co- ordination between encoder and decoder for neural machine translation. In Advances in Neural Information Processing Systems, pp. 7944-7954, 2018.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. In Advances in Neural Information Processing Systems, pp. 6626-6637, 2017.
Diederik Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR. 2015.
Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The Gumbel-top-k trick for sampling sequences without replacement. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3499-3508, Long Beach, California, USA, 09-15 Jun 2019. PMLR. URL http::/pproceeaangs..o preess press/
v97 1997/ooll9a... html
Taku Kudo. Subword regularization: Improving neural network translation models with multiple subword candidates. arXiv preprint 109559, 2018.
Taku Kudo and John Richardson. Sentencepicce: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226. 2018.
Guillaume Lample and Alexis Conneau. Cross-lingual language model pretraining. arXiv preprint arXiv:1901.07291, 2019.
Chia-Wei Liu, Ryan Lowe, Tulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. How not to evaluate your dialogue system: An empirical study of unsupervised eval- uation metrics for dialogue response generation. arXiv preprint arXiv.1603.08023 2016.
Ehsan Montahaei, Danial Alihosseini, and Mahdieh Soleymani Baghshah. Jointly measuring diver- sity and quality in text generation models. arXiv preprint arXiv: /000:///040097770 2019.

---

Under review as a conference paper at ICLR 2020
Jekaterina Novikova, Ondřej Dušek, Amanda Cereas Curry, and Verena Rieser. Why need new evaluation metrics for nlg. arXiv preprint arXiv: I 170.006875 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting association for computational linguistics, pp. 311-318. Association for Computational Linguistics, 2002.
Mitchell Stern, William Chan, Jamie Kiros, and Jakob Uszkoreit. Insertion Transformer: Flexible Sequence Generation via Insertion Operations. In ICML, 2019.
Ilya Sutskever, Oriol Vinyals, and Quoc Lc. Sequence to Sequence Learning with Neural Networks. In NIPS, 2014.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In NIPS, 2017.
Sean Welleck, Kiante Brantley, Hal Daume, and Kyunghyun Cho. Non-Monotonic Sequential Text Generation. In ICML,, 2019.
Yingce Xia, Tianyu He, Xu Tan, Fei Tian, Di He, and Tao Qin. Tied transformers: Neural machine translation with shared encoder and decoder. In Proceedings of the AAAI Conference Artificial Intelligence, volume 33, pp. 5466-5473 2019.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint 2019.
Yaoming Zhu, Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang, Jun Wang, and Yong Yu. Texygen: A benchmarking platform for text generation models. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1097-1100. ACM, 2018.
10

---

Under review as a conference paper at ICLR 2020
A
APPENDICES
A.1
ADDITIONAL QUALITY-DIVERSITY CURVES FOR CONDITIONAL GENERATION
Quality-Diversity BLEU (test 2017 __fcklikcrr
English| French
English Gerrann,
50 J
45
Joint EN->*
35
35
Joint *->EN
Multitarget EN->*
30
Multitarget +>EN
Bilingual EN->*
Bilingual *- EN
20
20
90
80
70
60
50
40
30
20
10
90
80
70
60
50
40
30
20
10
Self-BLEU
Self-BLEU
Figure& 8: on the Multi30k text 2017 Flickr test set. Dotted diagonal line signifies BLEU equals Self- BLEU. Points indicate different temperatures, from 0.1 (low diversity, left in graph) to 1.0 (high diversity, right in graph)
BLEU curve for several KERMIT models (bilingual, multitarget, joint)
Quality-Diversity BLEU (test 2017
2010_0000o000 mscoco)
English
French
English / German,
45
40
Joint EN->*
Joint +.>EN
Multitarget EN->*
30
30
Multitarget +.>EN
Bilingual EN->*
Bilingual *->EN
15
15
10
50
30
10
Self-BLEU
Self-BLEU
Figure 9: Diversty BLEU curve for several KERMIT models (bilingual, multitarget, joint) on the Multi30k text 2017 MSCOCO test set. Dotted diagonal line signifies BLEU equals Self- BLBU, Points indicate different temperatures, from 0.1 (low diversity, left in graph) to 1.0 (high diversity, right in graph)
11

---

Under review as a conference paper at ICLR 2020
A.2 ADDITIONAL MULTI30K TRANSLATION RESULTS
Model
Inference
Test2016
Test2017
MSCOCO
Bilingual (EN -> FR)
EN ->
58.80
50.35
42.82
Bilingual (EN <> FR)
EN -> FR
59.29
52.13
42.17
Multi-target (EN - Rest)
EN ->
58.08
50.39
42.19
EN + FR,CS,DE
58.52
50.49
41.53
Multi-target (Any - Rest)
EN -> EN + FR,CS,DE
57.64
50.01
40.18
57.35
48.13
39.98
Joint (p(EN, FR, CS, DE))
EN + FR 50.87 48.85
40.69
33.93
EN + FR,CS,EE
39.92
33.45
Table 2: Multi30k English French test BLEU.
Model
Inference
Test2016
Bilingual (EN + CS)
EN CS
28.58
Bilingual (EN + CS)
EN + CS
29.03
Multi-target (EN > Rest)
- 30.48
EN - FR.CS,EE 30.15
Multi-target (Any Rest)
30.11
EN + FR,CS,DE 30.11
Joint (p F.N, FR, CS, DF))
CS
26.45
EN - FR,CS,DE
26.35
Table 3: Multi30k English Czech test BLEU.
Model
Inference
Test2016
Test2017
MSCOCO
Bilingual (DE EN)
DE EN
39.40
34.90
27.75
Bilingual (EN + DE)
DE + EN
40.52
35.66
28.61
Multi-targer (DE Rest)
DE + EN
40.75
36.38
28.91
DE + EN, FR,CS
39.72
35.95
28.20
Multi-target (Any Rest)
DE + EN
40.69
36.02
28.89
DE + EN, FR,CS
39.97
37.07
28.62
Joint (p( (EN, FR, CS, DE))
DE DE + EN, FR.CS
38.44 36.30
30.82
25.46
29.68
24.87
Table 4: Multi30k German - English test BLEU.
12

---

Under review as a conference paper at ICLR 2020
A.3
UNCONDITIONAL SAMPLING GENERATION
Figure T0
illustrates the serial sampling (one token at a time) from the joint model, every 20
timesteps
Iterations
Language
Generated Sentence from Joint Model
English
French
Czech
Mladý
German
20
English
French
descendant
Czech
Mladý muž modré bundë stoupá po
German
Mann klettert.
40
English
blue jacket walking up mountain.
French
veste descendant paroi rocheuse en Mladý muž modré bundě stoupá po horách. Mann klettert.
Czech
German
60
English
man blue jacket walking up a mountain. veste bleue descendant une paroi rocheuse en horu. Mladý muž modré bundë stoupá po horách.
French
Czech
German
Mann einer blauen klettert eine hach.
80
English
A young man in blue jacket walking up mountain.
French
bleue descendant une paroi rocheuse horu.
Czech
Mladý muž modré hundë stoupá po horách.
German
Ein junger Mann in einer blauen Jacke klettert eine Felswand hoch.
English
young man in blue jacket walking up a mountain.
French
Un jeune homme en veste blcue descendant une paroi rocheuse en horu. Mladý muž modré bundē stoupá horách. Ein junger Mann in einer blauen Jacke klettert eine Felswand hoch.
Czech
German
Example serial sampling unconditional text generation from the joint p(E,, FR, CIS, DF) model, over 96 insertion time steps. Note that the model generates long sequence and we split them into the resulting four sentences in each language here.
Figure 10:
13


Reviews:
[{"review_id": "SJlT6ASJcS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission belongs to the area of multi-view modelling. In particular, the submission describes construction of multi-view language models that (i) can generate text simultaneously in multiple languages, (ii) can generate text in one or more languages conditioned on text from another language. This submission extends previously proposed KERMIT from two views to more than two views. I believe this paper could be of interest to multi-view modelling/learning community. \n\nThough the original KERMIT approach is very interesting and you application of it to more than two views is also interesting I find the presentation to be poor. In particular I find section 2 to be hard if not impossible to understand without referring to the original paper where the story, equations, nomenclature are much more clearly explained. Even though your extension from two views to multiple is simple I find reliance on a diagram to be a mistake as I find your description not to be very clear. Given that there are no equations to support the reader and that the original equations are not adequate I find it hard to understand Sections 2 and 3. The key experimental result in Table 1 is only briefly commented on despite featuring multiple models with different strength and weaknesses, multiple types of inference. If space is of concern I would suggest removing Figure 2 (or changing input from non-English to English and removing or removing another qualitative table).\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "S1gYvmxpFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a multichannel generative language model (MGLM), which models the joint distribution p(channel_1, ..., channel_k) over k channels. MGLM can be used for both conditional generation (e.g., machine translation) and unconditional sampling. In the experiments, MGLM uses the Multi30k dataset where multiple high quality channels are available, in the form of multilingual translations.\n\nI feel that this paper is not ready for publication at ICLR due to the following major issues:\n\n* Missing important related work: This paper seems unaware of an important related work \"Multi-Task Learning for Multiple Language Translation\" by Dong et al, ACL 2015. In fact, Dong et al. investigated the problem of learning a machine translation model that can simultaneously translate sentences from one source language to multiple target languages. Although machine translation is just an example of MGLM, Dong et al. is highly relevant to the conditional generation with MGLM, needless to say that they share the same multi-language translation problem domain. Thus, this paper will be much stronger if comparison with important baseline methods is provided.\n\n* Limited novelty: This paper extends Chan et al.'s KERMIT by applying its objective on tasks with more than 2 sequences, in order to learn the joint distribution p(channel_1, ..., channel_k) over k channel sequences. Most of the math in this paper can be found in the original Chan et al.'s paper. The extension to the multichannel case is incremental as it is hard to justify the challenge of such extensions.\n\nBesides, as minor suggestions, it would help readers if more illustrations of Figure 1 (especially the inference part) can be provided.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "H1g0adr3Kr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Paper summary]\nThis work is an extension of KERMIT (Chan et al., 2019) to multiple languages and the proposed model is called “multichannel generative language models”. KERMIT is an extension of “Insertion Transformer” (Stern et. al, 2019), a non-autoregressive model that can jointly determine which word and which place the translated words should be inserted. KERMIT shares the encoder and decoder of insertion Transformer, and the source sentence and target sentence are concatenated to train a generative model (also, various loss functions are included). In this work, parallel sentences from more than two languages are concatenated together and fed into KERMIT. Each language is associated with a language embedding. This work demonstrates that a joint distribution p(x1, . . . , xk) over k channels/languages can be properly modeled through a single model. The authors carry out experiments on multi30k dataset.\n\n[Pros] Some discoveries of this work are interesting, including: (1) It is possible to use a single model to translate a sentence into different languages in a non-autoregressive way. (2) The unconditional multilingual generation in Section 4.5 is interesting, especially, the generation order is determined by the model rather than left-to-right.\n\n[Questions]\n1.\tThe authors work on multi30k dataset, which is not a typical dataset for machine translation. \n(A)\tThe dataset and the corresponding information is at https://github.com/multi30k/dataset. The number of words in a sentence is smaller than 15, which is too short for a machine translation. Also, the pattern of sentences is relatively simple.\n(B)\tFor real world application, I am not sure whether it is possible to collect a large amount of k-parallel data where $k>2$. Therefore, the application scenario is limited. What if we have a large amount of bilingual data instead of k-parallel data? How should we leverage the large amount of monolingual data?\n2.\tFor novelty, this is an extension of KERMIT to a multilingual version, which limits the novelty of this wok.\n3.\tThe best results on En->De in Table 1 are inconsistent. On tst16, bilingual en<->de is the best; on tst17, en<->{rest} is the best; on mscoco, any<->rest is the best. In Table 2, seems using bilingual data only is the best choice. This makes me confuse about how to use your proposed method. However, \n", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a multi-view generative model which is applied to multilingual text generation. Although all reviewers find the overall approach is important and some results are interesting, the main concern is about the novelty. At the technical level, the proposed method is the extension of the original two-view KERMIT to multiviews, which I have to say incremental. At a higher level, multi-lingual language generation itself is not a very novel idea, and the contribution of the proposed method should be better positioned comparing to related studies. (for example, Dong et al, ACL 2015 as suggested by R#3). Also, some reviewers pointed out the problems in presentation and unconvincing experimental setup. I support the reviewers’ opinions and would like to recommend rejection this time.
I recommend authors to take in the reviewers’ comments and polish the work for the next chance. 
Decision: Reject