Title: MULTICHANNEL GENERATIVE LANGUAGE MODELS

Abstract: A channel corresponds to a viewpoint or transformation of an underlying meaning. A pair of parallel sentences in English and French express the same underlying meaning but through two separate channels corresponding to their languages. In this work, we present Multichannel Generative Language Models (MGLM), which models the joint distribution over multiple channels, and all its decompositions using a single neural network. MGLM can be trained by feeding it k way parallel-data, bilingual data, or monolingual data across pre-determined channels. MGLM is capable of both conditional generation and unconditional sampling. For conditional generation, the model is given a fully observed channel, and generates the k -1 channels in parallel. In the case of machine translation, this is akin to giving it one source, and the model generates k -1 targets. MGLM can also do partial conditional sampling, where the channels are seeded with prespecified words, and the model is asked to infill the rest. Finally, we can sample from MGLM unconditionally over all k channels. Our experiments on the Multi30K dataset containing English, French, Czech, and German languages suggest that the multitask training with the joint objective leads to improvements in bilingual translations. We provide a quantitative analysis of the quality-diversity trade-offs for different variants of the multichannel model for conditional generation, and a measurement of self-consistency during unconditional generation. We provide qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.

1. INTRODUCTION:

A natural way to consider two parallel sentences in different languages is that each language is expressing the same underlying meaning under a different viewpoint. Each language can be thought of as a transformation that maps an underlying concept into a view that we collectively agree is determined as 'English' or 'French'. Similarly, an image of a cat and the word 'cat' are expressing two views of the same underlying concept. In this case, the image corresponds to a high bandwidth channel and the word 'cat' to a low bandwidth channel. This way of conceptualizing parallel viewpoints naturally leads to the formulation of a fully generative model over each instance, where the transformation corresponds to a particular generation of the underlying view. We define each of these views as a channel. As a concrete example, given a parallel corpus of English and French sentences, English and French become two channels and the corresponding generative model becomes p(English, French). One key advantage to this formulation is that single model can be trained that can capture the full expressivity of the underlying concept, allowing us to compute conditionals and marginals along with the joint. In the case of parallel sentences, the conditionals correspond to translations from one channel to another while the marginals correspond to standard monolingual language models.

In this work, we present a general framework for modeling the joint distribution p(x 1 , ..., x k ) over k channels. Our framework marginalizes over all possible factorizations of the joint distribution. Subsequently, this allows our framework to perform, 1) unconditional generation and 2) conditional generation. We harness existing recent work on insertion-based methods that utilize semi-autoregressive models that are permutation-invariant to the joint factorization. Specifically, we show a proof-of-concept multichannel modeling by extending KERMIT (Chan et al., 2019) to model the joint distribution over multiple sequence channels. Specifically, we train KERMIT on the Multi30K (Elliott et al., 2016) machine translation task, consisting of four lan-guages: English (EN), French (FR), Czech (CS), and German (DE). One advantage of multilingual KERMIT is during inference, we can generate translation for a single target language, or generate translations for k -1 languages in parallel in logarithmic time in the token length per language. We illustrate qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages.

The key contributions in this work are:

1. We present MGLM, a multichannel generative modeling framework. MGLM models the joint distribution p(x 1 , . . . , x k ) over k channels.

2. We demonstrate both conditional generation (i.e., machine translation) and unconditional sampling from MGLM.

3. In the case of conditional generation over multiple languages, we show that not only we are competitive in BLEU, but also with significant advantages in inference time and model memory savings.

4. We analyze the Quality-Diversity tradeoff from sampling MGLM and prior work.

We highlight that while we focus on languages as a specific instantiation of a channel, our framework can generalize to any arbitrary specification, such as other types of languages or other modalities.

2. BACKGROUND:

Traditional autoregressive sequence frameworks (Sutskever et al., 2014; Cho et al., 2014) model the conditional probability p(y | x) of an output sequence y conditioned on the input sequence x with a left-to-right factorization. The model decomposes p(y | x) as predicting one output token at time, conditioning on the previously generated output tokens y <t and the input sequence x:

EQUATION

Recent encoder-decoder models with attention such as Transformer (Vaswani et al., 2017) have been successfully applied to various domains, including machine translation. If we were to apply this left-to-right autoregressive approach towards multichannel modeling, we would require to choose a particular factorization order, such as p(w, x, y) = p(w)p(x|w)p(y|x, w).

Instead of assuming a fixed left-to-right decomposition, recent autoregressive insertion-based conditional modeling frameworks (Stern et al., 2019; Welleck et al., 2019; Gu et al., 2019) consider arbitrary factorization of the output sequence by using insertion operation, which predicts both (1) content token c ∈ C from the vocabulary, and (2) location l insert, relative to the current partial output ŷt :

EQUATION

Subsequent work, KERMIT (Chan et al., 2019) , simplified the Insertion Transformer model by removing the encoder and only having a decoder, and the trick is to concatenate the original input and output sequence as one single sequence and optimize over all possible factorizations. Consequently, KERMIT is able to model the joint p(x, y), conditionals p(x | y), p(y | x), as well as the marginals p(x), p(y).

Unlike with the left-to-right autoregressive approach, the exact computation of the log-likelihood equation 3 is not possible due to the intractable marginalization over the generation order z, where S n denotes the set of all possible permutations on n elements. However, we can lower bound the log-likelihood using Jensen's inequality:

EQUATION

EQUATION

The loss term can be simplified by changing the summation and careful decomposition of the permutation, leading to:

L(x) = z∈Sn p(z) log n i=1 p((c z i , l z i ) | x z,i-1 1:i-1 ) = n i=1 z1:i-1 p(z 1:i-1 ) zi p(z i | z 1:i-1 ) log p((c z i , l z i ) | x z,i-1 1:i-1 )

Inference can be autoregressive via greedy decoding:

EQUATION

or partially autoregressive via parallel decoding:

EQUATION

which is achieved by inserting at all non-finished slots. Stern et al. (2019) has shown that using a binary tree prior for p(z) led to ≈ log 2 n iterations for n token generation.

3. MULTICHANNEL GENERATIVE LANGUAGE MODELS:

In multichannel generative language modeling, our goal is to learn a generative model given a dataset consisting of a set of sequences {x

EQUATION

k } M i=1 from up to k channels, where x

(i) k = [x (i) j,1 , . . . , x (i) j,n

] represents a sequence of tokens from the j-th channel for the i-th example. The resulting MGLM models a joint generative distribution over multiple channels. While there are many possible implementation of Multichannel Generative Language Models, we chose to extended the work of Chan et al. (2019) to investigate applying the KERMIT objective on tasks with more than 2 sequences, in order to learn the joint distribution p(x 1 , . . . , x k ) over k channel sequences. For example, these channel sequences can denote different languages, such as learning p(EN, F R, CS, DE).

Data for Model:

Target Source

Bilingual (Uni-direction) Joint [SEP] [SEP] EN FR CS [SEP] DE [SEP] [SEP] [SEP] EN FR CS [SEP] DE [SEP]

Multi-Target (Any to Rest)

[SEP] [SEP] EN FR CS [SEP] DE [SEP] [SEP] [SEP] EN FR CS [SEP] DE [SEP] [SEP] [SEP] EN FR CS [SEP] DE [SEP] Legend [SEP] [SEP] EN FR FR [SEP] [SEP] EN [SEP] [SEP] [SEP] EN DE [SEP] [SEP] [SEP] EN [SEP] [SEP] EN [SEP] [SEP] EN DE [SEP] [SEP] EN [SEP]

Training Sets Inference:

Decoding Iteration 

[SEP] [SEP] A [SEP] C F F H I W Y Z Softmax B D,E G X

Self-Attention:

+ + + + + + + [SEP] EN [SEP] [SEP] [SEP] EN [SEP] Decoding Iteration FR [SEP] [SEP] EN [SEP] [SEP] [SEP] EN CS [SEP] [SEP] [SEP] EN [SEP] [SEP] EN [SEP] [SEP] EN CS [SEP] [SEP] EN [SEP]

Inference:

Decoding Iteration

[SEP] EN [SEP] [SEP] [SEP] EN [SEP]

Decoding Iteration During inference, MGLM can generate output sequence for a single target language channel (top), or for multiple language channels in parallel (bottom), conditioning on source channel sentence and partial translations of multiple language channels.

Multichannel:

We illustrate an example data input consisting of 3 channels in Figure 1 (left). We concatenate the sequences together from all channels for each example, separate by a SEP token. Even with shared vocabulary, each channel results in a different token embedding, via an addition of a channel-specific (learnable) embedding, or simply having a separately learned token embedding per channel. After passing through the dense self-attention layers as in per Transformer architecture, the contextualized representation at each output time step predicts the possible tokens to be inserted to the left of the current input token.

At inference (generation) time, we can generate unconditionally by seeding the canvas with the [SEP] token and predicting the first actual token, or provide as much, or as little, partial/complete sequence in each channel. Figure 1 (right) shows two possible decoding inference modes: a single target language channel (top), or multiple target language channels in parallel (bottom).

4. EXPERIMENTS:

We experiment on a multilingual dataset to demonstrate that we can learn MGLM. We perform both qualitative and quantitative experiments. We highlight the model's capabilities ranging from conditional generation (i.e., machine translation) to unconditional sampling the joint distribution over multiple languages.

We experiment on the Multi30k (Elliott et al., 2016; 2017; Barrault et al., 2018) , a multilingual dataset which consists of 29000 parallel training sentences in English (EN), French (FR), Czech (CS), and German (DE) sentences. We use Multi30k because multiple high quality channels (multilingual translations in this case) is readily available to highlight our framework. We implement MGLM as a base Transformer decoder, without any causal masking, with 6 hidden layers and 1024 dimensional hidden representation. We concatenate all 4 language raw text training examples and use SentencePiece (Kudo & Richardson, 2018) to learn an universal subword unigram (Kudo, 2018) tokenizer with a shared 32K vocabulary size. We follow a similar training set up to BERT (Devlin et al., 2019) , using Adam (Kingma & Ba, 2015) optimizer with learning rate of 1e-4, warmup over the first 10% of the total training iterations varying between 10k to 50k iterations. We can train 3 different variants of MGLM by altering the sampling ratio of training data seen by the model:

1. Bilingual (e.g., EN → FR). We give the model a fully observed source (e.g., EN ), and ask the model to infill the target (e.g., F R).

2. Multi-target (e.g., any 1 → Rest). We give the model a fully observed source (e.g., EN ), and ask the model to infill the rest of the targets (e.g., DE, F R, CS).

3.:

Joint. We ask the model to infill all the targets, consequently we learn a joint distribution over all the languages p(en, fr, de, cs).

4.1. TRANSLATION PERFORMANCE:

The goal of MGLM is not conditional generation (i.e., machine translation), but nevertheless, we demonstrate its ability to do conditional generation in this section. We report the BLEU scores on the three test sets: test 2016 Flickr, test 2017 Flickr, test 2017 MSCOCO, for different English → {German, French, Czech} translations. We use parallel greedy decoding (Stern et al., 2019; Chan et al., 2019) , i.e. inserting to all incomplete slots. Table 1 summarizes the results for English to German and vice versa, respectively. Additional results for English to French, English to Czech, and German to English are shown in Appendix A.2. We observe that the Multitarget models performed similar to slightly better than the bilingual models trained only on a single language pair. This is particularly useful when multiple machine translation targets are desired. We now only need one MGLM model which is competitive to the bidirectional expert models. This implies we only need 1 model for inference over multiple languages, as opposed to N models (i.e., saving substantial memory).

We also observe the full generative joint model has a BLEU gap compared to the bilingual baseline, which is consistent with the findings in Chan et al. (2019) . We hypothesize this is due to the joint distribution being a more challenging task. We further hypothesize that in particular, during training the Joint model needs to fantasize additional details when conditioning on partial sequence in each of the channels. This results in fantasizing additional details not present in the original source sentence during translation tasks.

4.2. PARALLEL GREEDY DECODING: PARALLEL IN TARGET LANGUAGES:

As alluded conceptually in Figure 1 and in the previous section, our KERMIT-based MGLM is also able to perform parallel greedy decoding that is also parallel in number of target languages. We illustrate this process in Figure 2 . By starting with K initial [SEP] tokens for K target output languages, MGLM can decode K target languages that has at most n output tokens per language in O(log n), i.e. constant in number of target languages.

We investigate the relative speed up in generating multiple target language outputs in parallel versus generating the targets in series, in terms of wall-clock time and number of decoding iterations. In Figure 3a , we plot the number of decoding iterations taken versus the total output length N for each Figure 2 : Example parallel greedy decode using the Multi-target (Any → Rest) KERMIT model, starting with an English sentence. Blue underlined tokens are the inserted tokens at each iteration, and the gray tokens are the final output that have not been generated yet. Note that the three target languages are generated together in parallel.

sentence in the test 2016 Flickr test set, using the Joint KERMIT model when decoding from a single source language to 3 target languages: English → {French, German, Czech}. When performing serial target decoding, we only output the target conditioned on English, i.e. English → French, English → German, English → Czech. We also plot several theoretical bounds: (1) upper bound (N ) when decoding entirely serially, (2) lower bound 3( log 2 (N/3) + 2) when decoding 3 languages serially but parallel within each language, (3) lower bound log 2 (N/3) + 2, when decoding the 3 target languages in parallel and parallel within each language, and (4) log 2 (N ) +2, if we decode the entire output in parallel as a single sequence. We observe that our model is able to meet the lower bound several times and in many cases decode below the fourth log 2 (N ) + 2 bound. Figure 3b compares the wall-clock speed up when decoding targets in parallel vs. in series, with a linear regression line plotted. Our model achieving almost 3 times speed up in wall-clock speed. The parallel targets decoding is bottlenecked by the target language with the longest output sequence. Figure 3c compares the total output length when decoding the targets in series versus in parallel. We observe that there is a linear relationship between the output lengths using the two modes.

4.3. CONDITIONAL BILINGUAL GENERATION: QUALITY-DIVERSITY TRADE-OFF:

We first evaluated the models on conditional generation task by sampling bilingual translations (1 source, 1 target language) for each of the 12 language pair directions. We sample the token and location (c, l) ∼ p(c, l|x, ŷ) from the partial canvas at each iteration, generating 100 hypothesis the output length when decoding each target language serially vs. in parallel, compared to various logarithmic bounds. We have shown that the model is able to achieve close to the theoretical lower bound log 2 (N/k) +2 where number of target languages k = 3. (Middle) Relative wall-clock speed up when using the parallel target languages decoding vs. serial, achieving slightly under 3 times the performance. (Right) Total output length for the 3 target languages when using serial vs parallel target language generation. While not identical, we observe a linear relationship between the output length using the two different modes translations per source sentence, at softmax temperature τ = 0.1, 0.5, 1.0. At each temperature and model, we computed the quality of the generated samples by computing the BLEU Papineni et al. (2002) score between the reference translation and the samples, and the diversity by computing the pairwise BLEU between the 100 samples per source, also known as Self-BLEU Zhu et al. (2018) .

Lower Self-BLEU indicates the higher the diversity as there is less overlap between the samples. Figure 4 illustrates the Quality-Diversity trade-off for the three models for different translation pairs involving English as one of the language. The top right portion of the graph is the ideal area. We observed that the Multitarget model outperformed the Bilingual model at lower temperature (both higher quality and diversity), and at higher temperature slightly above or below in quality but still higher diversity. Note that only one single Multitarget model was used for all language pair at inference time, while each bilingual model was different for each language pair curve. Therefore, a single Multitarget KERMIT model could outperform specialized bilingual KERMIT models.

4.4. PARTIAL CONDITIONING MULTILINGUAL GENERATION:

We demonstrate our model's ability to generate infilling for partial conditioning over the multiple channels. To be explicit, we seed each channel with a few (different) words, and sample from the model. We ask the model what text completions would best fit under the model's posterior. Figure 5 highlights several examples for (English, French, German) sentence completion. We took an example from the test 2016 Flickr test set and split it into 3 chunks-beginning in English, 

4.5. UNCONDITIONAL MULTILINGUAL GENERATION:

We then evaluated the models on unconditional multilingual generation task, to generate a sentence each in all 4 languages such that they correspond to each other. For the Joint model, we perform 3 types of sampling: (1) unrestricted, (2) chain, and (3) common cause. For unrestricted, we sampled one (token, location) at each iteration starting from an empty canvas, allowing the model to insert a token in any language, until all slots were marked as completed. In the chain generation, we first restrict to generating English sentence one token at a time, then sampled French, German, and Czech in order, conditioned on the last sentence in the previous language. For common cause, we reuse the same English and French sampled sentences, and generate the German and Czech conditioned on the English sentence (i.e. 3 languages are all conditioned on English).

Given these sets of sentences in 4 languages, for each pair of language direction, we computed a pseudo target by using a separately trained (on Multi30k) vanilla Transformer (Vaswani et al., 2017) and performed beam search (size 5) to translate the chosen source language sample. Figure 6 visualizes the pseudo target BLEU score for different source-target language pairs when comparing the Joint model under different types of sampling. The shaded colour represents the difference between the current sampling scheme versus the unrestricted reference. We observe that letting the model sample in unrestricted order was better than either the chain or the common cause sampling.

5. RELATED WORK:

While we have demonstrated a KERMIT implementation of a MGLM, many other variants of Transformer models contain similar properties. Xia et al. (2019) and He et al. (2018) both consider shared encoder/decoders while KERMIT removes altogether the distinction between the encoder and decoder. XLNet (Yang et al., 2019) also learns over all permutation of the factorization order, in addition to architectural modification for two-stream attention parameterization to resolve ambiguity in the targets. The idea of concatenating pairs of source and target sequences from different language channels have been explored by Lample & Conneau (2019) . However, unlike the insertion model (bottom) . Note that the Joint model generates one long sequence and we split them into the resulting four sentences in each language here, while Bilingual generate a complete sentence in each language conditioned on previous sentence.

objective, their model is trained through Masked Language Modeling as in BERT (Devlin et al., 2019) , and therefore was not readily able to be used for generation.

Evaluation of text generative models remain a challenge (Liu et al., 2016; Novikova et al., 2017) . Quality versus diversity plots have been used to compare the trade-off at different output softmax temperatures, as such in Stochastic Beam Search (Kool et al., 2019) which used a simpler n-gram diversity instead of Self-BLEU (Zhu et al., 2018) . However, we are the first to characterize the Q-D behaviour of insertion based models, versus existing left-to-right language models. Other metrics summarize the quality and diversity trade-off as a single number, such as Frechet BERT Distance (Montahaei et al., 2019) inspired by the FID score (Heusel et al., 2017) used in computer vision, or take into account human evaluation (Hashimoto et al., 2019) .

6. CONCLUSION:

We have demonstrated that a multichannel model implemented with KERMIT can learn a joint distribution over more than two sequences. Furthermore, our multichannel KERMIT model allows for efficient inference of multiple target languages in parallel using a single model. Our work focused on a specific instantiation of channels in the case of languages. However, there are no model limitations that inhibit further generalization to other notion of channels. In future work we aim to consider the addition of multimodal channels, such as images as well as other textual channels, such as paraphrases, premises and hypotheses, as well as questions and answers. Fully generative models still often lag behind purely discriminitive counterparts in terms of performance, but we believe it is crucial to make steps towards other model formulations that have high potential. We also intend to explore the limits on the number of channels that can be considered, such as building generative models over dozens or even hundereds of languages. We hope this initial line of work motivates future research on building generative models of the world. 

:



Figures and Tables:

Figure1: (Left) An example multichannel modeling over 3 languages(English, French, Czech), where the model predicts the missing tokens at each location across multiple channels. (Right) During inference, MGLM can generate output sequence for a single target language channel (top), or for multiple language channels in parallel (bottom), conditioning on source channel sentence and partial translations of multiple language channels.

Figure3: (Left) Number of decoding iterations vs. the output length when decoding each target language serially vs. in parallel, compared to various logarithmic bounds. We have shown that the model is able to achieve close to the theoretical lower bound log 2 (N/k) +2 where number of target languages k = 3. (Middle) Relative wall-clock speed up when using the parallel target languages decoding vs. serial, achieving slightly under 3 times the performance. (Right) Total output length for the 3 target languages when using serial vs parallel target language generation. While not identical, we observe a linear relationship between the output length using the two different modes

Figure 4: Quality-Diversity BLEU curve for several KERMIT models (bilingual, multitarget, joint) on the Multi30k text 2016 Flickr test set. Dotted diagonal line signifies BLEU equals Self-BLEU. Points indicate different temperatures, from 0.1 (low diversity, left in graph) to 1.0 (high diversity, right in graph)

Figure 5: Partially conditional generation samples drawn from our model. The seed text is shown in gray, with several different in-filling samples from the model in black. The samples show reasonable consistency and diversity across samples.

Figure 6: Unconditional multilingual generation Pseudo-Target BLEU for self-consistency when generating sentences in multiple languages. Colour shading indicates the difference compared to the Joint model (unrestricted) generation.

Figure 7: Example unconditional text generation samples from the Joint (top) and chain of Bilingualmodel (bottom). Note that the Joint model generates one long sequence and we split them into the resulting four sentences in each language here, while Bilingual generate a complete sentence in each language conditioned on previous sentence.

Figure 8: Quality-Diversity BLEU curve for several KERMIT models (bilingual, multitarget, joint) on the Multi30k text 2017 Flickr test set. Dotted diagonal line signifies BLEU equals Self-BLEU. Points indicate different temperatures, from 0.1 (low diversity, left in graph) to 1.0 (high diversity, right in graph)

Figure 9: Quality-Diversity BLEU curve for several KERMIT models (bilingual, multitarget, joint) on the Multi30k text 2017 MSCOCO test set. Dotted diagonal line signifies BLEU equals Self-BLEU. Points indicate different temperatures, from 0.1 (low diversity, left in graph) to 1.0 (high diversity, right in graph)

Multi30k English → German test BLEU. : A man sits on a bench holding his dog and looking at the water. Parallel Decode: FR: Un homme est assis sur un banc , ten ant son chien et regardant l ' eau . [SEP] CS: Muž sedí na lavičce a drží své ho psa a dívá se na vodu . [SEP] DE: Ein Mann sitzt auf einer Bank und hält seine n Hund und schaut auf das Wasser . [SEP]

Table content: <table><tr><td>Model</td><td>Inference</td><td colspan="3">Test2016 Test2017 MSCOCO</td></tr><tr><td>Bilingual (EN → DE)</td><td>EN → DE</td><td>36.14</td><td>28.32</td><td>24.15</td></tr><tr><td>Bilingual (EN ↔ DE)</td><td>EN → DE</td><td>37.08</td><td>28.69</td><td>26.11</td></tr><tr><td>Multi-target (EN → Rest)</td><td>EN → DE</td><td>36.83</td><td>28.35</td><td>25.14</td></tr><tr><td/><td>EN → FR,CS,DE</td><td>35.41</td><td>29.69</td><td>25.64</td></tr><tr><td>Multi-target (Any → Rest)</td><td>EN → DE</td><td>36.63</td><td>28.37</td><td>26.98</td></tr><tr><td/><td>EN → FR,CS,DE</td><td>36.51</td><td>28.53</td><td>25.84</td></tr><tr><td>Joint (p(EN, F R, CS, DE))</td><td>EN → DE</td><td>33.06</td><td>23.42</td><td>21.39</td></tr><tr><td/><td>EN → FR,CS,DE</td><td>32.53</td><td>23.78</td><td>20.97</td></tr></table>

Peer review (iclr2020):
This paper proposes a multichannel generative language model (MGLM), which models the joint distribution p(channel_1, ..., channel_k) over k channels. MGLM can be used for both conditional generation (e.g., machine translation) and unconditional sampling. In the experiments, MGLM uses the Multi30k dataset where multiple high quality channels are available, in the form of multilingual translations. I feel that this paper is not ready for publication at ICLR due to the following major issues: * Missing important related work: This paper seems unaware of an important related work *Multi-Task Learning for Multiple Language Translation* by Dong et al, ACL 2015. In fact, Dong et al. investigated the problem of learning a machine translation model that can simultaneously translate sentences from one source language to multiple target languages. Although machine translation is just an example of MGLM, Dong et al. is highly relevant to the conditional generation with MGLM, needless to say that they share the same multi-language translation problem domain. Thus, this paper will be much stronger if comparison with important baseline methods is provided. * Limited novelty: This paper extends Chan et al.*s KERMIT by applying its objective on tasks with more than 2 sequences, in order to learn the joint distribution p(channel_1, ..., channel_k) over k channel sequences. Most of the math in this paper can be found in the original Chan et al.*s paper. The extension to the multichannel case is incremental as it is hard to justify the challenge of such extensions. Besides, as minor suggestions, it would help readers if more illustrations of Figure 1 (especially the inference part) can be provided.