Multichannel Generative Language Models

1 INTRODUCTION . A natural way to consider two parallel sentences in different languages is that each language is expressing the same underlying meaning under a different viewpoint . Each language can be thought of as a transformation that maps an underlying concept into a view that we collectively agree is determined as ‘ English ’ or ‘ French ’ . Similarly , an image of a cat and the word ‘ cat ’ are expressing two views of the same underlying concept . In this case , the image corresponds to a high bandwidth channel and the word ‘ cat ’ to a low bandwidth channel . This way of conceptualizing parallel viewpoints naturally leads to the formulation of a fully generative model over each instance , where the transformation corresponds to a particular generation of the underlying view . We define each of these views as a channel . As a concrete example , given a parallel corpus of English and French sentences , English and French become two channels and the corresponding generative model becomes p ( English , French ) . One key advantage to this formulation is that single model can be trained that can capture the full expressivity of the underlying concept , allowing us to compute conditionals and marginals along with the joint . In the case of parallel sentences , the conditionals correspond to translations from one channel to another while the marginals correspond to standard monolingual language models . In this work , we present a general framework for modeling the joint distribution p ( x1 , ... , xk ) over k channels . Our framework marginalizes over all possible factorizations of the joint distribution . Subsequently , this allows our framework to perform , 1 ) unconditional generation and 2 ) conditional generation . We harness existing recent work on insertion-based methods that utilize semi-autoregressive models that are permutation-invariant to the joint factorization . Specifically , we show a proof-of-concept multichannel modeling by extending KERMIT ( Chan et al. , 2019 ) to model the joint distribution over multiple sequence channels . Specifically , we train KERMIT on the Multi30K ( Elliott et al. , 2016 ) machine translation task , consisting of four lan- guages : English ( EN ) , French ( FR ) , Czech ( CS ) , and German ( DE ) . One advantage of multilingual KERMIT is during inference , we can generate translation for a single target language , or generate translations for k− 1 languages in parallel in logarithmic time in the token length per language . We illustrate qualitative examples for parallel greedy decoding across languages and sampling from the joint distribution of the 4 languages . The key contributions in this work are : 1 . We present MGLM , a multichannel generative modeling framework . MGLM models the joint distribution p ( x1 , . . . , xk ) over k channels . 2 . We demonstrate both conditional generation ( i.e. , machine translation ) and unconditional sampling from MGLM . 3 . In the case of conditional generation over multiple languages , we show that not only we are competitive in BLEU , but also with significant advantages in inference time and model memory savings . 4 . We analyze the Quality-Diversity tradeoff from sampling MGLM and prior work . We highlight that while we focus on languages as a specific instantiation of a channel , our framework can generalize to any arbitrary specification , such as other types of languages or other modalities . 2 BACKGROUND . Traditional autoregressive sequence frameworks ( Sutskever et al. , 2014 ; Cho et al. , 2014 ) model the conditional probability p ( y | x ) of an output sequence y conditioned on the input sequence x with a left-to-right factorization . The model decomposes p ( y | x ) as predicting one output token at time , conditioning on the previously generated output tokens y < t and the input sequence x : p ( y | x ) = ∏ t p ( yt | , y < t ) ( 1 ) Recent encoder-decoder models with attention such as Transformer ( Vaswani et al. , 2017 ) have been successfully applied to various domains , including machine translation . If we were to apply this left-to-right autoregressive approach towards multichannel modeling , we would require to choose a particular factorization order , such as p ( w , x , y ) = p ( w ) p ( x|w ) p ( y|x , w ) . Instead of assuming a fixed left-to-right decomposition , recent autoregressive insertion-based conditional modeling frameworks ( Stern et al. , 2019 ; Welleck et al. , 2019 ; Gu et al. , 2019 ) consider arbitrary factorization of the output sequence by using insertion operation , which predicts both ( 1 ) content token c ∈ C from the vocabulary , and ( 2 ) location l insert , relative to the current partial output ŷt : p ( c , l|x , ŷt ) = InsertionTransformer ( x , ŷt ) ( 2 ) Subsequent work , KERMIT ( Chan et al. , 2019 ) , simplified the Insertion Transformer model by removing the encoder and only having a decoder , and the trick is to concatenate the original input and output sequence as one single sequence and optimize over all possible factorizations . Consequently , KERMIT is able to model the joint p ( x , y ) , conditionals p ( x | y ) , p ( y | x ) , as well as the marginals p ( x ) , p ( y ) . Unlike with the left-to-right autoregressive approach , the exact computation of the log-likelihood equation 3 is not possible due to the intractable marginalization over the generation order z , where Sn denotes the set of all possible permutations on n elements . However , we can lower bound the log-likelihood using Jensen ’ s inequality : log p ( x ) = log ∑ z∈Sn p ( z ) p ( x | z ) ( 3 ) ≥ ∑ z∈Sn p ( z ) log p ( x | z ) = : L ( x ) ( 4 ) The loss term can be simplified by changing the summation and careful decomposition of the permutation , leading to : L ( x ) = ∑ z∈Sn p ( z ) log n∏ i=1 p ( ( czi , l z i ) | x z , i−1 1 : i−1 ) = n∑ i=1 ∑ z1 : i−1 p ( z1 : i−1 ) ∑ zi p ( zi | z1 : i−1 ) log p ( ( czi , lzi ) | x z , i−1 1 : i−1 ) Inference can be autoregressive via greedy decoding : ( ĉ , l̂ ) = argmax c , l p ( c , l|x̂t ) , ( 5 ) or partially autoregressive via parallel decoding : ĉl = argmax c p ( c | l , x̂t ) , ( 6 ) which is achieved by inserting at all non-finished slots . Stern et al . ( 2019 ) has shown that using a binary tree prior for p ( z ) led to ≈ log2 n iterations for n token generation . 3 MULTICHANNEL GENERATIVE LANGUAGE MODELS . In multichannel generative language modeling , our goal is to learn a generative model given a dataset consisting of a set of sequences { x ( i ) 1 , . . . , x ( i ) k } Mi=1 from up to k channels , where x ( i ) k = [ x ( i ) j,1 , . . . , x ( i ) j , n ] represents a sequence of tokens from the j-th channel for the i-th example . The resulting MGLM models a joint generative distribution over multiple channels . While there are many possible implementation of Multichannel Generative Language Models , we chose to extended the work of Chan et al . ( 2019 ) to investigate applying the KERMIT objective on tasks with more than 2 sequences , in order to learn the joint distribution p ( x1 , . . . , xk ) over k channel sequences . For example , these channel sequences can denote different languages , such as learning p ( EN , FR , CS , DE ) . Data for Model TargetSource Bilingual ( Uni-direction ) Joint [ SEP ] [ SEP ] EN FR CS [ SEP ] DE [ SEP ] [ SEP ] [ SEP ] EN FR CS [ SEP ] DE [ SEP ] Multi-Target ( Any to Rest ) [ SEP ] [ SEP ] EN FR CS [ SEP ] DE [ SEP ] [ SEP ] [ SEP ] EN FR CS [ SEP ] DE [ SEP ] [ SEP ] [ SEP ] EN FR CS [ SEP ] DE [ SEP ] Legend [ SEP ] [ SEP ] EN FR FR [ SEP ] [ SEP ] EN [ SEP ] [ SEP ] [ SEP ] EN DE [ SEP ] [ SEP ] [ SEP ] EN [ SEP ] [ SEP ] EN [ SEP ] [ SEP ] EN DE [ SEP ] [ SEP ] EN [ SEP ] Training Sets Inference D ec od in g Ite ra tio n [ SEP ] EN [ SEP ] [ SEP ] [ SEP ] EN [ SEP ] D ec od in g Ite ra tio n We illustrate an example data input consisting of 3 channels in Figure 1 ( left ) . We concatenate the sequences together from all channels for each example , separate by a SEP token . Even with shared vocabulary , each channel results in a different token embedding , via an addition of a channel-specific ( learnable ) embedding , or simply having a separately learned token embedding per channel . After passing through the dense self-attention layers as in per Transformer architecture , the contextualized representation at each output time step predicts the possible tokens to be inserted to the left of the current input token . At inference ( generation ) time , we can generate unconditionally by seeding the canvas with the [ SEP ] token and predicting the first actual token , or provide as much , or as little , partial/complete sequence in each channel . Figure 1 ( right ) shows two possible decoding inference modes : a single target language channel ( top ) , or multiple target language channels in parallel ( bottom ) . 4 EXPERIMENTS . We experiment on a multilingual dataset to demonstrate that we can learn MGLM . We perform both qualitative and quantitative experiments . We highlight the model ’ s capabilities ranging from conditional generation ( i.e. , machine translation ) to unconditional sampling the joint distribution over multiple languages . We experiment on the Multi30k ( Elliott et al. , 2016 ; 2017 ; Barrault et al. , 2018 ) , a multilingual dataset which consists of 29000 parallel training sentences in English ( EN ) , French ( FR ) , Czech ( CS ) , and German ( DE ) sentences . We use Multi30k because multiple high quality channels ( multilingual translations in this case ) is readily available to highlight our framework . We implement MGLM as a base Transformer decoder , without any causal masking , with 6 hidden layers and 1024 dimensional hidden representation . We concatenate all 4 language raw text training examples and use SentencePiece ( Kudo & Richardson , 2018 ) to learn an universal subword unigram ( Kudo , 2018 ) tokenizer with a shared 32K vocabulary size . We follow a similar training set up to BERT ( Devlin et al. , 2019 ) , using Adam ( Kingma & Ba , 2015 ) optimizer with learning rate of 1e-4 , warmup over the first 10 % of the total training iterations varying between 10k to 50k iterations . We can train 3 different variants of MGLM by altering the sampling ratio of training data seen by the model : 1 . Bilingual ( e.g. , EN→ FR ) . We give the model a fully observed source ( e.g. , EN ) , and ask the model to infill the target ( e.g. , FR ) . 2 . Multi-target ( e.g. , any 1→ Rest ) . We give the model a fully observed source ( e.g. , EN ) , and ask the model to infill the rest of the targets ( e.g. , DE , FR , CS ) . 3 . Joint . We ask the model to infill all the targets , consequently we learn a joint distribution over all the languages p ( en , fr , de , cs ) .

Summary: This work is an extension of KERMIT (Chan et al., 2019) to multiple languages and the proposed model is called “multichannel generative language models”. KERMIT is an extension of “Insertion Transformer” (Stern et. al, 2019), a non-autoregressive model that can jointly determine which word and which place the translated words should be inserted. KERMIT shares the encoder and decoder of insertion Transformer, and the source sentence and target sentence are concatenated to train a generative model (also, various loss functions are included). In this work, parallel sentences from more than two languages are concatenated together and fed into KERMIT. Each language is associated with a language embedding. This work demonstrates that a joint distribution p(x1, . . . , xk) over k channels/languages can be properly modeled through a single model. The authors carry out experiments on multi30k dataset.