AriEL: volume coding for sentence generation comparisons

1 Introduction . Representation regularization , through the normalization and bounding of data , representations and gradients , is fundamental to fast deep learning training ( Ioffe and Szegedy , 2015 ; Kingma and Welling , 2014 ; He et al. , 2015 ; Perez et al. , 2018 ) . However , it seldom offers guarantees for boundedness , only encouraging it through initial conditions and loss summands . The final conditions , i.e . the representations learned , can be empirically explored through the sampling of the latent space . This exercise reveals how often the data not seen during training has no bounded representation , which can be regarded as undesirable if we want architectures that can quickly generalize outside the training bias for successful transfer learning . However it is difficult to find the learned patterns through latent sampling , since typically neural networks map an input to a point in Rd ( Hochreiter and Schmidhuber , 1997 ; Vaswani et al. , 2017 ; LeCun et al. , 1989 ) . Some models do map inputs to volumes , to ease retrieval through random sampling . Variational Autoencoders ( Kingma and Welling , 2014 ; Bowman et al. , 2016 ; Chen et al. , 2018 ) encourage volume representations : by encoding an input into a probability distribution that is sampled before decoding , neighbouring points in Rd can end up representing one input . However , it requires two loss summands , a log-prior and a log-likelihood , that fight for two different causes . A smooth and volumetric representation , encouraged by the log-prior regularization , can worsen performance , encouraged by the log-likelihood . By giving partially up on smoothness , we propose AriEL , a method to construct volumes , without a loss to encourage them . It maps sentences to volumes in Rd for efficient retrieval with random sampling , or a network that operates in its continuous space . It fuses arithmetic coding ( AC ) ( Elias and Abramson , 1963 ) and k-d trees ( KdT ) ( Bentley , 1975 ) , so we name it Arithmetic coding and k-d trEes for Language ( AriEL ) . For simplicity we focus on dialogue language , even if AriEL works with any variable length sequence of symbols . AriEL can be used as a benchmark to understand natural language processing and generation models use of latent space . It fills completely the latent space with the language learned , using information theory , and bounding its representations within [ 0 , 1 ] d. Its language model splits the latent space in volumes , guided by the probability assigned to the next symbol in a sentence . It can provide an agent with a simpler interface with a pretrained language model , e.g . a GPT-2 ( Radford et al. , 2019 ; Wolf et al. , 2020 ) , where the agent could choose the optimal d. We prove how such a volume representation eases the retrieval of stored learned patterns and how to use it to set references for other models . Our contributions are therefore : • AriEL , a novel unsupervised volume representation based on arithmetic coding and k-d trees ( Section 3.1 ) , to retrieve learned patterns with random sampling ; • the HouseQ dataset , consisting of a large context-free grammar and a random bias ( Section 3.3 ) , to automatically generate and evaluate sentences generated by trained models , and find them in their latent space ; • the notion that explicit volume coding ( Section 4 ) can be a useful technique in tasks that involve the generation of sequences of discrete symbols , such as sentences ; • the observation that conventional learned codes like AE , VAE or Transformer , do not use the latent space effectively ( Section 4 ) , in the AriEL entropic coding sense . 2 Related Work . Volume codes : We define a volume code as two functions , an encoder and a decoder functions , where the encoder maps an input x into a set that contains compact and connected sets of Rd ( Munkres , 2018 ) , and the decoder maps every point within that set back to x . It is a distributed representation ( Hinton et al. , 1984 ) since the input x is represented by at least one Rd point . We call the volume code implicit , when the volumes are encouraged through a loss term ( Bengio et al. , 2013 ; Ng and Jordan , 2002 ; Kingma and Welling , 2014 ; Jebara , 2012 ) and explicit , when the volumes are constructed through the model ’ s operations , independently from any loss and optimizer choice . Sentence generation through random sampling : Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) map random samples to a learned generation through a 2-players game procedure . Yu et al . ( 2017 ) ; Kusner and Hernández-Lobato ( 2016 ) ; Scialom et al . ( 2020 ) significantly improved GAN performance in text generation . Random sampling the latent space is used as well by Variational Autoencoders ( VAE ) ( Kingma and Welling , 2014 ) , to smooth their representations . Bowman et al . ( 2016 ) ; Yang et al . ( 2017 ) ; Li et al . ( 2021 ) , and others , have refined their performance for text representation . AriEL can be used as a generator or a discriminator in a GAN , or as an encoder or a decoder in an autoencoder . However it differs from them in the explicit procedure to construct volumes . It fills the entire latent space with the learned patterns , to ease retrieval by uniform sampling . Arithmetic coding and neural networks : AC is one of the most efficient lossless data compression techniques ( Elias and Abramson , 1963 ; Witten et al. , 1987 ) . AC assigns a sequence to a segment in [ 0 , 1 ] with length proportional to its frequency . When converted into bits , frequent symbols take less bits than unfrequent . AC is used for neural network compression ( Wiedemann et al. , 2019 ) and neural networks are used in AC to perform prediction based compression ( Jiang et al. , 1993 ; Pasero and Montuori , 2003 ; Tatwawadi , 2018 ) . We generalize AC to Rd , to combine its properties with the properties of highdimensional spaces , neural networks domain . K-d trees and neural networks : KdT ( Bentley , 1975 ) is a data structure for storage that can handle different types of queries efficiently . It is typically used as a fast approximation to k-nearest neighbours in low dimensions ( Friedman et al. , 1977 ) . It gives a binary label to the data with respect to its median . It moves through the k dimensions of the data and repeats the process . Neural networks are typically used in conjuction with KdT to reduce the dimensionality of the search space , for KdT to be able to perform queries efficiently ( Woodbridge et al. , 2018 ; Yin et al. , 2017 ; Vasudevan et al. , 2009 ) . We use KdT to make sure that the multidimensional AC uses all the space available . 3 Methodology . 3.1 AriEL : volume coding of language in continuous spaces ac S a | b | aa | ab | ac | bc | abc | bcc aa abc ab b bc bcc a 0 1 0 0 1 1 b bc a aa ab ac Arithmetic Coding AriEL 0 0 1 1 b a d1 = 1 mod 2 d1= 1 0 0 1 1 b bc bcc a aa ab ac abc d2 = 2 mod 2 d2 = 0 d3 = 3 mod 2 d3 = 1 grammar and bias Figure 1 : Arithmetic coding and AriEL . In this illustrative example , the generating context-free grammar ( CFG ) is S → a|b|aa|ab|ac|bc|abc|bcc , and the bar plot on top indicates the frequency of those sentences in the dataset , as an extra bias to the language . Arithmetic Coding ( middle ) encodes any sequence of this CFG over a single dimension within [ 0 , 1 ] , and the frequency of the sentence determines the length assigned on that segment . AriEL ( bottom ) is a multidimensional extension of AC ( here in 2D ) , where the frequency information is preserved in the volumes . The Language Model provides the boundaries where the next symbols are to be found . For a 2D latent space , d = 2 , the axis to split to find symbol st is dt = t mod d. dt = 0 , 1 represent the horizontal and vertical axis . AriEL maps the sequence/sentence ( s1 , · · · , sn ) = ( st ) nt=1 of length n , to a d-dimensional volume of size P ( ( s1 , · · · , sn ) ) = Πnt=1P ( st| ( st′ ) t′ < t ) in the [ 0 , 1 ] d hypercube . When the symbol is used as a random variable we refer to it as s , s′ , while st represents the observed sample at time step t. The words belong to a finite vocabulary s ∈ { 1 , · · · , Vsize } , Vsize ∈ N. To adapt KdT to more splits than binary , we split axis dt = t mod d , into Vsize segments , one for each possible st . The segment has length proportional to st probability . Then we turn to the following axis dt+1 , and continue the process of splitting and turning ( figure 1 and algorithm 1 ) . In figure 1 , st ∈ { a , b , c , } . The initial token s1 = a is given a portion on d1 of length P ( a ) , larger than the portion given to s1 = b or s1 = c , since there are less sentences that start with b than with a , and there is none that starts with c : P ( a ) > P ( b ) > P ( c ) = 0 . Then , we split d2 according to the probability of the next symbol s2 . In this case the second most likely symbol after symbol s1 = a is s2 = c , so ac ends with a larger volume than aa , ab and a . For sentences longer than d , next symbol is assigned an axis d3 previously split , but only the volume selected up to t− 1 is further split . So , the sentence abc takes a portion of ab equal to P ( c| ( ab ) ) , while ‘ ab ’ takes a portion equal to P ( | ( ab ) ) . We estimate language statistics with a Language Model ( LM ) , PLM ( st| ( st′ ) t′ < t ) . This will approximate the frequency information that makes AC entropically efficient . The sentence is finally encoded as the center of the volume bounded by those segments for simplicity , and any point within it is decoded to the same sentence . The extension to a larger [ a , b ] d hypercube is straightforward , and could provide higher precision , but we restrict ourselves to [ 0 , 1 ] d. AriEL has a computational complexity of O ( nD2Vsize ) for encoding and decoding ( algorithm 1 ) , where n is the length of the sequence , D is the dimensionality of the LM latent space , and Vsize the vocabulary size . AriEL has a minimum number of sequential operations of O ( n ) for both encoding and decoding , on par with conventional seq2seq recurrent networks .

Summary: This paper introduces an entropic coding method that is able to compress a sentence into a latent space and perform text generation through a uniform sampling from the implicit latent space. Experiments are conducted on a toy dataset and a narrow domain text dataset to show the effectiveness of the proposed method w.r.t. AE, VAE, and Transformer models.