AriEL: volume coding for sentence generation comparisons

1 Introduction . Representation regularization , through the normalization and bounding of data , representations and gradients , is fundamental to fast deep learning training ( Ioffe and Szegedy , 2015 ; Kingma and Welling , 2014 ; He et al. , 2015 ; Perez et al. , 2018 ) . However , it seldom offers guarantees for boundedness , only encouraging it through initial conditions and loss summands . The final conditions , i.e . the representations learned , can be empirically explored through the sampling of the latent space . This exercise reveals how often the data not seen during training has no bounded representation , which can be regarded as undesirable if we want architectures that can quickly generalize outside the training bias for successful transfer learning . However it is difficult to find the learned patterns through latent sampling , since typically neural networks map an input to a point in Rd ( Hochreiter and Schmidhuber , 1997 ; Vaswani et al. , 2017 ; LeCun et al. , 1989 ) . Some models do map inputs to volumes , to ease retrieval through random sampling . Variational Autoencoders ( Kingma and Welling , 2014 ; Bowman et al. , 2016 ; Chen et al. , 2018 ) encourage volume representations : by encoding an input into a probability distribution that is sampled before decoding , neighbouring points in Rd can end up representing one input . However , it requires two loss summands , a log-prior and a log-likelihood , that fight for two different causes . A smooth and volumetric representation , encouraged by the log-prior regularization , can worsen performance , encouraged by the log-likelihood . By giving partially up on smoothness , we propose AriEL , a method to construct volumes , without a loss to encourage them . It maps sentences to volumes in Rd for efficient retrieval with random sampling , or a network that operates in its continuous space . It fuses arithmetic coding ( AC ) ( Elias and Abramson , 1963 ) and k-d trees ( KdT ) ( Bentley , 1975 ) , so we name it Arithmetic coding and k-d trEes for Language ( AriEL ) . For simplicity we focus on dialogue language , even if AriEL works with any variable length sequence of symbols . AriEL can be used as a benchmark to understand natural language processing and generation models use of latent space . It fills completely the latent space with the language learned , using information theory , and bounding its representations within [ 0 , 1 ] d. Its language model splits the latent space in volumes , guided by the probability assigned to the next symbol in a sentence . It can provide an agent with a simpler interface with a pretrained language model , e.g . a GPT-2 ( Radford et al. , 2019 ; Wolf et al. , 2020 ) , where the agent could choose the optimal d. We prove how such a volume representation eases the retrieval of stored learned patterns and how to use it to set references for other models . Our contributions are therefore : • AriEL , a novel unsupervised volume representation based on arithmetic coding and k-d trees ( Section 3.1 ) , to retrieve learned patterns with random sampling ; • the HouseQ dataset , consisting of a large context-free grammar and a random bias ( Section 3.3 ) , to automatically generate and evaluate sentences generated by trained models , and find them in their latent space ; • the notion that explicit volume coding ( Section 4 ) can be a useful technique in tasks that involve the generation of sequences of discrete symbols , such as sentences ; • the observation that conventional learned codes like AE , VAE or Transformer , do not use the latent space effectively ( Section 4 ) , in the AriEL entropic coding sense . 2 Related Work . Volume codes : We define a volume code as two functions , an encoder and a decoder functions , where the encoder maps an input x into a set that contains compact and connected sets of Rd ( Munkres , 2018 ) , and the decoder maps every point within that set back to x . It is a distributed representation ( Hinton et al. , 1984 ) since the input x is represented by at least one Rd point . We call the volume code implicit , when the volumes are encouraged through a loss term ( Bengio et al. , 2013 ; Ng and Jordan , 2002 ; Kingma and Welling , 2014 ; Jebara , 2012 ) and explicit , when the volumes are constructed through the model ’ s operations , independently from any loss and optimizer choice . Sentence generation through random sampling : Generative Adversarial Networks ( GAN ) ( Goodfellow et al. , 2014 ) map random samples to a learned generation through a 2-players game procedure . Yu et al . ( 2017 ) ; Kusner and Hernández-Lobato ( 2016 ) ; Scialom et al . ( 2020 ) significantly improved GAN performance in text generation . Random sampling the latent space is used as well by Variational Autoencoders ( VAE ) ( Kingma and Welling , 2014 ) , to smooth their representations . Bowman et al . ( 2016 ) ; Yang et al . ( 2017 ) ; Li et al . ( 2021 ) , and others , have refined their performance for text representation . AriEL can be used as a generator or a discriminator in a GAN , or as an encoder or a decoder in an autoencoder . However it differs from them in the explicit procedure to construct volumes . It fills the entire latent space with the learned patterns , to ease retrieval by uniform sampling . Arithmetic coding and neural networks : AC is one of the most efficient lossless data compression techniques ( Elias and Abramson , 1963 ; Witten et al. , 1987 ) . AC assigns a sequence to a segment in [ 0 , 1 ] with length proportional to its frequency . When converted into bits , frequent symbols take less bits than unfrequent . AC is used for neural network compression ( Wiedemann et al. , 2019 ) and neural networks are used in AC to perform prediction based compression ( Jiang et al. , 1993 ; Pasero and Montuori , 2003 ; Tatwawadi , 2018 ) . We generalize AC to Rd , to combine its properties with the properties of highdimensional spaces , neural networks domain . K-d trees and neural networks : KdT ( Bentley , 1975 ) is a data structure for storage that can handle different types of queries efficiently . It is typically used as a fast approximation to k-nearest neighbours in low dimensions ( Friedman et al. , 1977 ) . It gives a binary label to the data with respect to its median . It moves through the k dimensions of the data and repeats the process . Neural networks are typically used in conjuction with KdT to reduce the dimensionality of the search space , for KdT to be able to perform queries efficiently ( Woodbridge et al. , 2018 ; Yin et al. , 2017 ; Vasudevan et al. , 2009 ) . We use KdT to make sure that the multidimensional AC uses all the space available . 3 Methodology . 3.1 AriEL : volume coding of language in continuous spaces ac S a | b | aa | ab | ac | bc | abc | bcc aa abc ab b bc bcc a 0 1 0 0 1 1 b bc a aa ab ac Arithmetic Coding AriEL 0 0 1 1 b a d1 = 1 mod 2 d1= 1 0 0 1 1 b bc bcc a aa ab ac abc d2 = 2 mod 2 d2 = 0 d3 = 3 mod 2 d3 = 1 grammar and bias Figure 1 : Arithmetic coding and AriEL . In this illustrative example , the generating context-free grammar ( CFG ) is S → a|b|aa|ab|ac|bc|abc|bcc , and the bar plot on top indicates the frequency of those sentences in the dataset , as an extra bias to the language . Arithmetic Coding ( middle ) encodes any sequence of this CFG over a single dimension within [ 0 , 1 ] , and the frequency of the sentence determines the length assigned on that segment . AriEL ( bottom ) is a multidimensional extension of AC ( here in 2D ) , where the frequency information is preserved in the volumes . The Language Model provides the boundaries where the next symbols are to be found . For a 2D latent space , d = 2 , the axis to split to find symbol st is dt = t mod d. dt = 0 , 1 represent the horizontal and vertical axis . AriEL maps the sequence/sentence ( s1 , · · · , sn ) = ( st ) nt=1 of length n , to a d-dimensional volume of size P ( ( s1 , · · · , sn ) ) = Πnt=1P ( st| ( st′ ) t′ < t ) in the [ 0 , 1 ] d hypercube . When the symbol is used as a random variable we refer to it as s , s′ , while st represents the observed sample at time step t. The words belong to a finite vocabulary s ∈ { 1 , · · · , Vsize } , Vsize ∈ N. To adapt KdT to more splits than binary , we split axis dt = t mod d , into Vsize segments , one for each possible st . The segment has length proportional to st probability . Then we turn to the following axis dt+1 , and continue the process of splitting and turning ( figure 1 and algorithm 1 ) . In figure 1 , st ∈ { a , b , c , } . The initial token s1 = a is given a portion on d1 of length P ( a ) , larger than the portion given to s1 = b or s1 = c , since there are less sentences that start with b than with a , and there is none that starts with c : P ( a ) > P ( b ) > P ( c ) = 0 . Then , we split d2 according to the probability of the next symbol s2 . In this case the second most likely symbol after symbol s1 = a is s2 = c , so ac ends with a larger volume than aa , ab and a . For sentences longer than d , next symbol is assigned an axis d3 previously split , but only the volume selected up to t− 1 is further split . So , the sentence abc takes a portion of ab equal to P ( c| ( ab ) ) , while ‘ ab ’ takes a portion equal to P ( | ( ab ) ) . We estimate language statistics with a Language Model ( LM ) , PLM ( st| ( st′ ) t′ < t ) . This will approximate the frequency information that makes AC entropically efficient . The sentence is finally encoded as the center of the volume bounded by those segments for simplicity , and any point within it is decoded to the same sentence . The extension to a larger [ a , b ] d hypercube is straightforward , and could provide higher precision , but we restrict ourselves to [ 0 , 1 ] d. AriEL has a computational complexity of O ( nD2Vsize ) for encoding and decoding ( algorithm 1 ) , where n is the length of the sequence , D is the dimensionality of the LM latent space , and Vsize the vocabulary size . AriEL has a minimum number of sequential operations of O ( n ) for both encoding and decoding , on par with conventional seq2seq recurrent networks .

Summary: The paper proposes to use an autoregressive language model's hidden state to index multidimensional arithmetic coding representations of vocabularies and sentences. This way the latent space's geometry is directly connected to the words chosen at the cost of sensible inductive bias between similar sentences. Evaluations of unconditional samples on toy data and small-scale real data are made. While interpolations are presented, the proposed model and the autoencoders it compares to are only ever evaluated for fluency and grammaticality, but not for semantic relatedness.