AriEL: volume coding for sentence generation comparisons
Authors: Luca Celotti
Simon Brodeur
Jean Rouat
Venue: ICLR 2022 Submitted (2022)

Abstract: Saving sequences of data to a point in a continuous space makes it difficult to retrieve them via random sampling.  Mapping the input to a volume makes it easier, which is the strategy followed by Variational Autoencoders.  However optimizing for prediction and for smoothness, forces them to trade-off between the two.  We analyze the ability of standard deep learning techniques to generate sentences through latent space sampling.  We compare toAriEL, an entropic coding method to construct volumes without the need for extra loss terms. We benchmark on a toy grammar, to automatically evaluate the language learned and generated, and find where it is stored in the latent space. Then, we benchmark on a dataset of human dialogues and using GPT-2 inside AriEL. Our results indicate that the random access to stored information can be improved since AriEL is able to generate a wider variety of correct language by randomly sampling the latent space. This supports the hypothesis that encoding information into volumes, leads to improved retrieval of learned information with random sampling.

Under review as a conference paper at ICLR 2022
ARIEL: VOLUME CODING FOR SENTENCE GENERATION
A Anonymous authors Paper under double-blind review
A BSTRACTT
Saving sequences of data to a point in a contimuous space makes it difficult to retrieve them via random sampling. Mapping the input to volume makes it easier, which is the stralegy followed by Variational Autoencoders. How- ever optimizing for prediction and for smoothness, forces them to trade-off between the two. We analyze the ability of standard deep learning tech- niques to generate sentences through representation sampling. We propose AriEL, entropic coding method to construct volumes without the need for extra loss terms, and compare those standard learning techniques with toy grammar, to auto-
its use of the latent space. We benchmark matically evaluate the language learned and generated, and find where it is stored in the latent space. Then, we benchmark dialogues and using GPT-2 inside AriEL. Our results indicate that the ran- dom access stored information can be improved, since AriEL is able generate a wider variety of correct language by randomly sampling the la- tent space. This supports the hypothesis that encoding information into volumes, improves retricval of learned information with random sampling.
dataset of human
INTRODUCTION
Representation regularization, through the normalization and bounding of data, representa- tions and gradients, fundamental fast deep learning training (Ioffe and Szegedy, 2015; Kingma and Welling, 2014; He et al., 2015; Perez et al., 2018). However, it seldom offers guarantees for boundedness, only encouraging through initial conditions and loss sum- mands. The final conditions, i.e. the representations learned, can be empirically explored through the sampling of the latent space. This exercise reveals how often the data not seen during training has no bounded representation, which can be regarded as undesirable if we want architectures that can quickly generalize outside the training bias for successful trans- fer learning. However it is difficult to find the learned patterns through latent sampling, since typically neural networks map an input to a point in Rd (Hochreiter and Schnidhuber, 1997; Vaswani et. al., 2017; LeCun et. al., 1989).
Some models do map inputs to volumes, to ease retrieval through random sampling. Vari- ational Autoencoders (Kingma and Welling, 2014; Bowman et al., 2016; Chen et al., 2018) encourage volume representations. by encoding an input into a probability distribution that is sampled before decoding, neighbouring points in Rdd can end up representing one input. However, it requires two loss summands, log-prion and a log-likelhhood, that fight for two different causes. A smooth and volumetric representation, encouraged by the log-prior regularization, can worsen performance, encouraged by the log-likelihood.
By giving partially up on smoothness, we propose AriEL, a method to construct volumes, without a loss to encourage them. It maps sentences to volumes in Id for efficient retrieval with random sampling, or a network that operates its continuous space. fuses arithmetic coding (AC) (Elias and Abramson, 1963) and k-d trees (KdT) (Bentley, 1975), so name it Arithmetic coding and k-d trEes for Language (AriEL). For simplicity we focus on dialogue language, even if AriEL works with any variable length sequence of symbols. AriEL can be used as a benchmark to understand natural language processing and generation models use of latent space. It fills completely the latent space with the language learned, using

---

Under review as a conference paper at ICLR 2022
information theory, and bounding its representations within [0, 1j Its language model splits the latent space in volumes, guided by the probability assigned to the next symbol in sentence. It can provide an agent with simpler interface with pretrained language model, e-g. a GPT-2 (Radford et al., 2019; Wolf et al., 2020), where the agent could choose the optimal d. We prove how such volume representation eases the retrieval of stored learned patterns and how to use it to set references for other models.
Our contributions are therefore:
AriEL, a novel unsupervised volume representation based on arithmetic coding and k-d trees (Section 3.1), to retrieve learned patterns with random sampling;
the HouseQ dataset, consisting of a large context-free grammar and a random bias (Section 3.3), to automatically generate and evaluate sentences generated by trained models, and find them in their latent space;
the notion that explicit volume coding (Section 4) can be a useful technique in tasks that involve the generation of sequences of discrete symbols, such sentences;
the observation that conventional learned codes like AE, VAE or Transformer, do not use the latent space effectively (Section 4), in the AriEL entropic coding sense.
2 RELATED WORK
We define a volume code as two functions, an encoder and a decoder func- tions, where the encoder maps input x into set that contains compact and connected sets of Rd (Munkres, 2018), and the decoder maps every point within that set back to æ. It is distributed representation (Hinton et. al., 1984) since the input x is represented by at least. one IR point. We call the volume code implicit, when the volumes are encour- aged through a loss term (Bengio et al., 2013; Ng and Jordan, 2002; Kingma and Welling, 2014; Jebara, 2012) and explicit, when the volumes are constructed through the model's operations, independently from any loss and optimizer choice.
Volume codes:
Sentence generation through random sampling: Generative Adversarial Networks (GAN) (Goodfellow et al., 2014) map random samples to a learned generation through a 2-players game procedure. Yu et al. (2017); Kusner and Hernández-LLobato (2016); Scialom et al. (2020) significantly improved GAN performance text generation. Random sampling the latent space is used as well by Variational Autoencoders (VAE) (Kingma and Welling, 2014), to smooth their representations. Bowman et al. (2016); Yang et al. (2017); Li et al. (2021), and others, have refined their performance for text representation. AriEL can be used as a generator or a discriminator in a GAN, or as an encoder or a decoder in an autoencoder. However it differs from them in the explicit procedure to construct volumes. It fills the entire latent space with the learned patterns, to ease retrieval by uniform sampling.
Arithmetic coding and neural networks: AC is of the most efficient lossless data. compression techniques (Elias and Abramson, 1963; Witten et al., 1987). AC assigns a sequence a segment in [0, 1] with length proportional to its frequency. When converted into bits, frequent symbols take less bits than unfrequent. AC is used for neural network compression (Wiedemann et al., 2019) and neural networks are used in AC to perform prediction based compression (Jiang et al., 1993; Pasero and Montuori, 2003; Tatwawadi, 2018). We generalize AC Rdd, to combine its properties with the properties of high- dimensional spaces, neural networks domain.
KdT (Bentley, 1975) is a data structure for storage that can handle different types of queries efficiently. It is typically used as a fast approximation to k-nearest neighbours in low dimensions (Friedman et al., 1977). It gives a binary label to the data with respect to its median. It moves through the k dimensions of the data and repeats the process. Neural networks are typically used in conjuction with KdT to reduce the dimensionality of the search space, for KdT to be able to perform queries efficiently (Woodbridge et al., 2018; Yin et al., 2017; Vasudevan et al., 2009). We use KdT to make sure that the multidimensional AC uses all the space available.
K-d trees and neural networks:
2

---

Under review as a conference paper at ICLR 2022
3 METHODOLOGY
3.1
ARIEL: VOLUME CODING OF LANGUAGE IN CONTINUOUS SPACES
Arithmetic coding and AriEL. In this illustrative example, the generating context-free grammar (CFG) and the
Figure 1:
| 00
ab
Dc
bc
abc
bcc
is S + (bjjaajabaa/bejjbbl bar plot top indicates the frequency of those sentences in the dataset, as an extra bias to the language. Arithmetic Coding (middle) encodes any sequence of this CFG over a single dimension within [o, 1], and the frequency of the sentence determines the length assigned on that segment. AriEL (bottom) is a multidi- mensional extension of AC (here 2D), where the frequency information is pre- served in the volumes. The Language ac
grammar and bias
ab
bc
D0
abc Arithmetic Coding
bec
bc
bc
80
ab
BC
aa
ab
Model provides the boundaries where the next symbols are to be found. For a 2D latent space, d 2, the axis to split to find symbol 8, is de = mod d. de = 0, represent the horizontal and vertical axis.
0
0
61 1 mod 2
6, = mod
6. - 3 mad
di- 3
1
AriEL
AriEL maps the sequence/sentence (s1..... ..) = (st) of length n, to a d-dimensional volume of size P((s1,... .. sn)) P(s((()))))
in the [0. 1|" hypercube. When the symbol is used a random variable we refer to it as 8,8', while 8t represents the observed sample at time step The words belong to a finite vocabulary 8 E (1,..., Vsize), Vazee E N.
To adapt KdT to more splits than binary, we split axis de = mod d, into Vsize segments, one for each possible st. The segment has length proportional to 8, probability. Then we turn to the following axis de+1, and continue the process of splitting and turning (figure 1 and algorithm 1). In figure 1, St E ba, -. The initial token $1 = is given a portion on dN of length P(a). larger than the portion given to 81 b or 81 c, since there are less sentences that start with than with a, and there is none that starts with c: P(a) > P(b) > P(c) O. Then, we split d2 according to the probability of the next symbol 82. In this case the second most likely symbol after symbol 81 = a is 82 = c, so ac ends with a larger volume than aa, ab and a.. For sentences longer than d, next symbol is assigned an axis da previously split, but only the volume selected up to t f-1 is further split. So, the sentence abc takes a portion of ab equal to P(c (ab)), while 'ab_' takes a portion equal P(-I(ab)). We estimate language statistics with Language Model (LM), PLas (st (ez)). This will approximate the frequency information that makes AC entropically efficient.
The sentence is finally encoded as the center of the volume bounded by those segments for simplicity, and any point within is decoded to the same sentence. The extension to larger [a, bjd hypercube is straightforward, and could provide higher precision, but we restrict ourselves to [0, 1jd.
AriEL has computational complexity of O(nDDViize for encoding and decoding (algorithm 1), where n is the length of the sequence, D is the dimensionality the LM latent space, and Vaize the vocabulary size. AriEL has minimum number of sequential operations of Onn) for both encoding and decoding, on par with conventional seq2seq recurrent networks.
3.2 NEURAL NETWORKS: MODELS AND EXPERIMENTAL CONDITIONS
We compare ArEEL to standard approaches to map variable length discrete spaces to fixed length continuous spaces. These are the sequence to sequence recurrent autoencoders (AE) (Sutskever et al., 2014), their variational version (VAE) (Bowman et al., 2016) and Tran-
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1 AriEL encoder and decoder Bup and Brow stand for the upper and lower bounds that define AriEL volumes, main differences between encoder and decoder in blue. The PLM cumulative distributions (Cup: Clow) define the volume's limits and size (a). and represent the vector words in the vocabulary, and s the observed value at time t. (Left) AriEL. Encoding: from sentence to continuous space. The volumes represented by their central point z for simplicity. (Right) AriEL decoding: from continuous space to sentence. 2 used identify which volume has be picked next. Operation find is defined in SM10.
AriEL Encoding
AriEL Decoding
Input: sentence: = (st)²- Output: represents S in [0, 1]d
Input: 2 represents in [0, 1]d
Output: sentence:
(st)
1: function ARIEL.ENCODE(S
function ARIEL DECODE(Z) d = dimension (z)
2: latent space dimension
2:
3: Brow zeros(d)
Blow = zeros(d)
4: Bup = ones(d)
4:
Bup = ones(d)
5:
length(S))
5:
S = ART
6:
for 0,...
do D choose dimension to split
6:
for t= =.... Rmax -1 do
choose dimension to unsplit d, = mod Pnext()) (s
7:
di =t mod d Pnext(s) = PLAss(s (s
7:
8:
8:
9:
Clow (8) =>==>>>> E.>r P Prenee
9:
Clow (8) = - Preze(s')
10:
(sp(s) = -E>oo- P..Panc Pneze
10:
Cup (
11:
Bup Bup(ch) (low(dd)
11:
a Bup(d)) Blow
> update volume bounds
update volume bounds
(up(de) Blow (de) ) Cup (5t)
12:
Bsu (s) = Bup(dh + Cup(s)
13:
Blow (de) Blow (de) ) Clow (St)
13:
Bs (s) Brow (de) + Ctowo ($) b ony point in the volume is assigned
end for
represent the volume by its center z= -Btow + Bup)/2
the symbol
15:
14:
= find, Bstoo < z(()) < Bsup(s)
16:
return
17: end function
15:
(d) = Bsup
16:
Biow (de) Bs (st) S= S.append
17:
18:
end for
19:
return S
20:
end function
former (Vaswani et. al., 2017). We trained them for next word prediction, over the biased train set (section 3.3). All can be split into an encoder and a decoder that map the sentences to Fdd and back. More training details in SM7.
In this work, AriEL's language model neural network PLM consists of a word embedding of size 64, a 140-unit LSTM, leedforward layer and softmax. At test time the argmax is not applied directly to the softmax, but the latent space is used as the deterministic pointer that chooses the st, as shown in find definition in SM10. However the LM is trained for next lime step prediction through cross-entropy. For AE and VAF, we stack two GRU layers (Cho et al., 2014) with 128 units at both, the encoder and the decoder. The last encoder layer has either d = 16 units or d 512 for all methods. The decoder outputs a softmax.
Transformer (Vaswani et al., 2017) is the state-of-the-art in many S2S problems (Dai et al., 2019; Radford et al., 2018). Since at the word level it is fixed-length representation and is variable-length at the sentence level, we padded all sentences to the maximum length in the dataset to be able to compare its latent space capacity to the other models. We take as its latent dimension the connection between the encoder and decoder, with dmoddd size, that will take value of 16 512. We choose most parameters in the original work: the
4

---

Under review as a conference paper at ICLR 2022
mumber of attention heads Rhead 8, the key and value dimension dkey dualue = 64, dropout regularization of 0.1. We change the stack of identical decoders and encoders to Ptayers 2, and the dimension of the inner geed-forward network to dy 256 to have number of parameters similar to the other methods. On the Guess dataset (De Vries et al., 2017) we tested Mayers 20 to have an amount of parameters comparable for d = 16 to the other methods, but performed worse than Magers = 2, so we report the smaller one.
All architectures are tested either for next time-step prediction or sampled from the latent space, and all of them are constrained to a small latent space. All of them are 2-layers of encoder and decoder, except for AriEL, since they improved performance compared when they had only one layer. All of them have about the same number of parameters. All of them could be used as Language Models inside AriEL.
3.3 DATASETS: TOY AND HUMAN SENTENCES
We perform our analysis on two datasets. A loy dataset of sentences generated from context-free grammar (CFG) and a realistic dataset of sentences written by humans playing cooperative game.
The toy dataset:
we generate questions about objects with a CFG (Supplementary Ma- terial 1). To stress the models we choose CFG with large vocabulary and numerous grammar rules, rather than classic alternatives (e.g. REBER, Hochreiter and Schmidhuber call it the HouseQ dataset.
(1997)). All are questions about house objects,
We distinguish between unbiased sentences, simply sampled from the CFG, and biased sentences, selected according to additional structural constraint after being sampled from the CFG. To do so we generate an adjacency matrix of words that can occur together in sentence, and we that to bias the sentences. Once sentence is produced from the CFG, if all its words can be together in sentence judged by the adjacency matrix, biased, and unbiased otherwise. For simplicity the adjacency
the sentence is considered matrix is random matrix of zeros and ones, generated once for all the experiments, making sure that some symbols like the, it or 7, can be found in all sentences. We want emulate a CFG constrained by realistic scenes, where not all the grammatically correct sentences are semantically correct: e.g. 'Is it the wooden shower in the kitchen ?' could be grammatical, but semantically incorrect since it is unusual. We use it to detect how each learning method extracts the grammar and the roles of each word, despite a bias that makes it harder.
It has 840 words vocabulary with maximal and mean sentence length of 19 and 9.9 symbols. We split the biased dataset into 1M train, 10k test and 512 validation sentences, with no overlap between them. We created 10k unbiased test sentences with the same CFG, only with sentences that do not follow the adjacency matrix. We train on the biased sentences and test if they grasped the grammar behind, with the unbiased.
The real dataset: we choose the GuessWaat?! dataset (De Vries et al., 2017), a dataset. of questions asked by humans to solve cooperative game. It has a vocabulary of 10,469 words. The maximal and mcan length of the sentences are of 57 and 5.9 symbols.
3.4
EVALUATION METRICS
3.4.1 QUANTITATIVE EVALUATIONS ON THE TOY GRAMMAR, HOUSEQ
Metries chosen take advantage of the possibilities opened by the CFG and the random bias, that allow to automatically check the grammatical and bias correctness of the language produced. Standard metrics like perplexity, only check if the word predicted matches the exact time step the word in reference sentence. In our view, it makes it an obscure metric for language quality, used in the lack of a better one for (non CFG) human language. We measure generation, prediction and generalization quality. We study networks with latent dimcnsion of 16 units, to understand their compression limits, and for 512 units, often taken as the default size (Kingma and Welling, 2014; Vaswani et al., 2017).
inneration/Decoding Quality is evaluated with sentences produced randomly sampling the latent space of each decoder. The sampling done uniformly within the maximal hyper-
5

---

Under review as a conference paper at ICLR 2022
cube defined by the encoded biased test sentences. We sample 10k sentences and evaluate: i) vocabulary coverage (VC) as the ratio of sampled words, over the size the complete vocabulary; ii) uniqueness (U) as ratio of unique sampled sentences; and tii) validity (V) as a ratio of valid sampled sentences, defined as the unique and grammatically correct. sentences, the most important of our metrics.
Prediction Quality is ovaluated by encoding and docoding the 10k biased test sentences as follows: i) prediction accuracy biased (PAB) as ratio of correctly reconstructed sentences (i.e. all words must match); ùi) grammar accuracy (GA) as a ratio of grammatically correct reconstructions (i.e. can be parsed by the CFG, even if the reconstruction not accurate). and iii) bias accuracy (BA) as the ratio of reconstructions that keep the training set bias.
Generalization Quality is evaluated using the 10k unbiased test sentences while the em- beddings were trained on the biased training set. The prediction accuracy unbiased (PA U), as PAB, is the ratio of corroctly reconstructed unbiased sentences. It measures how well the latent space generalizes to grammatically correct sentences outside the training bias.
3.4.2 QUANTITATIVE EVALUATIONS ON THE REAL DATASET, GUESSWHAT?!
Humans use spontancously ungrammatical constructions. We therefore quantify the quality of the language learned with two measures: memorization and subjective interpretability. Memorization is the pgrcentage of sentences that are unique and are found in the training data, indicating how easy it is to retrieve the learned information. To measure subjective interpretability, we asked 5 surveyees to evaluate the interpretability of the generated sen- tences. They were assigned 5 sentences for each method and d, with each sentence from a different training soed. Samples shown to the surveyees can be seen in SM12.
3.4.3 INTERPOLATIONS WITHIN ARIEL
AriEL organization of language in volumes changes with d, using the same Language Model. In figure 2 we show how many valid sentences can be found on a straight line between two random points in [0, 1]4 AriEL's latent space. Valid refers to unique and grammatically correct, in the toy dataset case. Since the average is taken over the segment and over the random pairs of points, we call this metric interpolation diversily. For each d the average is taken over 15 different random sentence pairs, and over 100 interpolation steps on the line that connects them. The Language Model tested is the one trained on the toy grammar.
3.4.4 QUALITATIVE EVALUATIONS
We show (1) samples of reconstruction via next word prediction of unbiased sentences, to understand the generalization capabilities of different models (d = 16, table 3 and S2), (2) generated senlences by uniformly sampling the latent space, after training on the loy datasel, to undersland the generation capabilities (d 16, table 4 and S3) (3) samples generated by AriEL with a pretrained GPT2 as its language model, for small latent spaces (d = 1,5, table 5 and S4) and long sentences (100 symbols), to see if the folding process and the floating point precision represent a limitation for AriEL. To avoid cherry picking, display the first samples produced.
4 RESULTS
4.1 QUANTITATIVE EVALUATIONS
AriEL improves over the rest for all the 7 measures on the toy data (table 1 and figure S2), outperforming them by large margin for validity, i.c. unique and grammatical sentences generated, the most important the metrics. Transformer performs remarkably well at not overfitting and it is able to reconstruct biased and unbiased sentences better than the other non-AriEL methods, even under-pante aada--paramtterized (d= 16). However, it performs very poorly at generating diverse valid sentences by random sampling. VAE 16 despite the poor gener- alization to the biased and the unbiased test set, results in the best non-AriEL generator, measured by validity. The conflict between log-prior and log-likelihood, encouraged VAE to
6

---

Under review as a conference paper at ICLR 2022
Generation
Prodiction
Generalization
prediction accuracy biased
prediction accuracy
vocabulary coverage
bias accuracc
graminan accuracy
param
validity
uniqueness
unbiased
d AriEL
237K
70.4
0.2%
97.8 1: 0.2%
99.7 1 0.1%
100.0
0.0%
100.0
0.0%
100.0 1 0.0%
53.1 0.4%
Transformer 258K
70.1
0.8%
4.7 + 2.7%
99.1
0.5%
99.98
0.01%
99.95
0.02%
99.92
0.02%
49.0 + 0.1%
AE 258K
6.89
0.7%
11.5
4.2%
13.9 + 5. 5.1%
89.5
2.3%
98.0
1.7%
0.0 + 0.1%
0.0 + 0.1%
VATE
258K
11.5
2.6%
16.0
9.2%
24.3
14.8%
85.4
5.2%
85.1
8.8%
0.0 + 0.1%
0.0 0.1%
d 512
AriEL
237K
70.2
0.1%
0.0%
0.0%
100.0
0.3%
Transformer
9M
67.3
0.9%
17:2
6.3%
87.2 = 7.5%
99.99 1 0.01%
99.91
0.03%
99.86 1 0.05%
49.0 1 0.1%
120M
39.3
21.0
71.8
3.5%
1.3%
VAE
120M
28.9
2.4%
26.5
2.4%
95.2 + 3.8%
73.8
2.2%
89.5
2.8%
4.3 + 3.7%
4.9 + 3.6%
Table 1: Evaluation of continuous sentence embeddings on the toy dataset. (d = 16, 512). Each experiment is run 5 times. AriEL, achieves almost perfect perfor- mance in most metrics, especially in validity, which quantifies how many random samples were decoded into a unique and grammatical sentence. Transformer performed exception- ally, except for validity. All methods improved their performance increasing d, particularly in validity, but still achieved less than one third the performance AriEL. VAE the second best in validity, supporting the hepothesis, that volume coding facilitates retrieval of information by random sampling.
interpolation diversity
prediction accuracy
subjoctive Interpeetability
memorization
0.80
AriEL
2,301K
97.2
0.2%
21.9 + 0.4%
100.0 + 0.0%%
Transformer
588IC
58.4
0.4%
0.5 1. 0.4%%
17.6 15.3%
0.70
AE
2,757K
11.9 0.6%
2.4 :: 0.6%%
73.6 31.4%
VAE
2,787K
1.6%
17.1%
512
16
Tel
162
169
AriEL
2,901K
97.3 0.25%
20.9 =: 0.4%
92.0 =: 15.0%
latent dimension
Transformer
18, 50,
58.7 1: 1.7%
3.46 1.7%
45.2 17.4%
AE
4,903K
15.9 2.8%
3.4 0.9%
77.6 16, 1%
VAE
5,025K
27.2
16.7%
Figure 2: Interpolations between ran- dom points in AriEL's latent space, and amount of valid sentences gener- ated in between. For low latent dimen- sion all sentences are very densely packed, and in the extreme of 1, all sentences are found on the same axis. As d increases, the sentences are redistributed in [0, 1]d and less sentences are found in given direction. The lower bound at 0.746 can change depending on the training dataset.
Table 2: Performance on the Guess- What?! Questioner data. For the real dalaset the pattern is repeated: AriEL shows that a larger value of valid sentences is possible. Transformer 16 gave better re- sults when Rlgeere = 2 than Magers = 20, that was tested to increase its learnable pa- rameters from 588K to 2,666K.
look for sentences outside the bias, since it was able produce more grammatically cor- rect sentences, albeit unbiased, than AE. Increasing the learned parameters (d = 512), had no effect on Transformer, that was already excellent in several of the metrics, apart from significant. improvement in validity. However, a larger latent space and the increase in number of parameters that followed, prevented AE, and VAF, from overfitting (better PAU and PAB). When trained human sentences, the What?! dataset, AriEL sets again a large memorization and interpretability margin compared to the other approaches. Generated samples be found in table S5.
In the interpolation diversity study (figure 2) we see that for low d, we have to pass through many sentences in between two random points in the latent space, while we augment the dimensionality, we distribute the sentences different directions. Therefore we find less sentences when we straight line between two random points.

---

Under review as a conference paper at ICLR 2022
Input Sentences
is it huge and teal ?
is the thing transparent, huge and slightly heavy ?
AriEL
AriEL
the objert that thny very light set
is huge and teachable ?
is the thing a tiny destroyable abstraction Transformer
is the thing transparent huge and slightly heavy ?
Transformer
is an tomato slot box made of decoration facing stone? ? is the thing short and spring heavy slightly heavy potang AE
is luge and magenta ?
is the thing transparent, huge and slightly heavy? ?
is the ohject that light light laminate ?
thing a light small und small laminaate
is it this average-sized and average-sized laminate ?
VAE
is the thing very heavy, heavy and very heavy
the thing light and deep office ?
VAE
is light light and light and pink
is the light doep bedroom ?
is the thing textured, textured and moderately heavy"
Table Generation: output of the de- coder when sampled uniformly in the latent space. Red defines grammatically incorrect generations according to the CFG the models are trained on. AriEL. produces an extremely varied set of grammatically correct sentences, most of which keep the bias of the training set. Transformer re- veals itself to be hard to control via random sampling of the latent. space, since it almost never produces correct sentences with this method. AE and VAE manage to produce several different sentences, the latter pro- ducing more non grammatical, but well more varied grammatical ones.
Table 3: Generalization: next word prediction of unbiased sentences at test time. Blue means that the incor- rect. reconstruction complies with the bias and purple means that it is still unbiased. Most reconstructions seem grammatically correct. In practice AriEL also made er- rors. Some failed reconstructions comply with the training bias, some do not. In- terestingly the errors made by Transformer tend to turn the unbiased input sentence into a biased version. AE produced only biased sentences whose structure resembled the unbiased ones. VAE behaved similarly, producing more unbiased sentences.
4.2 QUALITATIVE EVALUATIONS
Table 3 and S2 show the generalization study. AE and VAE fail to generalize to the unbiased language, but both keep the structure of the input sentence. Their behavior improved with d= 512, and its increase of parameters. In theory, AriEL can reconstruct any sequence by design. In practice, it failed slightly less often than Transformer. Both produce reconstruc- tions of the unbiased input at similar rate (table 3, and figure S2). This means that codes for data not seen during training are available for AriEL and Transformer. Instead, the latent space scoms to be taken exclusively by the training set for AE and VAE, since sentences that are not seen during training (e.g. unbiased) cannot be reconstructed.
A very high number (97.6 of AriEL's generations are valid (table and 1). AE and VAE perform well despite the small latent space. VAE triples AE in validity when d = 16 (table 1). Transformer struggles to generate grammatical sentences sampling the latent space. However, increasing d, Transformer, AE and VAE improve in validity, remaining at one third AriEL's reference.
Samples generated with GPT-2 as AriEL's LM can be seen in table 5 and S4, for d = 5, 10, 50, 100. Despite the large vocabulary of GPT-2, 50K subwords, and the long sentences, 100 symbols, the floating point precision and the unit hypercube latent space of AriET., seem to still be able to generate high quality language.
DISCUSSION
AriEL latent space d is a free parameter. The size of the latent space AriEL can be defined at any time, for fixed Language Model. It could therefore be controlled with learnable parameter or with the activity of another neuron. As we increase d, the volumes will have more neighbouring volumes that represent different sentences (figure 2).
8

---

Under review as a conference paper at ICLR 2022
d=1
Where to go from here?'n more attention to when they come in to another site. If you're looking for good stuff to read on other sites, then get on your site and read their storics. If you're looking for things that work for your communily, then look up what their leaderboard is and where they' re based. If you're re looking for things that make me love them, then say hey. You'l 'e really doing your
looking for special interests and fans to pay much
Saul Niguez is not going to let go of his dream of playing for Hillary Clinton. \n The Florida native will plan to accept the Democratic presidential nomination for president on November 8. E.An\'nThe ycar-ollldd machinist has called Trump 's presidential nominee a "posst-moern" bull on social media. AnlmnIn an interview with DailyMail.com Niguez told her you could still name her after Trump. AnwnnnnnTTTTaaa This is my father 's
d=
Utility Shells are just as dependent on human labor as steel, yet they had been enormously more efficient than nuclear power in incongruity's late 70s. US clients including the nation's 's largest. retiree retirement fund already manufactured many of their own carbon- free cars. But what about non-node Americans? How could they simply cool down at home for work for years at time? Inventing these subs would seem like smart thing to do. But there's one
Out featured: on Alcoholism, Marriage and Dating: l'm No Timber [Decision note: 19th August 2013] with Runnymede Colman and producer Dan Patrick((nnnParrt transcript as follows: /n DDDDADDD WEISES; After 20 years of marriage to my Malcastian partner, we've lost our lives to drugs. And so it's our hope that by allowing my Malcastian partner to become an alcoholic and grow up to be successful husband and
Table 5: AriEL samples using GPT-2 as Language Model. Even for a large language model such as the small GPT-2, 117M parameters, over 50K subwords, and 100 symbols sentences, AriEL results in high quality samples for a wide range of latent dimensions (SM 11). Floating point precision does not seem to be of concern up to this limit.
Evidence for volume codes. The results suggest that AriEL volumes are responsible of its success. We provided evidence on how volume codes improve the retrieval of information composed of discrete, variable length sequences, by random sampling, compared to other codes. AriEL generates more valid sentences, an explicit volume code. VAE is the second on the toy dataset, an implicit volume code, but it is worse than AE on the real dataset.
Transformers are hard to sample from the latent space. The Transformer has been used in this work in an uncommon way: by sampling its latent space instead of its output space. Its low validity score in this work reflects it. Our aim was to understand the latent organization of language, so, we do not suggest this is the most effective way to use Transformer. Transformer is excellent when sampled in the output space, but it's difficult to sample from the latent space. This is because Transformer represents each word in d dimensions while the other approaches represent a sentence in d dimensions. Transformer needs very high dimensional vector to represent sentence, d, where is the number of words in it. This makes it hard to find sentences by uniform sampling the latent space.
6 CONCLUSIONS
We proposed AriEL, a mapping of language into volumes, that we used as reference system for language generation. AriEL fuses arithmetic coding and k-d trees to construct volumes that preserve the statistics a dataset, On the one hand, this study helps to realize how much of the latent space lies unused by standard architectures. On the other hand, when compared to standard techniques it highlights room for improvement in their capacity for generation, prediction and generalization. AriEL allows to in theory the training set probability distribution and in practice a diverse set of sentences, as demonstrated on the toy, on the human dataset. and using GPT-2 as its language model.
9

---

Under review as a conference paper at ICLR 2022
REFERENCES
Abadi, M., Agarwal, A., Barham, P., Brevdo, E.,, Chen, Z., Citro, C., Corrado, G. S., Davis, A., Dean, J., Devin, M., Ghemawat, S., Goodfellow, I., Harp, A., Irving, G., Isard, M., Jia, Y., Jozefowicz, R., Kaiser, L., Kudlur, M., Levenberg, J., Mané, D., Monga, R., Moore, S., Murray, D., Olah, C., Schuster, M., Shlens, J., Steiner, B., Sutskever, I,, Talwar, K., Tucker, P., Vanhoucke, V., Vasudevan, V., Viégas, F., Vinyals, O., Warden, P., Wattenberg, M., Wicke, M., Yu, Y. and Zheng, X. (2015). "TensorFlowe Large-scale machine learning on heterogeneous systems'. Software available from tensorflow.co .r.g..
Bengio, Y., Courville, A. and Vincent, P. (2013), 'Representation learning: A review and new perspectives', IEEE transactions on pattern analysis and machine intelligence 35()), 1798 1828.
Bentley, J. L. (1975), 'Multidimensional binary search trees used for associative searching', Communications of the ACM 18(9), 017-
Bowman, S. R., Vilnis, L.,, Vinyals, O., Dai, A., Jozefowiez, R. and Bengio, S. (2016), Generating sentences from continuous space, in '20th SIGNLL CoNLL', Association for Computational Linguisties, pp. 10 21.
Chen, T. Q. Rubanova, Y., Bettencourt, J. and Duvenaud, D. K. (2018), Neural ordinary differential equations, 'NeurIP'', pp. 6571-6583.
Cho, K., van Merrienboer, B., Gulcchre, C., Bahdanau, D., Bougares, F., Schwenk, H. and Bengio, Y. (2014), Learning phrase represenlations using rnn encoder decoder for stalistical machine translation, in 'EMNLP', Association for Computational Linguisties, pp. 1724-17 1734.
Dai, Z., Yang, Z., Yang, Y., Carbonell, J., Le, Q. and Salakhutdinov, R. (2019), Transformer- XL: Attentive language models beyond a fixed-length context, in 'Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics' Association for Computational Linguisties, Florence, Italy, pp. 2978 2988.
De Vries, H., Strub, F., Chandar, S., Pietquin, O., Larochelle, H. and Courville, A. C. (2017), Guesswhat?! visual object discovery through multi-modal dialogue., in 'IEEE CPVR', Vol. 1.
Edelsbrunner, H., Kirkpatrick, D. and Scidel, R. (1983), 'On the shape of set of points in the planc', IEEE Transactions on Information Theory 29(4). 551-559.
Flias, and Abramson, N. (1963), Information Theory and Coding, Flectronic Science, 1st edn, McGraw-Hil Inc., US, pp. 72 89.
Friedman, J. H., Bentley, J. L. and Finkel, R. A. (1977), 'An algorithm for finding best matches in logarithmic expected time', ACM TOMS 33),, 209-226.
Glorot, X. and Bengio, Y. (2010), Understanding the difficulty of training deep feodforward neural networks, in JMMLR W&CP: 13th AISTATS', Vol. 9, pp. 249 256.
Goodfellow, I., Pouget-Abaddis J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A. and Bengio, Y. (2014), Generative adversarial nets, in Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence and K. Q. Weinberger, ods, 'NeurIPS 27', Curran Associates, Inc., pp. 2672-2680.
He, K., Zhang, X., Ren, S. and Sun, J. (2015). Delving deep into rectifiers: Surpassing human- performance on imagenct classification, in PProccddings of the IEEE interna- tional conference on computer vision', pp. 1026 1034.
Hinton, G. E., McClelland, J. L. and Rumelhart, D. E. (1984), Distributed representations, Carnegie- Mellon University Pittsburgh, PA.
Hochreiter, S. and Schmidhuber, (1997), 'Long short-term memory', Neural computation 9(8), 1785-1780.
10

---

Under review as a conference paper at ICLR 2022
loffe, S. and Szegedy, C. (2015), Batch normalization: Accelerating deep network training by reducing internal covariate shift, in 'International conference on machine learning', PMLR, pp. 448-456.
Jebara, T. (2012), Machine learning: discriminative and generative, Vol. 755, Springer Science & Business Media.
Jiang, W., Kiang, S., Hakim, N. and Meadows, H. E. (1993), Lossless compression for medi- cal imaging systems using linear amncarnoonlinal prediction and arithmetic coding, in 'ISCAS', IEEE, pp. 286.
Johnson, N. W. (2018), Geometries and transformations, Cambridge University Press.
Kingma, D. P. and Ba, J. (2015), Adam: A method for stochastic optimization, in 'ICLR'. Kingma, D. P. and Welling, M. (2014), Aulo-encoding variational bayes., in 'ICLR'.
Kusner, M. J. and Hornández-Lobato, J. M. (2016), 'Gans for sequences of discrete elements with the gumbel-softmax distribution', arXiv preprint urXi::611.00055.
LeCun, Y., Boser, B., Denker, J. S., Henderson, D., Howard, R. E., Hubbard, W. and Jackel, L. D. (1989), 'Backpropagation applied to handwritten zip code recognition', Neural com- putation (4), 541 551.
Li, R., Peng, X., Lin, C., Rong, W. and Chen, Z. (2021), On the low-density latent regions of vac-based language models, in 'NeurIPS 2020 Workshop on Pro-rogistration in Machine Learning', PMLR, pp. 343 357.
Miller, G. A. (1995), 'Wordnet: A lexical database for english', Commun ACM 38(1), 39- 41.
Munkres, J. R. (2018), Flements of algebraic topology, CRC Press.
Ng, A. Y. and Jordan, M. I. (2002), On discriminative vs. generative classifiers: A compar- ison of logistic regression and naive bayes, in 'NeurIPS', pp. 841-848.
Pasero, E. and Montuori, A. (2003), Neural network based arithmetic coding for real-time audio transmission on the tms320c6000 dsp platform, in 'ICASSP', Vol. 2, IEEE, pp. II- 761.
Perez, E., Strub, F., de Vries, H., Dumoulin, V. and Courville, A. C. (2018), Film: Visual reasoning with a general conditioning layer, in 'AAAI'
Radford, A., Narasimhan, K., Salimans, T. and Sutskever, 1. (2018), 'Improving language understanding by gencrative pre-training', Technical report, OpenAI.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, 1. (2019), 'Language models are unsupervised multitask learners', OpenAl Blog 1(8), 9.
Saxe, A. M., McClelland, L. and Ganguli, S. (2014), Exact solutions the nonlinear dynamics of learning in deep linear neural networks, in 'ICLR'
Scialom, T., Dray, P--.,, Lamprier, S., Piwowarski, B. and Staiano, J. (2020), Coldgans: Taming language gans with cautious sampling strategies, in H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan and H. Lin, eds, 'Advances in Neural Information Processing Systems', Vol. 33, Curran Associates, Inc., pp. 18978-18989.
URL: https://procceeeee neurips.. cc/ heppppr///aae//peeeeeee /200 http:///088BB8.8..00000000000000 Paper.pdf
Sutskever, I., Vinyals, O. and Le. Q. V. (2014), Sequence to sequence learning with neural networks, 'N'ur MIT Press, Cambridge, MA, USA, pp. 3104-3112.
Tatwawadi, K. (2018), 'Deepzip: Lossless compression using recurrent networks', URL https://web. stanford. http//uas//s222////or//////00000 pdf.
11

---

Under review as a conference paper at ICLR 2022
Vasudevan, S., Ramos, F,, Nettleton, E. and Durrant-hhyte, Whyte, H. (2009), 'Gaussian process modeling of large-scale terrain', Journal of Field Robotics 26(10), 812-840.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. and Polosukhin, I. (2017), Attention is all you need, in 'NeurPSS', pp. 5998 6008.
Wiedemann, S., Kirchhoffer, H., Matlage, S., Haase, P., Marban, A., Marine, T., Neu- mann, D., Osman, A., Marpe, D., Schwarz, H. et al. (2019), 'Deepcabac: Context- adaptive binary arithmetic coding for deep neural network compression', arXiv preprint 1905.08315.88
Williams, R. J. and Zipser, D. (1989), 'A learning algorithm for continually running fully recurrent ncural networks', Neural Computation 1(2), 270-280.
Witten, I. H., Neal, R. M. and Cleary, J. G. (1987), 'Arithmetic coding for data compression', Communications of the ACM 30(6), 520 540.
Wolf, T., Debut, L.,, Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowiez, M.. Davison, J., Shleifer, S., von Platen, P., Ma, C., Jernite, Y., Plu, J., Xu, C., Scao, T. L.., Gugger, S., Drame, M., Lhoest, Q. and Rush, A. M. (2020), Transformers: Statte-ffttheart natural language processing, in PProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations' Association for Computational Linguistics, Online, pp. 38-45.
Woodbridge, J., Anderson, H. S., Ahuja, A. and Grant, D. (2018), Detecting homoglyph attacks with a siamese neural network, in 'SPW", IEEE, pp. 22-28.
Yang, Z., Hu, Z., Salakhutdinov, R. and Berg- Kirkpatrick, T. (2017), Improved variational autoencoders for text modeling using dilated convolutions, in 'Proceedings of the 34th International Conference on Machine Learning- Volumee 70', JMLR. org, pp. 3881-3890.
Yin, H., Ding, X., Tang, L., Wang, Y. and Xiong, R. (2017), Efficient 3d lidar based loop closing using deep neural network, in 'ROBIO', IEEE, pp. 481-486.
Yu, L., Zhang, W., Wang, J. and Yu, Y. (2017), Seggan: Sequence generative adversarial nets with policy gradient, in Thirty-First AAAI Conference Artificial Intelligence'.
12

---

Under review as a conference paper at ICLR 2022
SUPPLEMENTARY MATERIAL
CONTEXT-PREE GRAMMAR (CFG) USED IN THE EXPERIMENTS
The context free grammar used to generate the biased and unbiased sentences is composed by the following rules:
gword adjective ' adjective 'and' adjective '8 qword adjective 'and" adjective 'y3
gword adjective "q"
qword 'made" "af" noan material '97
qword preposition up '
qword up 19"
np + determiner adjective adjective adjective noun determiner adjective "," adjective 'and" adjective + determiner adjective 'and" adjective noun 'made" 'of" materia
ap 4 determiner adjective adjective noun determiner adjective 'and adjective noun
ap + determiner adjective noen 'mede" 'of" nown.material determiner
'made" "af" natn.naterial ap + determiner adjective nomn ap determiner
sword + 'is" 'it" 'is objntt mateemalerial noun.roomtype preposition + preponition.mtteiall
'object"
"tee "thing
adjective adjective.color adjectioe.
adjective
adjective-relattee rel.true adjective adjective. reloteereeliieee..ee per- dimeneton.sise adjective mass adjective state adjective other
noun_object
'accordion 'acoustic" 'gramophone" 'bar 'berrier' 'seating" ...
basket
'outdoor" 'lamp" 'outdoor
nown_meterial bbrikks grass
'corpet' "decoration "stone 'ground" 'laminate" 'leather 'wood" ...
"facing" 'stone"
nounrroomtype "eeaation 'balcong" '
"bathroom" 'bedroom" 'bosler' 'room "guest 'room' "halt "hallway" 'kitchen
gerage
determiner 'en' "that" | 'the ? "this
preposition matersal 'made" ! 'af'
adjective.coior
'antique' 'white 'magento" 'maroon" 'slate "gray 'white 'yellow
adjectice.afoordace
'actable : 'destrogable divudabbee 'movable
'addable 'addressable" 'deliverable
adjective.size adjectine.avcrall.s.
adjectioe-overral.ee overall 'vverage sized adjective.relatiee. per dimension size
'huge 'large 'small' 'tiny" deep "narrow" 'shallow ssortt 'tall" wide
adjective mass
"keavy 'light "llighlly
'moderately' 'heavy" 'moderately" Mlight "very" "heavy "wery "light"
'heavy adjective 'closed" 'opened"
adjectine other 'teatured "transparent"

---

Under review as a conference paper at ICLR 2022
2 SIZE OF THE CFG LANGUAGE SPACE
From the CFG used in the experiment, it is possible to extract a total of 15,396 distinct grammar rules, some shown below. However, for simplicity, we defined only 4, related to the number of adjectives in it. In the case of the unbiased datasct, those rules can produce total of 9.81e+ unique sentences. The total number of unique sentences for the biased datas dataset is expected to be an order of magnitude smaller.
sgoord
prep.matericl, determiner, adjastate, 'and', adj.other, noun.roomtype, '91
qwoord prep_spatial, determiner, adj-other, adj_state, adj_state, noun_object, '91]
qword
determiner, adj_other, ", adj_mass, 'and', adh. noun.roomtype, 19
qwoodd
determiner, rdj-relative-per dimension size, adj-overall size, noun object
guoord
determiner, otgeollallssss
adj 'and" adj-state, cown.materidl 2
prep_spatial, determiner, adj-other, adj_mass, udj.effordence. noun.materiat. .qword.
[qword. adj-state, 'and" odj.relatsve.nize '9']
[qoord:
prep.material, determiner, adj.mass, adj.other, adj.other, noen.material, 1911 'qword:
prep_spatial, determiner, adj_state, adj_other, adj-color, noun_object, 19
[qword. determiner, rel 'and". adj-overall.
noun-material 191
[qword. determimer, adj state adj overall size, adj other , noun roomlype, '91 [qword, determiner, adj adj-state adj_mess, !qword. determiner, edj.overall.sise [quord. determiner, adj.color, adj-other, noun.object, 191]
matemmalrial, '97]
'and'. adj-other nosm.material, '97
sgoord; prep.spatial.rel, determiner, adj.mass, adj.calor. nown '87 swordd determiner, adi_state, 'and', adj. noun_object, '9 gwordd determiner, adj-color, adj-color, adj relative..e noun.materiol, '
gword. determiner, adj affoafdordance, noun ooject, '91/ qwordd determiner, adj-other, adj-other, adj-state, nown.roomiype, "
3 EXAMPLE OF SENTENCES GENERATED FROM THE CFG
3.1 BIASED SAMPLE SENTENCES
is it large, light yellow and light ?
is it white, deep pink and average-sized ?
is it light, huge and shallow laminate ?
is the object average-sized and light.
is the object fashionable, ghost white and pale turquoise is the thing huge, huge and khaki ?
is the thing small, ignitable and very light ?
is the object notable very light orange carpet ?
is the object this small wood made of facing stone
is the object a textured and combinable floor cover made of laminate
3.2
UNBIASED SAMPLE SENTENCES
is the object the huge tiny lovable guest room
is the object the closed closed transparent. textile ?
is the thing a transparent, narrow and slightly heavy lextile
is it steorable, dark orange and light ?
is it gray, very heavy and textured is it closed, heavy and moderately light
is it transparent, transformable and moderately light is the thing averago-sized and dark red
is the thing large and deep garage ?
is it that slightly heavy stucco made of grass ?
2

---

Under review as a conference paper at ICLR 2022
4 CFG VOCABULARY
Annotation
Nb. of classes
Example of classes
Noun
86
air conditioner, mirror, window, door, piano
WordNet category (Miller, 1995)
580
instrument, living thing, furniture, decoration
Location
24
kitchen, bedroom, bathroom, office, hallway, garage red, royal blue, dark gray, sea shell transparent, textured
Color
139
Color property
Material
15
wood, textile, leather, carpet, decoration stone light, modcrately light, heavy, very heavy
Overall mass
Overall size
tiny, small, large, huge
ralegory-relative size
10 liny, small, large, huge, short, shallow, narrow, wide
State
opened, closed
Acoustical capability
sound, speech, music attach, bend, divide, play, shake, stretch, wear
Affordance
100
Table S1: Description of vocabulary used.
5 USE OF LATENT SPACE
In figure S1, each dot represents sentence in the latent space. the first row the dot in the latent space is passed as inpul. to the decoder, while in the second and third row the dol is the output the encoder when the biased test sentence fed at its input. Two random axis in Hd are chosen for the generator, first row, while two axis were chosen subjectively among the first components of a PCA for the encoder, second and third row. In every case, the values in the latent space where normalized between zero and one to case the visualization. Lines are used to ease the visualization of the clusters of data with their label, since the point clouds overlap and are hard to see. The curves are constructed as concave hulls of the dots based on their Delaunay triangulation, method called alpha shapes Edelsbrunner et al. (1983).
We can see in figure S1 (first row) how easy it is to find grammatical sentences when randomly sampling the latent space for each model. AriEL practically only generates gram- matical sentences and AE and VAE perform reasonably well too, while Transformer fails. AriEL failures are plot on top, to remark how few they are, while AE, VAE and Transformer failures are plot at the bottom, otherwise they would hide the rest given how numerous they are. the same figure (rows two and three) we can observe how different methods structure the input in the latent space, each with prototypical clusters. The Transformer presents interesting structure clusters whose purpose remains unclear. Interestingly, the encoding maps seem to be more organized than the decoding ones. All the models seem to cluster data belonging to different classes at the encoding, that could be taken advantage of by a learning agent placed in the latent space. However it seems hard to the Transformer as a generator module for an agent. The good performance of AriEL is a consequence of the fact that all the latent space is utilized, and in no directions large gaps can be observed. This can be seen in the two encoding rows, where the white spaces around the cloud of dots are consequence of the rotation performed by the PCA, otherwise all the space between and would be utilized by AriEL.
3

---

Under review as a conference paper at ICLR 2022
AriEL
Transformer
AE
VAE
9
10
10
10
1
ungrammatical
length percentile 75-100
0 adjectives
length percentile 50-75
1 adjective
length percentile 25-50
2 adjectives
length percentile 0-25
3 adjectives
Figure S1: Random--ammpling-aases generation in the first row, and encoding of input sentences in the remaining rows. A sentence is represented by a point in the latent space. First row shows the proportion of grammatically correct sentences that can be decoded by random uniform sampling the latent space. AriEL sampled almost only grammatical sentences (ungrammatical are so few that are placed on top in the plot). Transformer mainly yielded ungrammatical sentences, while AE and VAE were able to produce many grammatical sentences (ungrammatical are below, otherwise they would cover up the grammatical).. Each dot is labeled according to how many adjectives the sentence generated has. Second and third rows show the clusters points in the latent space for the test sentences they are mapped by the encoders. All models seem to shift the clusters to some degree according to the number of adjectives in the sentence, in the second row. A similar conclusion applies the third row, that shows where sentences of different length are encoded. For all panels, we searched subjectively for the dimensions that would better reveal some clustering, with the help of PCA. We scaled all latent representations between [0,1] for visualization.
4

---

Under review as a conference paper at ICLR 2022
VISUALIZATION OF PERFORMANCE ON TOY DATA
R" 16
R112
Pandfomer VAZ
Genoratioo
BAA
GC
0c VU GAIAAU
GA
Preiction
PAU
PAU
YU GA HAU
PAR
Figure S2: Radar Chart of the Quantitative Assessment. Latent space of R on the left and R512 on the right. Training was performed on biased sentences. The metrics are defined in Methodology: Gencralization is measured by prediction accuracy of unbiased sentences (PAU), Prediction by prediction accuracy of biased sentences (PAB), grammar accuracy (GA) and bias accuracy (BA) and Generation by uniqueness (U), validity (V), vocabulary coverage (VC) and grammar coverage (GC). AriEL excels all the metrics. Most importantly AriEL outperforms every other method in Generation Validity (V) and it doesn't 't require a large latent space to do so (R similar to R512). VAE performs remarkably well at generating unique and grammatical sentences (validity, V) when the latent space is small (R16), probably given the volume-code nature the method. Transformer performs exceptionally not overfitting in the reconstruction tasks and generalizing, it manages to cover all grammar rules, even with very small number of parameters (R11). Transformer proved to be an inefficient generator using random sampling input (validity) but improved with larger latent space. For a larger latent space of RR522 AE and VAE overfit less (PAU and PAB) and improve their Generation (V).
TRAINING DETAILS
We go through the training data times, mini-batches of 256 sentences. We applied teacher forcing (Williams and Zipser, 1989) during training. We use the Adam (Kingma and Ba, 2015) optimizer with a learning rate of 1e-3 and gradient clipping at 0.5 magni- tude. Learning rate was reduced by factor of 0.2 if the loss function didn't decrease within epochs, with a minimum learning rate of 1e-5. For all RNN-based embeddings, kernel weights used the Xavier uniform initialization (Glorot and Bengio, 2010), while recurrent weights used random orthogonal matrix initialization (Saxe et al., 2014). Biases are initial- ized to zero. Embeddings layers are initialized with uniform distribution between [-, 1]. For Transformer the multihead attention matrices and the feedforward module matrices, used the Xavier uniform initialization (Glorot and Bengio, 2010), the beta of the layer nor- malization uses zeros, and its gamma uses ones for initialization. AE and VAE are trained with a word dropout of 0.25 at the input, and VAE trained with KT. loss annealing that moves the weight of the KT. loss from zero to one during the 7th epoch, similarly to the original work (Bowman et al., 2016). Our code is in TensorFlow (Abadi et al., 2015). We run our experiments took less than one day to converge.
NVIDIA TITAN Xp and an NVIDIA Tesla K40c. Each experiment
ARIEL IS AN EXPLICIT VOLUME CODE PROOF
AriEL uses a bounded region Rdd, the interval [0, 1]d so encoder and decoder map each input to and from compact set. Moreover, any sequence is assigned to reperrre (Johnson, 2018) and back. Since hyper-reetangles cannot be divided into two disjoint non- empty closed sets, they connected (Munkres, 2018). Therefore AriEL is volume code.
5

---

Under review as a conference paper at ICLR 2022
AriEL is an explicit volume code since its LM is trained only next word prediction log- likelihood loss, without a regularization term that encourages smoothness, and the volumes are constructed by arranging the softmax outputs into a d dimensional grid, operation performed with any choice of loss or optimizer.
MORE SENTENCES FROM THE QUALITATIVE STUDY
Input Sentences
is the thing this linen carpet made of tile is hoge and toal ?
is the thing transparent huge and slightly heavy ?
is the object antique white, tiny and closed
AriEL
AriEL
is the ohject that tiny very light set is the thing tiny destroyable abstraction thing this mint cream textured organic structure ?
carpet made tile?? is it huge and teachable ?
is thing
is the thing transparent huge and slightly heavy? ?
is che abject this small large wearable textile ?
is the object antique while, Liny and closed
Transformer
Transformer
is the thing elightly heavy heavy stone squeezable closed heavy
is the thing this stretehable carpet made tile
is it huge and magenta ?
is pale lime exocutable excentable shallow decoration drah tuzquoise, heavy and potang
is thing transparent, hugo and slightly heavy? ?
is the object antique white, tiny and closed
is an tomato slot box made of decoration facing stone? ?
AE
is the thing short and spring hewv slightly heavy potang
is the thing this small toilet, made of lamimate
is this averago-sized and averago-sized laminate
is the object that light light laminate ?
is the thing very heavy, heavy and very heavy?
the thing light small and small laminate ?
is the object light pink, small and textured ?
is the thing that thny small desoration stone" is the object the average-sized texctured and average-sized laminate VAE thing a light and deep office ? is it light light and light and pink?
VAE
is the thing small and textured deep stone
is light deep bedroom ?
is the thing textured, textured and moderately heavy? is the thing light, moderately heavy and light green ?
the object dark, light piuk
is the object a light deep living room
Table S2: Generalization: next word prediction of unbiased sentences at test time. An unbiased sentence is encoded and decoded by each model. Color means that the word was incorrectly reconstructed. Blue mcans that the sentence complies with the bias and purple means that the incor- rect reconstruction reconstructions seem grammatically correct. In practice AriEL also made errors. Some of its failed reconstructions comply with the training bias, some do not. Transformer per- forms remarkably well, and interestingly the errors made tend to turn the unbiasod input sentence into a biased version at the output. AE produced only biased sentences whose structure resembled the unbiased ones. VAE behaved similarly, producing more unbiased sentences.
Table S3: Generation: output of the decoder when sampled uniformly in the latent space. Red defines grammat- ically incorrect generations according to the CFG the models are trained on. AriEL pro- duces an extremely varied set of grammati- cally correct sentences, most of which keep the bias of the training set. Transformer re- veals itself to be hard to control via random sampling of the latent space, since it almost never produces correct sentences with this method. AE and VAE manage to produce several different sentences, the latter pro- ducing more non grammatical, but well more varied grammatical ones.
still unbiased. Most

---

Under review as a conference paper at ICLR 2022
10
find OPERATION
The find operation defines within which bounds the latent vector is pointing at, and therefore what is the next word selected by AriEL. This operation is performed every time axis d; selected. We define with an argmax operation at the end, but other solutions can be defined. To give it pseudonim we call it DetRouletle, from Deterministic Roulette.
Algorithm 2 find operation
Input: bounds on axis d;: Bstow (s), Bs (s): value of latent space on d; z(d:) Output: 9, is the word z was pointing at,
1: function FIND, (BSiow < z(d)) < (s))
2: a(s) B 8) x(d,)
3: b(s) Bstow (8) 2(d))
4: logits(s) = (s)) b(s)
5: 8, argmax, (logits(s))
6: return
7: end function
S
2
S3
S4
S5
z(d))
Figure S3: The latent space does not necessarily point the most likely word. Here the fiidd function would return word 84, the least likely. The bounds define the limits of the different blue areas.

---

Under review as a conference paper at ICLR 2022
11 SAMPLES FROM ARIEL WITH CPT-2 AS LANGUAGE MODEL
d 1010
He must), 've been very clear that the missive was a sign of trouble since the MTA was lukewarm on keeping maintenance intact. "It only takes a few seconds for you to walk through the lines and see the notice," Kris Johnson, who worked on the 14th Street Bus line in January, said of the notice. "But we don) 't hire people that don) 't want to work long as they work." work."\nnyuu AnlnHoww many times have you seen Why $15 is $15
Dear Reader, As you can imagine, more people are reading The Jerusalem Post than ever beforc. Neverthcless, traditional business models are longer sus- tainable and high-quality publications, like ours, are being forced to look for new ways to keep going. Unlike many other news organizations, we have not put up paywall. We want to keep our journalism open and accessible and be able keep providing you with news and analyses from the frontlines of Israel, the Middle East and the Jewish World.
d 50
Melrose Creek is slated for redevelopment in where hundreds of 150-odd resi- dents live in the effort to develop a pedestrian-friendlyy and quieter downtown in a neighborhood that has been a struggling housing market for their entire lives. /n AnAter decades of straddling one of the most culturally open places in San Francisco, Melrose Creek's market-driven nature has resulted in popular zoning problems problens-includingg iinlluding a seismic event lifting off of lent park and opulent vistas sunbathed against the bicycle-friendly
Light Air Rating/ BBearing Temperature. Can Handle Winds from Down (1 to 15 ft/min. 30 It should be able to raise to about 53 mph, 35 mph 24% duty on stiff and T-shielded loads). Can Handle Medium Dry conditions and Good Condition's n n nHybrid Extra Performance." Two Driven In Woofer Lighting Stations! 1967-1992 \/ nAlso applicable to the best gasket material available. No Gasket Limit forlyst glass. (n)n)n
d = 100
In the last three years, items were being traded around the world for less in bitcoin. Reuters estimates that its stock price dropped 19%. (noonnnnyyeseedd China has kind of happened. On Yahoo Finance, Rory Schrimpf wrote A"A good amount of Chinese billionaires are planning formal plans for a market of bitcoin for a fund," his Twitter feed says). The gist itself looks like this:)n)nn " "bitcoin meeting" that can potentially all but guarantee big deals in bitcoin might come down to"
Even though the story wants to be understood, many people who have expe- rienced hardfought battles of adversity are still going to think that what they experienced in the Battle of Telluride will help define their narratives in a way which will bring this story to the public's attention and invite new dialogue and reconciliation, news organisations have widely concluded that dreams of revolu- tion come easy to ordinary people who make difficult choices. What we often forget is that urbanisation has devastating dams, fragmentation, and denial of services and employment
Table S4: AriEL samples using GPT-2 as Language Model. Even for large language model such the small GPT-2, which has 117M parameters, and over 50K subwords vocab- ulary, AriEL folding resulted in high quality samples for wide range of latent dimensions. Here the samples are 100 subword symbols long. This proves the float 32 representation constraint does not represent limitation at least up to this limit.
8

---

Under review as a conference paper at ICLR 2022
12
SAMPLES FROM ARIEL TRAINED ON GUESSWHAT?!
16
512
AriEL
it silver? ?
bowel in white colour ?
it the trash can look like food ?
is it the steel top in white is it in left does it the taller ?
Transformer
foreground am exactly exactly by left same cat dog by bed about bike base base us on electricity poster clothing a clothing clothing clothing clothing clothing clothing clothing clothing trees stoplight ground ground bird side bikes
is it she is ?
is it that directly ?
AE
a person the table ?
is it a left side ?
are they in the left side ?
in the ?
person
VAE
is the the person on a person a ? they in the left of the left side of the
of us
something it to white right side ? all a person blue right ?
all white blue of front ?
Table S5: AriEL samples training the Language Model on the real dataset Guess- What?!. Most samples created through AriEL seem easily interpretable if not fully gram- matically correct. Transformer appears to be very difficult to sample from the latent space, and it produces very poor language for d = 16, and more reasonable even though hardly interpretable for 512. AE and VAE seem to produce more reasonable sentences, still often hardly interpretable, and less diverse than AriEL.


Reviews:
[{"review_id": "u38uWlLLRk", "reviewer": "Reviewer_SHoR", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- A novel approach for latent modelling\n- Strong results in terms of validity\n\n- Weakness\nIn Table 1, the validity percentage of a Transformer with 512 latent dimensionality is only 17.2%. This low score strongly contrasts with our knowledge that a well-trained Transformer language model is very strong at producing valid sentences. One hypothesis is that the amount of training data is not sufficient for Transformer. If this is the case, the proposed method may lose its edge when the training data is abundantly available.\n\nIn additional to this point, if the main reason for the Transformer to fail in validity is because the vanilla Transformer was not trained on sampled latent space. We shall at least inject random noise during training to make the Transformer robust to sampling.", "summary_of_the_paper": "This paper describes a novel methodology of volume coding for encoding and decoding sentences. The algorithm is based on arithmetic coding. The qualitative result reveals that the generations from AriEL to be more valid comparing to other models.\n\n\n\n\n", "summary_of_the_review": "I have reviewed this paper last year, I can see the authors have reorganized the content significantly. I will give a weak acceptance this time and willing to discuss among reviewers.  ​", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "fM3EtDhItI", "reviewer": "Reviewer_JWsh", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. I feel the paper is poorly-organized. I repeat like 5-6 times to get what it's trying to do in introduction. This could immediately be a reason to reject the paper as I think it's not gonna benefit audience in ICLR.\n\n2. After digesting it, I think I like to problem this paper discussed, and it's very pertinent to the representation learning. However, I am not sure if I got 100% understanding of the content as it's again poorly presented. Leaving all the materials in an algorithm table without much explanations make it hard to understand. I think a running example should be provided to make reader easier to understand what's going on. More importantly, authors seem to assume readers know what AC is but honestly I don't see the necessity. \n\n3. From the start to end I am not sure why Volume Code is good. This isn't well motivated. And I honestly don't know how the metric is picked. I don't see these properties are necessarily good. If my VAE or other models are learning a representation toward a specific task then why it's necessary to follow these designs?\n\n4. I feel all the comparison models can be used as language models. So my understanding is that Ariel can be used to generate code provided an underlying LM. Why not train VAE or other stuff in LM task and then apply Ariel to see the results? \n\n5. Following 4, for this type of paper, an observation is provided and usually it should also propose a way to augment the existing methods to match the proposed metric. So it reads to me that authors should propose a way to show that representation learned by VAE can be improved by adding some components. And in my point 4, I do feel treating it as a LM can be tested. \n\n", "summary_of_the_paper": "The paper proposed a method to encode/decode a sentence (or say a sequence of tokens) into/from a volume instead of just a point(vector) in d-dimensional. The paper shows some good attributes of this method via some experiments. ", "summary_of_the_review": "Overall, I think it's not a presented in a straightforward way and it's lack of many details for readers to fully digest it. I feel the topic is interesting but in practice I don't know why it's important and author didn't motivate this. I will be considering to raise the score if the introduction is re-written in a motivating form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "7Z43vll8Qqd", "reviewer": "Reviewer_JLnX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1. The writing of this paper has much room for improvement. For example, the description of the method part in Sec 3.1 is super unclear. It's hard to fully understand the method without referring to the pseudo-code. Some of the key operations (e.g., \"find function\") are moved to the appendix. And the notations in the pseudo-code are also confusing, e.g., a word token is sometimes used with or without a subscript -- Inconsistent notations of \"(s_j)_{j<i}\" and \"s > s'\" seem to denote previous words before the i-th word.\n2. Conceptually it's still not clear why splitting the latent space using the LM (trained on biased text data) could be beneficial to the generalization for unbiased text. Even though the coding method could efficiently utilize its latent space to fill sentences in all its d dimensions, wouldn't it also cover the bias from the LM trained on biased text data?\n3. I don't see most of the evaluation metrics used in this paper in the literature. I am a bit skeptical whether these metrics are convincing enough to evaluate these encoding methods on more realistic, complex datasets in addition to the toy & narrow-domain datasets used in this paper.\n4. The other models in comparison seem to be artificially tuned in some ways which lead to pretty low performances. For example, using only 2-layers of encoder & decoder or a small dimension for Transformer, sampling the latent space of Transformer. It's not clear how the proposed method compared with stronger models. ", "summary_of_the_paper": "This paper introduces an entropic coding method that is able to compress a sentence into a latent space and perform text generation through a uniform sampling from the implicit latent space. Experiments are conducted on a toy dataset and a narrow domain text dataset to show the effectiveness of the proposed method w.r.t. AE, VAE, and Transformer models.\n", "summary_of_the_review": "I would suggest a rejection of this paper given the current draft. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "5jgAu1CEIw6", "reviewer": "Reviewer_qCdr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "My first concern is that the proposed method essentially is arithmetic coding in multiple dimensions---and the mental abstraction I have in mind is that it is closer to the ZIP file format than an autoencoder, focused very much on perfectly reproducing training data and essentially overfitting it.\n\nMy next biggest concern is that I don't think I quite understand the problem this paper is trying to solve in terms of downstream use. For example, an autoencoder's encoder is useful because it can be a way to unsupervisedly obtain continuous representations for discrete string data---those representations could've been tested for their informativeness, e.g., by trying to predict sentence properties or cluster from them. Likewise, an AE's decoder is useful for use as a pretrained generator from latent codes: given a generation task with little data, it may be easier to learn a model that learns to generate latent codes (which are then spelled out by the AE decoder) than one that tries to generate text end-to-end. These kinds of ideas and considerations are missing for me in this document that seems very focused on showing the overfitting and recoverability capabilities of the proposed AriEL. (None of these experiments should be hard to conduct as AriEL through returning centers can be made to \"look like\" an ordinary autoencoder, I just have a feeling that given its focus on overfitting the training data, the interpolations, while maintaining grammaticality, will not be semantically useful, leading to low performance in these downstream tasks---but I would love to be proven wrong!)\nIn the same vein, I do not understand why the particular real-world dataset was chosen as opposed to any other collection of strings or datasets that are used by and compared to on previous work.\n\nIf that wasn't clear enough, here's a different way to phrase the issue: the evaluations in this paper as far as I can see are only concerned with *unconditional* sample quality, i.e., what do samples from randomly drawn points look like. That is interesting to look at to get a sense of the space, yes, but is ultimately of no relevance to any practical application---because if unconditional samples were truly what we wanted, we would almost certainly be better off just sampling from the base LM that AriEL uses to begin with! Or does this base LM perform worse? If you do want to claim AriELs use in unconditional sampling then that really is the dead simple baseline to outperform in your metrics like validity.\n\nNext, I must admit I do not understand the \"biasing\" fully. Does it categorically rule out some combinations while leaving others untouched (as Section 3.3 paragraph 3 seems to imply, essentially leading to all strings having probability 0 or p where p is the same for all sentences that have nonzero probability) or is it a weighting device that essentially turns the CFG into a PCFG in which not all strings are equally likely and no string that is possible under the CFG ends up with probability 0 (which is what Figure 1 seems to show)?\n\nFinally, I have doubts about the inductive bias of using dimensions in this round-robin per-word fashion and would like to see whether effects related to that periodicity and subword tokenization appear. One possible experiment to run here to gauge the problem might be to look at where errors occur in data or whether any other patterns arise that pertain to position (early/late/periodical for low dimensionalities)\n\nOther issues or questions:\n- the abstract says \"We compare to\" which sounds like AriEL is an existing technique and not a new contribution of this paper\n- Section 1 line 6 says \"such type\"---what type?\n- Section 1 paragraph 3 says \"objective benchmark\"---what is objective about it and compared to what \"subjective\" other benchmarks?\n- Section 2 line 4 calls volume codes distributed representations but defines the latter as points in R^d rather than values, leading to a contradiction\n- Section 2 paragraph 2's citations strike me as old; I am not intimately familiar with advances on these methods since 2017, but would be surprised if there weren't any\n- Are the letters in A|B|AA|... in Figure 1 nonterminals or terminals? I suppose they are terminals for the explanation/picture to make sense, but then convention would have them be lower-case instead of upper-case :)\n- The last paragraph of section 3.1 mentions RNN-based langauge models, but AriEL should work with any *softmax-based* autoregressive LM as I understand it, including Transformer-based models like the GPT family\n- I also admittedly do not understand why there is no linear dependence on vocabulary size in AriEL---lines 9 and 10 in both sides of Algorithm 1 iterate over the vocabulary in a linear way, do they not? (Some of that could be asymptotically recovered with clever sampling, but I think this setting does not lend itself to such asymptotic optimizations, though that might still be good to mention.)\n- Unless I missed it, Algorithm 1 is also never referenced in the main text.\n- Line 1 on page 4 spells \"Transformer\" as \"Tranformer\"...\n- ...and it is unclear to me how it is a fixed-length representation unless you cheat by establishing a maximum length and truncating/padding (which you later say you do---this might be good to state upfront to avoid confusion)\n- instead of $d_{ff}$ try $d_{\\mathit{ff}}$ or something similar to get proper spacing between letters in subscripts\n- Testing only 2 or 20 layers is a *huge* difference and even a priori I would expect that both the most commonly chosen and the best-working number of layers is somewhere in the middle\n- Section 3.3 paragraph 4 describes \"unbiased\" sentences as those that go *against* the bias defined (whatever that actually is, see the major confusion above)---shouldn't we then call them something like \"opposingly-biased\"?\n- Generation/Deocding Quality is missing a metric that counts how many of the 10k are grammatical. Yes, if they are all the same sentence and that sentence happens to be correct that measure would give a 100% that looks more impressive than it is, but with that caveat I would rather *know* this number than now know it.\n- Relatedly, it took me embarassingly long to understand why validity is the \"most important of the metrics\" as section 4.1 puts it---may benefit from spelling out that this is indeed the principal metric right where it is defined :)\n- I unfortunately also have some quesy feelings about the uniqueness in metrics like validity. For example, if I draw 5 samples, chances are they will all be unique so I can get a high ratio score. Great! But if I draw 5 million samples, it is very unlikely that I will get out 5 million unique sentences, so I will get a low score here. Of course, as will other models, so comparisons are fair, but the fact that as the number of samples tends to infinity the scores tend to 0 strikes me as a sign that the metric definitions aren't fully thought through. If a specific suggestion helps: define validity not as a ratio but as a count of unique sentences and let the number of samples tend to infinity.\n- Prediction Quality metrics are ordered in that PAB < BA < GA... is that correct? Might be useful to specify that way :)\n- Section 3.4.2 talks about \"interpretability,\" a very loaded technical term---do you mean something like fluency or legibility? Similarly to the choice of \"biased\" poorly chosen terminology may hurt this paper more than the actual contribution deserves.\n- Section 3.4.3: 15 pairs feels like a low number, do estimates truly converge with this few samples? Put differently, is Figure 2 as smooth as it looks? Putting in the actual points may help show just how cleanly sampled that space is.\n- Speaking of Figure 2, first, what is the y axis? Validity as defined in the ratio way (see above for my concerns on that) or something new? And what is this lower bound 0.746 that is \"related to the language complexity\"? Can you spell out that currently somewhat nebulous relationship?\n- Section 4.2 paragraph 2: can you quantify \"almost all\"?", "summary_of_the_paper": "The paper proposes to use an autoregressive language model's hidden state to index multidimensional arithmetic coding representations of vocabularies and sentences. This way the latent space's geometry is directly connected to the words chosen at the cost of sensible inductive bias between similar sentences. Evaluations of unconditional samples on toy data and small-scale real data are made. While interpolations are presented, the proposed model and the autoencoders it compares to are only ever evaluated for fluency and grammaticality, but not for semantic relatedness.", "summary_of_the_review": "The paper suffers from a lack of problem that is demonstrably made headway on, i.e., where a proposed method improves over sensible baselines on sensible metrics, largely because data, baseline, and metrics are chosen and justified poorly in my opinion.\n\nThat is why I currently cannot see what anyone could get out of this paper as it is (other than remembering that arithmetic coding and k-d trees exist and possibly being tempted to think about volumes in latent spaces and the relation between overfitting and representational capacity a bit more, no thanks to the paper which does not ever talk about these issues itself).\n\nFinally, I should disclose: I have reviewed this paper before (but do not know its authors or anything beyond that) and some more fundamental concerns are unchanged from my last review, though the paper has improved noticeably in my mind (thank you for these improvements, it really has gotten better). My review is thus focused on what I believe is possible to still improve about this paper rather than trying to reiterate previous concerns---that is why I have little to say about the evaluations performed themselves.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an entropic coding approach for sentence embedding. 

Reviewers have spent good efforts in reviewing. They generally feel the problem is important/interesting, but also found it difficult to understand the paper. Thus, the authors are encouraged to thoroughly revise the paper according to the reviews provided, and another round of review is needed to better determine the merits of this paper.
Decision: Reject