Mathematically Modeling the Lexicon Entropy of Emergent Language
Authors: Brendon Boldt
David R Mortensen
Venue: NeurIPS 2022 Submitted (2022)
Keywords: emergent language
emergent communication
stochastic process
modeling

Abstract: We formulate a stochastic process, FiLex, as a mathematical model of lexicon entropy in deep learning-based emergent language systems. Defining a model mathematically allows it to generate clear predictions which can be directly and decisively tested. We empirically verify across four different environments that FiLex predicts the correct correlation between hyperparameters (training steps, lexicon size, learning rate, rollout buffer size, and Gumbel-Softmax temperature) and the emergent language's entropy in $20$ out of $20$ environment-hyperparameter combinations. Furthermore, our experiments reveal that different environments show diverse relationships between their hyperparameters and entropy which demonstrates the need for a model which can make well-defined predictions at a precise level of granularity.

Mathematically Modeling the Lexicon Entropy of Emergent Language
Anonymous Author(s)
Affiliation Address email
Abstract
We formulate stochastic process, FILEX, as a mathematical model of lexicon entropy in deep learning-bant braning-bassdd emergent language systems. Defining a model mathematically allows it to generate clear predictions which can be directly and decisively tested. We empirically verify across four different environments that FILEX predicts the correct correlation between hyperparameters (training steps, lexicon size, learning rate, rollout buffer size, and Gumbel-Softmax temperature) and the emergent language's 's entropy in 20 out of 20 environment-hye chyormmmttyyprrrrametterr combinations. Furthermore, our experiments reveal that different environments show diverse relationships between their hyperparameters and entropy which demonstrates the need for a model which can make well-defined predictions at precise level of granularity.
2
6
8
10
Introduction
13
The methods deep learning-bast -based emergent language provide a uniquely powerful way to study the nature of language and language change. In addressing these topics, some papers hypothesize 15 general principles describing emergent language. For example, Resnick et al. [2020] hypothesize a predictable relationship exists between compositionality and neural network capacity, and Kharitonov
17 et al.) 2020) hypothesize a general entropy minimization pressure deep learning-
emergent language. In many cases, these hypotheses derived from intuitions and stated in natural language; 19 this can lead to ambiguous interpretation, inadequate experiments, and ad hoc explanations. To this end, study a general principle of emergent language by proposing mathematical model which generates a testable hypothesis which can be directly evaluated through the empirical studies, akin to what find prototypically in natural science.
21
22
23 We formulate a stochastic process, FILEX, as a mathematical model lexicon entropy in deep learning-base based emergent language systems (ELS). We empirically verify across four different environments that FILEX predicts the correct correlation between hyperparameters (training steps, lexicon size, learning rate, rollout buffer size, and Humbel-Softmaal temperature) and the emergent language's 's entropy in 20 out of 20 sairoommnn-hhppepprrmmeee combinations.
25
27
28 There are three primary reasons for using explicitly defined model for studying a topic like
29 emergent language: clarity, testability, and extensibility. A mathematical model yields clear, unambiguous interpretation since its components have precise meanings; this is especially important when conveying such concepts in writing. is easier to test a model than a hypothesis articulated in natural language because the model yields clear predictions which can be shown to be accurate
31
33 or inaccurate; as a result, models can also be directly compared to one another. Our experiments
Emergent language system or ELS refers to the combination of agents (neural networks), the environment, and the training procedure used as part of emergent language experiment.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

34 reveal that different environments show diverse relationships between their hyperparameters and
35 entropy which demonstrates the need for such clarity in making well-defined predictions at a precise
36 level of granularity. Finally, mathematical models hypothesize mechanism for an observed effect and not simply the effect itself (with a possibly ad hoc explanation). This what facilitates their extensibility since multitude of hypotheses can be derived from these mechanisms; furthermore, this "mechanical" nature allows future work to build directly on top of the model.
37
38
39
40
As mathematical models are seldom used to their full potential in studying emergent language, this paper meant to serve as a reference and starting point for entire methodology of developing and testing such models. We articulate our contributions as follows:
41
42
43
Defining a mathematical model of lexicon entropy in emergent language systems which we demonstrate to be accurate in predicting hyperparameter-entroyy entropy correlations.
44
45
Presenting case study of defining and empirically evaluating mathematical model in emergent language.
46
Provide direct, intuitive comparison of the effects of hyperparameters on lexicon entropy across different environments.
48
49 We briefly discuss related work in Section 2 In Section 3
we introduce the mathematical model, FILEX, as well as the ELSs. Empirical evaluation is presented in Section 4 and discussed in
50
51 Section [ concluding with Section Code is available at ltttps://exampla: cww com/reponame (in
52
supplemental material while under review).
53
2
Related work
54
For a survey of deep learning-basee emergent language work, please see Lazaridou and Baroni [020)] Contemporary deep learning- based emergent language research often aims at establishing and refining general principles about emergent language. In large part, these principles can be expressed as relationships between certain characteristics of the environment agents (e-g., model capacity Resnick et af. 2020), population size [Rita et al. [002]) and properties of the emergent language
56
57
58
59
(e.g., compositionality [Resnick et al.. [2020] Rodríguez Luna al. 2020), entropy [Kharitonov et al.,
60
2020 Chaabouni et 20.1 Rita et al.] [2022], and generalizability [Chaabouni et al. |020]]
61
al. 202 Slowik et al. 2020).. Some of these works [Kharioonvv et al. [2020] Khomtchouk and Sudhakaran 2018 Resnick et al. [020] make use of mathematical models to describe parts of the
62
63
hypotheses and/or experiments, but these fall short of establishing clear model which generates
64
testable hypothesis which is then evaluated through the empirical studies.
Pre-deep learning emergent language research frequently relied mathematical models [Skyrms 2010 Kirby et al. 2013, Brighton et al.) 2005], but such models played a different role. Whereas these models were meant account for some property of language observed in human language, the model presented in this paper is accounting for emergent language directly (and human language only indirectly). Thus, this paper presents (mathematical) model of a (computational) model which, in the future, will be used to more directly study human language.
66
67
68
70
3 Methods
71
72
3.1
Model
73
FILEX ("fixed lexicon stochastic process") is a mathematical model developed from the Chinese restaurant process [Blii] 2007 Aldous 1985] stochastic process where each element in the sequence is stochastic distribution over the positive integers (i.e., a distribution over distributions). The analogy for the Chinese restaurant process is restaurant with tables indexed by the natural numbers; as each customer walks in, they sit at a random table with probability proportional to the number of people already at that table. The key property here is that the process is self-reinforcing: tables with many people are likely to get even more. By analogy to language, the more word is used the more
74
75
76
77
78
80 likely it is to continue to used. For example, speakers may develop a cognitive preference for it, or
81 it gets passed along to subsequent generations higher rate [Francis et al. 2021

---

Algorithm FILEX pseudocode
alpha: float > 0
beta: int
N: int > 0
S: int > 0
weighta array (size=5) (sreees weights.fill for in range (N):
S)
W += saaple..ulliooial(( 10 N_copy weights copy() (opy
sum (W), beta) / beta
for in range rbetaa) # equivalent normalized multinomial
12
i sample_categorical (u_cooy sum(v_copy)
13
weights[1] += alpha / beta
14
return weights / sum sumightts))
82 Formulation FILEX defined as a sequence stochastic vectors indexed by E N given by:
w((N)
FILEX(a,8,5,N) = |
w (n+11) (n) + an(m)
(2)
B
(m)))
x (n) Multi B,
(3)
1 5 -1,,
w (1)
where (n)) is vector of weights, E R5o controls the weight update magnitude, 3 E N+ controls the variance of the updates, S N is the size of the weight vector (i.c., lexicon), and Multi(k, p) is k-trial multinomial distribution with probabilities p R The pseudocode describing FILEX is given in Algorithm[]
84
86
[ Conceptually, the process starts with an S-element array of weights initialized to 1/S. At each iteration draw from a ß-trial multinomial distribution parameterized by the 88 normalized weights This multinomial sample is multiplied by a/ and added to the weights so that the update magnitude is This proceeds N times. Since the sequence elements the normalized weights, the elements are themselves probability distributions; thus, FILEX is technically a sequence
90
91 of distributions over distributions.
92 The two key differences between FILEX and the Chinese restaurant process are hyperparameters S and a FILEX has fixed number of parameters
to match the fact that the agents in the ELS 94 have a fixed-size bottleneck layer, that is, a fixed lexicon. Secondly, 3 is introduced to modulate the smoothness of parameter updates. It is closely connected the fact that certain RL algorithms 96 like PPO accumulate a buffer data points from the environment with the same parameters before performing gradient descent.
98
3.2
Environments
99
To evaluate, we use four different reinforcement learning environments in our experiments. These are inhabited by two deep learning- -based agents: (1) sender agent which receives an observation and produces a message and (2) a receiver agent which receives a message (and possibly additional observation) and takes an action. The agent architecture and optimization are detailed Section 3.3
100
102
NODYN The "no dynamics" environment is proof-of-concept environment which is not intended to be realistic but rather to match as closely as possible the simplifying assumptions which FILEX
104
2The 8-trial multinomial sample is written i.i.d. samples from categorical distribution draw parallels PPO Algorithm
Note that in FILEX is actually equivalent the inverse of in the Chinese restaurant process.

---

105
makes while keeping the same neural architecture in the environments below. As the name suggests, the primary simplification in this environment that there are trivial dynamics, that is, every episode immediately ends with reward of 1 no matter what the sender or receiver do. The sender input and receiver output are identical to those of NAV, defined below. Just as FIL.EX assumes that every instance of word use is reinforced, this process reinforces every message which the sender produces.
106
107
108
109
110
The reconstruction game [Chaabouni et al. 2020], in the general case, mimics a discrete autoencoder: the input value is translated into a discrete message by the sender, and the receiver tries to output the original input based on the message. For a given episode, the sender observes 1)) and produces message; the receiver's action is a real number z; yielding a reward (x - 2)²
RECON
112
113
114
115
The signaling game environment comes from Lewis [9770] and has been frequently used in the literature [Lazaridou et al.) 2017 Bouchacourt and Baroni 2018). In this setup, the data is partitioned into fixed number of discrete classes. The sender observes a datum from one of the classes and produces a message; the receivers observes this message, the sender's 's datum, and data points from other classes (i.e., "distractors"). The reward for the environment is 1 if the receiver correctly identifies the sender's 's datum among the distractors and 0 otherwise.
SIG
116
117
118
120
121 To eliminate the potential confounding factors from using natural inputs (e.g.. image embeddings I Lazaridou al. 2017). ). we use a synthetic dataset. For an n-dimensional signaling game, we have 2n classes. Each class is represented by an isotropic multivariate normal distribution with mean 123
122
124
(1i, p2,..., Un) where μ4, E {-3.3). Observations of a given are samples of its corresponding distribution. For example, in the 2-dimensional game, the 4 classes would be represented by the distributions: (V((3, -3), I2), AV(33, -3), /2), (((-3,3) 3), /2), and ((33)), /2) (we use a 5- dimensional signaling game for our experiments with 32 classes). The motivation for this setup is minimal need for feature extraction while still using vealevalued, stochastic inputs.
126
127
128
NAV For a multi-step environment, we use a 2-dimensional, obstacle-free navigation task. The sender agent observes the (x, y) position a receiver and produces a message; the receiver moves by producing an (x. v) vector. For a given episode, the receiver is initialized uniformly at random within a circle and must navigate towards a smaller circular goal region at the center. The agents are rewarded for both reaching the goal and for moving towards the center. An illustration is provided in The receiver's 's location and action are continuous variables.
130
132
134
AppendixIA A
135
3.3
Agents
136
Architecture Our architecture comprises two agents, conceptually speaking, but in practice, they are a single neural network. The sender and receiver are randomly initialized at the start of training, are trained together, and are tested together. The sender itself is 2-layer perceptron with tanh activations. The sender's input is envinomment--deeenden.. The output of the second layer is passed to 2017.Jang discrete, one-hot representation The activations of this layer can be thought of as the words forming the lexicon of the emergent language. Messages consist only of single one-hot vector (word) passed from sender to receiver. At evaluation time, the bottleneck layer functions deterministically as an argmax layer, emitting one-hot vectors. The receiver is a 1-layer perceptron which takes the output of the Gumbel-Softna layer as input. The receiver's output is environment--deeeneer An illustration and precise specification are provided in Appendices A and B
137
139
Gumbel-Softmal bottleneck laver [Maddison et al.) [017 Jang et al. 2017] which enables learning
141
143
145
146
147
Optimization Although only our NAV environment involves multi-step episodes, using a full reinforcement learning algorithm across all environments benefits comparability and extensibility in future work. Specifically, we use proximal policy optimization (PPO) (Schulman et al.] 2017] paired with Adam (Kingma and Ba 2015 to optimize the neural networks. PPO is widely used RL algorithm which selected primarily for its stability (e.g., training almost always converges, minimal hy hyperparameter tuning); attempts to train with "vanilla" advantage actor critic did not consistently
148
149
150
151
152
*Uiin a SSottax bottleneck layer allows for end-to-end backpropagation. making optimization faster and more consistent than using a backpropagation-fie free method like REINFORCE [Kharitonov et al. [2000] Williams| 1992). Nevertheless, future work may want to use REINFORCE for its more realistic assumptions about communication.

---

Algorithm 2 PPO pseudocode
n_updates: int >- buffer_size: int
for in (n_ 5 hollout_buffe for in range size) inner loop episode run_episod (model, environment) vollout_buffer append (episode) (model, rollout_buffer)
outer loop
153
converge. We use the PPO implementation of Stable Baselines (MIT license) built on Py PyTorch (BSD license) [Raflin et al. 2019 Paszke et al. 2019).
154
One relevant characteristic of PPO and similar algorithms is that in their training they contain an inner and outer loop analogous to FILEX (Algorithm[ T this is illustrated in Algorithm 2 The (main) outer loop consists of two steps: the inner loop which populates rollout buffer with "experience" from the environment and the updating of parameters based on that buffer. What is important to note
156
158
159
that the buffer is populated with data from the same model parameters, and it is not until after this
180
that model parameters change.
161
3.4
Hypothesis
162
Here we state the hypothesis used to evaluate FILEX. The sign of syperparameler-entrooy correlation observed in FILEX will be the same as what we observe for a corresponding hyperparameter in the
163
ELSs. We can state this more formally as: for each pair of corresponding hyperparameters (h, h²) in
184
185
FILEX and an ELS respectively,
sgn ()) sgn((cor(D'))
(5)
D {(ar, H(y)) x E XA: y Fi
(6)
D' {(E,, H(y)) x E Xw' y FLS
S. H(y) Σ yA loga yr
(8)
186
where corr(-) is the Kendall rank correlation coefficient (r) Kendall 1938). FILEX the distri- bution over frequency vectors yielded by the model for hyperparameter h set (assume likewise for EL H is Shannon entropy, and Xx. is the set of experimental values for hyperparameter h. A "sample" from ELS consists of training the agents in the environment, and estimating word frequencies by collecting the sender's messages over a random sample of inputs. Accordingly, our null hypothesis is that FILEX does meaningfully correspond to the ELSs, and thus the signs of correlation would be expected to match with a probability 0.5.
168
169
170
171
172
We intentionally formulate our hypothesis at this level of granularity: equality of direction (sign) of correlation rather stronger claims such as raw correlation: |corr(D) corr(D') < c or mean D' 2² We select this level of direction of correlation for few reasons. The level of simplicity of FILEX compared to the ELSs means that the unaccounted for factors would make supporting stronger hypotheses too difficult; furthermore, even if the hypothesis were defended, it would be less widely applicable for the same reasons. Additionally, the current literature tends speak of the general principles of emergent language at the level of "relationships" and "effects" rather than exact numeric approximations [Kharitonov et al. 2020 Resnick et al.) 2020)
174
squared error: X| ΣeEx(D(
176
177
178
179
180
181
Corresponding Hyperparameters A key component of the hypothesis is the correspondence of hyperparameters of the ELSs with those of FILEX. These correspondences are the foundation for applying reasoning about FILEX to the ELSs; accordingly, they also determine how the model will be empirically tested. We present five pairs of corresponding environment-agnoste hyperparameters in T Although environment-specific hyperparameters can easily correspond with those of FiLEX chose the agnostic for ease of experimentation and comparison.
182
185
185
5

---

Table Corresponding hyperparameters in the ELSs and FILEX.
ELS
FILEX
Time steps
N
Lexicon size
S
Learning rate
a
Buffer size
Temperature
Table 2: Kendall's 7's for various configurations. All values have significance of S 0.01.
Environment
Time Steps
Lexicon Size
Learning Rate
Buffer Size
Temperature
FILEX
-0.53
+0.67
-0.87
+0.93
+0.93
NoDYN
-0.81
+0.12
-0.74
+0.07
+0.58
RECON
-0.17
+0.93
-0.35
+0.84
+0.68
SIG
-0.49
+0.15
-0.16
+0.30
+0.49
NAV
-0.81
+0.36
-0.84
+0.20
+0.68
187
To identify these correspondences, it important understand the intuitive similarities between the ELSs and FILEX. Firstly, the weights of FILEX correspond the learned likelihood with which a given bottleneck unit is used in the ELS; turn, both of these correspond the frequency with which word is used in a language. Each iteration of FILEX's 's outer loop is analogous to whole cycle in the ELS of simulating episodes in the environment, receiving the rewards, and performing gradient descent with respect to the rewards (compare Algorithms/ aad
188
189
190
191
Based this analogy, the number of parameter updates taken throughout the course of training the ELS (i.e., the outer loop of PPO). S corresponds the size of the bottleneck layer in the ELS. a corresponds the learning rate (i.e., magnitude of parameter updates) in the ELS. The ELS has two analogs of 3. First, B corresponds the rollout buffer size of PPO because both control the number of iterations of the inner loop of training where episodes are collected before updating the weights, Second, B, more generally, control how smooth the updates to FILEX's weights are which makes analogous the temperature of the Gumbel-Softmai distribution in the ELS since a higher temperature results in smoother updates to the bottleneck's parameters.
explain the corresponding hyperparameters as follows. N corresponds
194
195
196
197
198
199
200
201
Experiments
202
Our experiments consist of comparing the correlation between the hyperparameters of FILEX and the ELSs and the Shannon entropy of lexicon at the end of training. The entropy for the ELSs is calculated based on the bottleneck unit (word) frequencies gathered by sampling from the sender's input distribution. To gather data for FILEX, we run Rust implementation of sampling algorithm. Each experiment consists of logarithmic sweep of hyperparameter plotted against the entropy yielded by those hyperparameters (see Appendix B for details).
204
206
208
209
Each point in the resulting scatter plots corresponds to an independent run of the model or ELS with the hyperparameter on the r-axis and entropy on the y-axis. The plots also include a Gaussian convolution of the data points (the solid line) to better illustrate the general trend of the data. The plots are presented Figure with the rank correlation coefficients Table
210
212
Discussion
213
5.1
Model evaluation
215
Looking at the signs of correlations shows that FILEX makes the correct prediction 20 out 20 times. Given simple one-sided binomial test, the empirical data rejects the null hypothesis at p 0.001.

---

Time Steps
Lexicon Size
Learning Rate
Buffer Size
Temperature
-8.53
+0.67
-0.87
T: +8.93
+0.93
FILEX
T: -4.81
T: +B.12
-0.74
T: +8.
+0.58
NoDYN
0.17
+0.93
0.35
T: +9.04
+0.68
RECON
T:
+5.
T: +8.38
S1G
T: -4.81
+0.36
-0.84
T: +8.29
+0.68
NAV
Figure 1: Plots of hyperparameters (I-axis, log scale) entropy (y-axis). Each corresponds to a particular environment. Each column corresponds to a particular hyperparameter. All y-axes are on the same scale with the dashed lines representing min/max entropy. The points are individual runs and the lines are a Gaussian convolution of the points.
Although this number drops 15 of 20 if require |r| 0.2, the binomial test rejects the null hypothesis with = 0.02 for this stronger hypothesis.
218
Though the directions of correlations predicted by FILEX are correct, looking at the plots show that ELSs do not always demonstrate the monotonicity predicted by the model. This is especially evident in Time Steps for RECON: moving left-to-right, the plot follows similar path to the other environment and FILEX at first but then diverges halfway through with increasing entropy. A possible explanation of this is that RECON allows learning new, useful words more easily than SIG or NAV, meaning that additional training can lead to further improvement. The conclusion we draw from these plots is that FILEX correctly predicts a sort of baseline correlation between the hyperparameters and entropy. Other works, Kharitonov et al. 2020), Chaabouni et al. [2021] for example, find similar correlations between entropy and bottleneck temperature. Nevertheless, this correlation can be overridden by the specifics of the environment.
220
222
224
225
226
228
229
5.2
Environment variability
230
When looking beyond just the direction of correlation the slopes and shapes of the curves, the four ELSs all present unique set of relationships between entropy their hyperparameters. This implies that none of these environments are reducible each other, that is, we cannot make observations about one environment and automatically assume they apply to other environments. Certainly this makes an researcher's 's task harder as learning general principles would not be possible from a single environment. Furthermore, there is a sensitivity to hyperparameters within given environment, which would imply that discovering general principles within single environment could not be done with just single set of hyperparameters.
232
233
234
296
7

---

238
Although this diversity in behavior makes modeling it more difficult, it also shows the importance of precision we get from a mathematical model. For example, say RECON has not been empirically tested and we wanted to predict the lexicon size-entropy relationship in RECON. It is the case that
239
241
we could simply observe the positive correlations in the other environments and predict the same RECON, but we could easily over-extrapolate and predict relatively shallow slope when RECON's
242
243
slope is relatively steep. What this paper's model, hypothesis, and evaluation offer in this situation is not& a more detailed prediction but "prepackaged" prediction which is precisely stated and supported
245
by data.
246
5.3
Applications to future work
247
There are two primary ways in which FILEX can be applied in future research. First, the model can be applied to and tested against further phenomena in emergent language (i.e., it is extensible). The fact that it is formulated mathematically means that it does not just predict correlations but mechanisms which account for the correlations. For example, FILEX's / hyperparameter was designed to account for Buffer Size and the Temperature experiment was conducted after the fact. The fact that FILEX describes both Buffer Size and Temperature with the same hyperparameter suggests that similar mechanisms account for their positive correlations with entropy. This statement about similar mechanisms, on the other hand, is not present set of one-off hypotheses about hyperparameter- entropy correlations derived from intuition. Second, FILEX and accompanying experiments provide an easy way for future research to discover confounding factors in their experiments. For example, an experiment might show that entropy decreases as rewards are scaled up, yet FILEX would suggest that this might be equivalent to simply increasing the learning rate rather than being its own unique
248
249
250
251
253
255
257
258
259
cause of the effect on entropy.
280
5.4
Methodological difficulties
261
The greatest challenge in the methodology of this work is not the formulation of the model but rather evaluating the quality of the model. In part, this is account of a lack of established baseline wwwel-coopparaaleeeaaea comparative analysis ("which is better?") is significantly casier than absolute analysis ("how good is this?") yet requires an adequate baseline to compare against. But more significantly, the granularity of experimentation is a design decision with no obvious answer.
283
265
266
For example, merely comparing the signs of rank correlations is very coarse grained as it makes minimal assumptions about the data (e.g., linearity, absence of outliers) and captures very little information about the data. Naturally, it is easier to apply such an analysis, and as mentioned before, researcher typically phrase hypotheses in terms of such correlations, but can only offer minimal support for applicability of the model to the actual system. On the other hand, evaluating the model's s ability to predict exact behavior of the system (e.g., measuring mean squared error of the model's predictions)
267
268
269
271
272
establish more precise link between model and system but might miss more general but important similaritics. For example, Lexicon Size for FILEX and NAv might show similar trends, but be different by constant, yielding high mean squared error.
273
275
subtle but significant methodological difficulty is the selection of hyperparameters. In RECON'S Time Steps plot, it is easy to see that changing the range of hyperparameters could easily yield either a positive or negative correlation when in reality there are both. To a certain extent, this can be resolved be choosing a "reasonable" range of hyperparameters based on values are typically, but this of little help to selection of FILEX's hyperparameters as there is no "typical usage." " For example, FILEX for 3 = 1 and 3 = 100 yield significantly different distributions, but there is no obvious a priori reason to say that one value of 3 should be preferred over the other for comparing to the ELSs Although additional hyperparameters increase the range of phenomena which the model can account for, the additional degrees of freedom can weaken the model's predictions by introducing confounding variables (cf. overparameterrizatonn)
276
277
279
281
282
283
285
One of the primary contributions of this work is to serve as a case study and example of working with explicitly defined models in studying deep learning- braneng-baseed emergent language. Thus, this paper
286
287
is starting point for future work to improve upon. One of the most important improvements would be finding more rigorous way to select "reasonable" experimental hyperparameters. Additionally, it would be better to develop the hypothesis and experimental in full before performing any evaluation; the process was somewhat iterative in this paper.
289
8

---

6
Conclusion
291
We have presented FILEX a mathematical model of lexicon entropy in deep harning-hased emergent language systems and demonstrated that, at the level of correlations, it accurately predicts the behavior of our emergent language environments. Opting for a mathematical model possesses the benefits of having a clear interpretation, making testable predictions, and being reused for new predictions in future studies. Although the model's hypothesis was testable, the process is not free
293
294
285
296
297
from non- con-taivial design decisions which affect the quality of evaluation. Nevertheless, this paper serves as starting point and example of how more rigorous models can be applied the study of emergent language.
296
References
300
David J. Aldous. Exchangeability and related topics. In P. L. Hennequin, editor, École d'Été de Probabilités de Saint-Flou XIII- - 1983, pages 1-198, Berlin, Heidelberg, 1985. Springer Berlin Heidelberg. ISBN 978-3-540-39316-0.
301
303
304
David Blei The chinese restaurant process, 2007. URL httpsttt//ww..c..rriiiie
edu/
pdf]
308
Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an emergent language game. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, page 981 1995, Brussels, Belgium. Oct 2018. Association for Computational Linguisties. doi: 10. 10.18663/v1/D18-11 1119. URL http::://aa/eeccc www.//ooonntooooyyyoomooooooooo. wtw.ogyy.rr//11-111111
307
308
309
310
Henry Brighton, Kenny Smith, and Simon Kirby. Language as evolutionary system. Physics of Life Reviews, 2: 2:1772226, 2005.
311
312
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020. doi: 10. 001533/111022.aclmmmaaa 407. URL htt.tt:://dd..ddd.or////// 10t.001163/////2222
314
315
doi
wwneelmman..444 407
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences, 118(12), Mar 2021. ISSN 0027-8424, 1091-6490, doi: 10.1073/pnas.
317
319
2016569118. WRt.tttps:///////. httph:ttt///:wt pnas. wwwooorgggteeeeee htt///nnnnnttt/// (acieeennn11////11 12/ 22211226611211111111111111111111
320
David Francis, Ella Rabinovich, Farhan Samir, David Mortensen, and Suzanne Stevenson. Quan- tifying Cognitive Factors in Lexical Decline. Transactions of the Association for Computa- tional Linguistics, 9:1529-1545, 12 2021. ISSN 2307-387X. doi: 10.1 1a04___00___11111111111111111111
322
URL
https://doi. org/ 10.
324
Shangmin Guo, Yi Ren, Simon Kirby, Kenny Smith, Kory Wallace Mathewson. and Stefano V. Albrecht. Expressivity Emergent Languages a Trade-off between Contextual Complexity and Unpredictability. September 2021. Utl.jtttps://operreiiiss wwt.:/oooummm.ddiii....... JWxjkM
325
326
327
Eric Jang, Shixian Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. In Proceedings of the 2017 International Conference on Learning Representations (ICLR), 2017.
328
329
URL htttp:://openrviiee/ net wtt.:eeettforiiae weeee/oorrmmiiddra3yeeeeee
330
M. G. Kendall. A new measure of rank correlation. Biometrika, 30(1-)):81 -13,, 06 1938. ISSN 0006- 3444. doi: 10. 1.0.i00mmet330.12......
org/ 10. www111999////////30 1-2.81
332
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimization in emergent languages. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 9220-52300 PMLR, 13-18 Jul 2020. URL http: htattt:://proceeeingge mlr, press//
333
335
336
v 1119/haaritonov20. html
Bohdan Khomtchouk and Shyam Sudhakaran. Modeling natural language emergence with integral transform theory and reinforcement learning. /8/20148/ [cs), Nov 2018. http
338
//arxiv. . org/ hatb//////..... 01431 arXiv: 1812.01431.

---

340
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR (Poster), 2015. htttp://aaaaii org/ abs/ 1412. 6980
341
342
Simon Kirby, Monica Tamariz, Hannah Cornish, and Kenny Smith. Compression and communication in the cultural evolution of linguistic structure. Cognition, 141:87-102, 2015. ISSN 0010-0277. doi: https://doi.org/ o00/.0.////////.oooo1o 10.
343
344
Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. orXiv:2006.02419 [cs], Jul 2020. URL 2006.02419.
348
02419
arXiv:
348
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language, 2017. URL https://openreviov.nn wttp:r/wwwwwio.eetoor/ouuiaaaa Hk8N3Sclg
349
350
351
David Lewis. Convention: A philosophical study. 1970.
Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relax- ation of discrete random variables. In Proceedings of the 2017 International Conference on Learn- ing Representations (ICLR), 2017.
353
355
dam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veti,o, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and R. Garnett, editors, Advances in Neurul Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http://t: http::///pppppeeeeeeippppeee neurips www.errip..ccpapppppppee cc/paper/ dee-
356
357
359
381
9015 pdf
-ibraay..
363
384
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave Anssi Kanervisto, and Noah Dormann. Stable baselines3. https://github. wwwchhu..coommmmi Cttpa//LRRRR///taaeeeeeaaaeeeeee basel 2019.
365
386
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M. Dai, and Kyunghyun Cho. Capacity, Bandwidth, and Compositionality in Emergent Language Learning. International Conference on Autonomous Agents and Multi-Agent Systems, April 2020. httt::t/a///rttii org/ hetg/abs/191 1910
368
389
11424
370
Mathicu Rita, Florian Strub, Bean-Bastienn Grill, Olivier Pietquin, and Emmanuel Dupoux. On the role of population heterogeneity in emergent communication. In International Conference Learning Representations, 2022. URL URL/https://///// /tt/itttps//opeerrviiee www.earrreeeeeneefffrrrrrr www./netfoouu dodidds500----- bZfI
372
373
Diana Rodríguez Luna, Edoardo Maria Ponti, Dicuwke Hupkes. and Elia Bruni. Internal and external pressures on language emergence: least effort, object constancy and frequency. In Findings of the Association for Computational Linguistics: EMNLP 2020, page 4428 4437,477,400 Online, 2020. Association for Computational Linguistics. doi: 10. N0.18863/y1/2000 1/t55/3311/222iiii/ wwndinns--nmnlpp....
374
375
376
URL
https://www, http://wwwwwwww....o.o.o
aclweb. org/ http//athooo//22o.oa1aaom fo220.fiddingge. emmlp. 397
378
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
379
Brian Skyrms. Signals: Evolution, learning, and information. OUP Oxford, 2010.
381
Agnieszka Slowik, Abhinav Gupta, William L. Hamilton, M. Jamnik, S. Holdenn, and C. Pal. Exploring Structural Inductive Biases in Emergent Communica- tion. undefined, 2020. URL https://www. www.:/seanniicccolrr..rr..g/o/oe
383
385
386
Ronald J Williams. Simple statistical gralient-folooinng algorithms for connectionist reinforcement learning, Machine learning, 8(3):229-256, 1992.
387
10

---

Checklist
388
389
1. For all authors...
390
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's s contributions and scope? [Yes] Claims in the abstract are specifically discussed in
381
Section [.]
393
(b) Did you describe the limitations of your work? [Yes] The primary limitations of the work are those discussed in Section 5.4
384
395
(c) Did you discuss any potential negative societal impacts of your work? [No] This work is basic research with few to no immediate applications. The nearest applications would be in evolutionary linguistics which we see as having minimal foreseeable negative societal impacts.
396
397
398
399
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
400
2. If you are including theoretical results...
402
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
403
(b) Did you include complete proofs of all theoretical results? [N/A]
404
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Supplemental material for submission; repo URL will be given if accepted.
406
407
408
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Choosing hyperparameters was not entirely straightforward, and this is discussed in Section[5.4]
410
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No] We did not run the entire set of experiments multiple times so to report error bars. Nevertheless, the individual experiments themselves account for stochasticity by displaying scatter plots; additionally we mention the p-values for the correlation values.
412
414
416
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] The ELS experiments took 36 hours on an in-house server with a 20-core i9-9900X CPU; no experiments used GPU. Although not tracked, we expect that less than 300 hours of total server computer time was used over the course of the whole project. The Rust implementation of FILEX takes on the order of seconds to run.
418
420
422
4. If you are using existing assets (c.g., code, data, models) or curating/releasing new assets...
423
(a) If your work uses existing assets, did you cite the creators? [Yes]
424
(b) Did you mention the license of the assets? [Yes] All assets have free licenses; supple- mental code is under the GPLv3 license.
425
426
(c) Did you include any new assets either in the supplemental material or as URL? [Yes] Code in supplemental material; repo URL will be given if accepted.
427
428
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
429
430
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
432
5. If you used crowdsoureing or conducted research with human subjects...
433
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
434
435
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
436
437
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
11

---

A Emergent language system illustration
Gumbal- Sattmax
S-agent (perceptron)
R-agent (perceptron)
Goal
[x. y) location
(x. acction
(a) The receiver (pictured) is rewarded for moving towards the goal region the center in the NAV environment.
(b) The agent architecture for NAV.
Figure 2
440 B Experiment parameters
Each experiment uses a logarithmic sweep across hyperparametens: the sweep is defined by Equation 442 where x and y are the inclusive upper and lower bounds respectively and n is the number steps to divide the interval into. The floor function is applied if the elements must be integers.
LS(x, v,
{
x
i E (0, ...., 1)
(9)
Hyperparameter
Default
Low
High
Steps
N
103
100
103
1000
26
23
2
1000
1
10-3
10
103
1000
B
8
10
103
1000
Table 3: Hyperparameters for the empirical evaluation FILEX. "Low" and "High" refer to the logarithmic sweep used for that experiment; default values used for all other experiments.
Hyperparameter
Default
Low
High
Steps
Time steps
105
102
106
600
Bottleneck size
26
23
28
600
Learning rate
3
10-0--
10
10-1
600
Buffer size
28
23
210
600
Temperature
1.5
10-1
10"
600
Table 4: Hyperparameters for the empirical evaluation of FILEX. "Low" and "High" refer to the logarithmic sweep used for that experiment; default values used for all other experiments. Please see code for further details and default values.
12


Reviews:
[{"review_id": "btNRqoeMr8m", "reviewer": "Reviewer_uYiF", "summary": "The paper proposes a stochastic process FILEX developed from the Chinese restaurant process, to mathematically model the lexicon entropy of emergent language between multiple agents (the speaker and the listener) in ELS (emergent language system) environments. The authors make correspondences between the FILEX and ELS to evaluate FILEX in real ELS environments. Experimental results on four ELS environments show that FILEX can correctly predict the correlation between hyperparameters and the lexicon entropy of well-trained emergent language in real ELS systems.", "questions": "1. Does the self-reinforcing assumption of emergent language in line 79 make sense in all ELS environments? Are there any evidences in the literature or experimental results?\n2. What’s the idea or intuitions behind the hyperparameter-entropy correlation in line 162? Why can it be used to evaluate whether FILEX matches the real ELS system?\n", "limitations": "According to the authors, there are weaknesses in this work that have not been addressed. Details are as above.\nThis paper does not have any potential negative societal impact.\n", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In this paper, the authors propose a mathematical model of lexicon entropy of emergent language between agents. Moreover, the authors conduct extensive experiments to verify the model, and the results show its effectiveness.\nStrengths:\n1.\tThe paper provides an inspiring idea of mathematically modeling the properties of the emergent language without training in real environments. Besides, the mathematical description is more precise and testable than natural language.\n2.\tThe experimental results demonstrate the effectiveness of the proposed FILEX, which verifies the feasibility of the idea.\nWeaknesses:\n1. The mathematical model may be not rigorous enough.\n----a) The proposed FILEX is based on the assumption that word use is reinforced in emergent language in sec 3.1. However, the paper only provides evidence in human language but not emergent language between agents. The paper does not prove the assumption fits all ELS environments either.\n----b) The correspondences of hyperparameters between FILEX and ELS in line 193 are based on analogy, thus the relationships between FILEX and ELS maybe not strong enough to support the alignments. Besides, beta in FILEX is corresponding to both buffer size and temperature in ELS, which may break the independence of the two hyperparameters despite the reason given in the paper.\n2. The experiments seem to be not well-organized and insufficient somewhat.\n----a) In sec 3.4, the meaning of the hyperparameter-entropy correlation and the reason why it can be used to evaluate whether FILEX matches ELS is not given.\n----b) It is better to add the experimental evidence of the self-reinforcing assumption of emergent language in the four ELS environments, which is the basic assumption of the proposed FILEX.\n----c) In line 173, as a testable mathematical model, FILEX is expected to provide more precise information, thus the equality of sign of correlation as the metric seems a little too weak. It would be better if the authors add some stronger metrics.\n3. The organization and presentation of the paper could be improved.\n----a) The introduction section misses necessary background introductions to the key concept “emergent language” and “lexicon entropy”, which may confuse readers from a broad domain.\n----b) The paper lacks a formal definition of the problem or task. Besides, it is better to give the input, output, and goal of FILEX before the technical details in sec 3.1.\n----c) In sec 3.4, it is better to provide more explanations about the obscure hyperparameter-entropy correlation, which may be unfamiliar to many readers.\n----d) The paper may be not well-organized. For example, the environments in sec 3.2 and result analysis in sec 5.1 should be included in the experimental parts.\n----e) There are some typos and small mistakes in the paper. For example, line 9 in Algorithm 1 is not correctly initialized and used elsewhere; “Section 3.3” -> “in Section 3.3” in line 102.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "JBG0VO9zL7B", "reviewer": "Reviewer_YbMp", "summary": "This paper fits into the Emergent Language literation and argues that prior work has been insufficiently rigorous when presenting their hypotheses. This work is presented as a starting point for mathematical methodology in developing and testing models of emergent language. \n\nTheir main contribution is an approach \"FiLex\" to understanding models of emergent language influenced by the Chinese Restaurant process (CRP). The relation here is to describe words as having a similar property to the tables in the CRP in that their usage is self-reinforcing. This approach is detailed in Section 3.1 with Algorithm 1 and the Formulation on page 3.\n\nTheir hypothesis, presented in Section 3.4, is that hyperparameters in FiLex correlate with hyperparameters commonly found in emergent language setups: time steps, lexicon size, learning rate, buffer size, and temperature. Their specific statement is that the sign of the correlation between those hyperparameters and entropy will be the same for FiLex as it is for the ELS setups. \n\nThey experimentally test this hypothesis by comparing FiLex against 4 ELS setups in Table 2 and Figure 1 and then make claims in Sec 5 (discussion) that this model lets practitioners more rigorously understand confounding factors.", "questions": "1. It's not clear in this paper whether we are asking whether the mathematical model that this process produces can answer questions of current ELS systems or are we asking whether it can help us illuminate where these systems are weak. Which one is it? The second is much more significant imo.\n\n2. Why should the sign of the per-hyperparameter entropy correlation be predictive about what my approach will do? Is there some foundation for that in other literature?\n\n3. What is the resulting word distribution of the CRP at steady state?\n\n4. My suggestion is to return to the models you've made (the 4 methods NoDyn Recon Sig and Nav) and try and see what, if anything, is invariant and unit-independent amongst these. I think that will bear more interesting fruit than this approach starting w CRP.", "limitations": "Societally, this is fine. Wrt limitations, see the above S&W section.", "rating": 3, "confidence": 4, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Originality:\n\nThis paper is original. It's taking a well-known idea (CRP) and applying it to a domain (emergent language) where the procedure is potentially beneficial for attaining understanding. It's unclear to me that this is actually the right approach for this domain and that's something that warrants greater discussion in the paper, but it's still an interesting direction that has merit.\n\n\nQuality:\n\nI don't think that this approach makes sense to do in ELS. I love the motivation to have better mathematical foundations in the space, but I disagree that this approach, or at least as presented, is the right path.\n\nFirst, why should it be the case that methods with same sign correlations in individual hyperparameters are actually indicative of each other? This is much too weak of a statement to be predictive as we could manufacture this artificially without much issue. The argument given at the end of section 3 that then qualitatively associates the variables doesn't fix this issue; changing just the algorithm used from PPO to something else that doesn't have a buffer size shows how fragile that is. \n\nThe explanation in 3.4 (L173-L180) points to this as well, suggesting that there are too many factors unaccounted for (due to FiLex's simplicity) to do much else besides this. But that's exactly what the paper set out to do at the beginning after discussing how prior work skipped out on this important step. Perhaps they did so for this same reason?\n\nSec 5, the evaluation, talks about making predictions (\"[... FiLex] makes the correct prediction 20 out of 20 times.\") and how that wouldn't happen if it wasn't predictive. But then the graphs (and the discussion) shows how tenuous that is. It's not very helpful for my research to run this method a bunch of times and hope that the correlation remains positive versus just varying a hyperparameter in my actual model and seeing what that does. Perhaps the authors are thinking that I cannot run my actual model that much because, say, it's a real robot. Okay that's fair, but I have no idea if this method would actually work in that setting and little belief that my model in totality would really be correlated given that a handful of hyperparameters are.\n\nClarity:\nThe paper is clear enough. The one area which would have been more helpful is to coalesce Table 2 and Figure 1 to be more apparently connected. It takes longer than I would have liked to understand what was going on in them (which are the results) and the stories they told.\n\nSignificance:\n\nThis paper is only significant in the world where the simple model is predictive of much bigger and more interesting models. The ones examined are not that, nor is it even assessed as predictive of anything in those settings.\n\nFor it to get to a place where we can confidently say that this is interesting for real-world settings, it would need to be aligned with actual (emergent) language learning. That is unfair to evaluate wrt learning new words in this setting (as neither FiLex nor most ELS systems are doing that), but it is fair to ask whether the learned language distributions are the same. Is that true? Is FiLex learning similar word distributions to the ELS models? More to the point, because FiLex's learned distribution won't ever be different given its simplicity, is it Zipfian? We expect that to if it's ever going to be able to model a full language. \n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EMud2WvFquC", "reviewer": "Reviewer_4yzD", "summary": "\nThis paper proposes a stochastic process, FILEX, to abstract out the essence of deep learning based emergent language systems. FILEX follows the intuition that the more a word is used, the more it will be used in the future. On four experimental settings, the authors show that the correlation between parameters of FILEX and the lexicon entropy is similar to the correlation between hyper parameters of neural networks and the lexicon entropy.", "questions": "How would the conclusion generalize to setting when the vocabulary is large and when the setting is compositional? ", "limitations": "The authors have properly discussed the limitations ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "strength \n\n- This work attempts to construct a simple enough theory for understanding emergent language. Such attempt is generally valuable for the overall community as the filed of emergent language itself is, emerging.\n- The proposed method is simple enough to mimic the model behavior thus providing a level of abstraction/ simplification for understanding. Additionally, I believe similar observation is also discussed in the previous VAE literature, just for the authors information[1].\n- This work is inspiring to me and I believe such work should also be inspiring to future work. Though I am not fully convinced with the existing experiments (see below), I am happy to see either future work may reinforce its conclusion or turn them over.\n\nweakness \n\n- The proposed methods is too simple to miss important details of many aspects of neural networks. When the authors link the parameters to of FILEX to the neural network hyper parameters, their reasons are more about intuition rather than rigorously mathematically discussions.\n- The experimental settings are too simple and may be hard to generalize to more complicated setting. One direct generalization is whether the conclusion will hold if the vocabulary size is large (say 10K) and compositional (say a sentence of length 10).\n\n[1] Alemi et. al. ICML 2018. Fixing a Broken ELBO", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject