On the Word Boundaries of Emergent Languages Based on Harris's Articulation Scheme
Authors: Ryo Ueda
Taiga Ishii
Yusuke Miyao
Venue: NeurIPS 2022 Submitted (2022)
Keywords: Emergent Communication
Emergent Language
Unsupervised Word Segmentation
Harris's Articulation Scheme

Abstract: The purpose of this paper is to investigate whether Harris's articulation scheme (HAS) also holds in emergent languages.
HAS is thought to be a universal property in natural languages that articulatory boundaries can be obtained from statistical information of phonems alone, without referring to word meanings.
Emergent languages are artificial communication protocols that arise between agents in a simulated environment and have been attracting attention in recent years.
It is considerd important to study the structure of emergent languages and the similarity to natural languages.
In this paper, we employ HAS as an unsupervised word segmentation method and verify whether emergent languages arising from signaling games have meaningful boundaries.
Our experiments showed that the emergent languages arising from signaling games satisfy some preconditions for HAS.
However, it was also suggested that the HAS-based segmentation boundaries are not necessarily semantically valid.

On the Word Boundaries of Emergent Languages Based on Harris's Articulation Scheme
Anonymous Author(s)
Affiliation
Address
email
Abstract
The purpose of this paper is to investigate whether Harris's articulation scheme (HAS) also holds in emergent languages. HAS is thought to be universal property in natural languages that articulatory boundaries can be obtained from statistical information of phonems alone, without referring to word meanings. Emergent languages are artificial communication protocols that arise between agents in a simulated environment and have been attracting attention in recent years. considerd important to study the structure of emergent languages and the simi- larity to natural languages. In this paper, we employ HAS as an unsupervised word segmentation method and verify whether emergent languages arising from signaling games have meaningful segments. Our experiments showed that the emergent languages arising from signaling games satisfy some preconditions for HAS. However, was also suggested that the HAS-based segmentation boundaries are not necessarily semantically valid.
10
11
12
Introduction
14
Communication protocols emerging among artificial agents in simulated environment are called emergent languages [Lazaridou and Baroni 2020 It is important to investigate their structure to recognize and bridge the gap between natural and emergent languages, several structural gaps have been reported [Koturr et al. 2017] Chaabouni et al. 2019]. For instance, Kottur et al. [2017] pointed out that emergent languages are not necessarily compositional. Such gaps undesirable because major motivations in this area are to develop interactive Al [Foerster et al. 2016 Mordatch and
16
18
20
Abbeel 2018 Lazaridou et al. 2020 and simulate the evolution of human language [Kibyy 2001 2001
22
Graesser et al. 2019 Dagan et al. 2021] Previous work examined whether emergent languages
have the same properties natural languages, such compositionality [e.g.. Kottur al., [017].
24
grammar (yan der Wal et al. 2020 entropy minimization [Kharitonov et al.) [2020]. and Zipf's ' law of
25
abbreviation (ZLA) [e.g., Chaabouni et al. 2019 Word segmentation would be another direction
26
understand the structure of emergent languages because natural languages not only have construction from word to sentence but also from phoneme to word [Martinet] [1600.. However, previous studies have not gone so far as to address word segmentation, as they treat each symbol in emergent messages as if were a "word" [Kottur al., 2017 van der Wal et al. 2020], ensure that whole message constructs just one "word" [Chaabouni et al. 2019 Kharitonov et al. 2020).
27
28
29
30
The purpose of this paper to study whether Harris's articulation scheme (HAS) Harris 1955 Ianaka-Ishil 2021] also holds emergent languages. HAS is a statistical universal in natural languages. Its basic idea is that we obtain word segments from the statistical information of phonemes, but without referring to word meanings? HAS can be used for unsupervised word
32
33
34
'ZL states that the more frequently word is used, the shorter it tends to be Zipf 1935
Note that this is different from the famous distributional hypothesis Harris 1954
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Sender
Input
ENC
DFC
Output R(mi)
Input /
Message m!!
ENC
DEG
Cross Entropy
Receiver R
Figure I: Illustration of a signaling game. Section 3.1 gives its formal definition. In each play, a sender agent obtains imput and converts it to sequential message. A receiver agent receives the messsage and converts it to an output. Each agent is represented as an encoder-decoder model.
35
segmentation (Tanaka- 2005] to allow us to study the structure of emergent languages. In addition, it should promising to apply such unsupervised methods, since word segments and meanings are not available beforehand in emergent languages.
36
37
The problem is whether emergent languages have meaningful segments. If not, then it means that find another gap between emergent and natural languages. In this paper, we pose several verifiable questions answer whether their segments are meaningful.
39
40
41 To simulate the emergence of language, we adopt Lewis's s signaling game Lewis 1969.. This game involves two agents called sender S and receiver R, and allows only way communication from S R. each play, obtains an input i E I and converts into a sequential message .M and predicts the original input. The goal of the game is the correct prediction R(m) = i. Figure illlustrates the signaling game. Here, we consider the set {m E M m S(E))iez the dataset of emergent language, to which the HAS-based boundary detection lanaka-Ishi) 2005 is applicable. The algorithm yields the segments of messages.
43
S(i) M. Then, R receives
45
47
Our experimental results showed that emergent languages arising from signaling games satisfy 49 two preconditions for HAS: (i) the conditional entropy (Eq. 2} decreases monotonically and (ii) the branching entropy (Eq. T) repeatedly falls and rises. However, it was also suggested that the HAS- based boundaries are not necessarily meaningful. Segments divided by the boundaries may not
51
serve meaning units, while words in natural languages do [Martinet 1960]. It is left for future work to bridge the gap between emergent and natural languages in terms of HAS, by giving rise to meaningful word boundaries.
53
2 Harris's Articulation Scheme
56 In the paper "From phoneme to morpheme" [Harris 1955], Harris hypothesized that word boundaries tend to occur points where the number of possible successive phonemes reaches local peak in
58 given context. Harris [1955] exemplifies the utterance "He's s clever" that has the phoneme sequence /hyyzllevor/" The number of possible successors after the first phoneme /h/ is 9: /w.yyi.e..,aaa.ook 60 Next, the number of possible successors after /hi/ increases to 14. Likewise, the number of possible
61 phonemes increases to 29 after /hiy/, stays 29 after /hiyz/, decreases to 11 after /hiyzk/, decreases
62 to after /hiyzkl/ and so on. Peak numbers are found at /yl, /z/, and /r/, which divides the phoneme sequence into /hth//////////// /klpe////// Thus, the utterance divided into "He", "s", and "clever".
63
64
Harris's s hypothesis can reformulated from an informatiion-thcorttic point of view by replacing the number of successors with entropy. In the following sections, review the mathematical formulation of the hypothesis as Harris's 's articulation scheme (HAS) and the HAS-based boundary 66
detection (Tanaka-Ishi 2005). HAS does involve statistical information of phonemes but does not involve word meanings. This important because it gives a natural explanation for a well-known linguistic concept called double articulation (Martinet] 1960). Martinet [1960) pointed that languages have two structures: phonemes (irrelevant to meanings) and meaning units (i.e., words and morphemes). HAS can construct meaning units without referring meanings.
68
89
70
71
2.1
Mathematical Formulation of Harris's Hypothesis
73
While Harris 1955 focuses on phonemes for word boundary detection, Tanaka-Ishii 2021 suggests that the hypothesis is also applicable to units other than phonemes. Therefore, in this section, a set
There may be other representations for the phonemes, but follow Harris's notation.

---

75
of units is called an alphaber A' as purely mathematical notion that is not restricted to phonemes. Tanaka Ishia [2005] uses characters for the same purpose. Moreover, Frantzi and Ananiadou [1996]
78
77
and Tanaka-Ishi and Ishii [207 investigate the detection of collocation from words.
Let X' be alphabet and n be the set of all n-grams X. We denote by X; random variable of X' indexed by i, and by Xıj random variable sequence from X, to X5. The formulation
79
80
by Tanaka-Ishie Issii (2055 involves two kinds of entropy: branching entropy and conditional entropy Cover and Thomas, 20066] | The branching entropy of a random variable Xn after sequence
81
To 82
Fn-1
Xn deñindd as:
((8) = H(X,, 10m-11 = - P(z loga P(x |s),
(1)
ZEX
where P(x 8) P(X, - I Xom- = 8). Intuitively, the branching entropy h(s) means how 85
many elements occur after or the uncertainty of the next element after In addition to h(s). a5 the conditional entropy of a random variable Xn, after an n-gram sequence Xo:n- is defined as:
H(n) H(X, (u =- Σ
P(s)
Σ
P(x
s) log2 P(x |ss
(2)
SEAt
ZEN
86
where P(s) P(Xon-1 = s). The conditional entropy H(n) can be regarded as the mean of 87 h(s) over 7-gram sequences E Xm since (n) E.Ex P(s)h(s). H (n)) known to decrease monotonically in natural languages [Bell et al.) 1990J. Thus, for partial sequence 20:-1 E X"",
88
89 h(x0n-2) > h(xmm-1) holds on average, although hiss) repeatedly falls and rises depending on specific Based on such properties, Harris's 's articulation scheme (HAS) is formulated as
90
If there some partial sequence Tomn-1
X (n 1)
(3)
s.t. h(Fo.-22 22 < h(Fon-1), then Fn is at a boundary.
91
2.2 Boundary Detection Algorithm Based on Harris's Articulation Scheme
In this section, we introduce the HAS-based boundary detection algorithm (Tanaka-ssh Ishii 2005). Let
93 s = 20 20.-.220 In- E X We denote by Stj its partial sequence 2; ...-.-20 Zy.. Given s and a parameter
94
threshold, the boundary detection algorithm yields boundaries B proceeds as follows:
85
1: < 0: w < 1; B +-
96
2: while i do
97
3:
Compute h h(855+011)
98
4:
if > 1 and h($,1+=-1) h (( threshold then
99
5:
B + B U w}
100
6:
end if
7:
if i + w < n 1 then
102
8:
w w+++] +1
103
9:
else
104
iti+ w
105
11:
end if
106
end while
107
Since our targets are emergent languages, the outputs of the boundary detection algorithm do not necessarily mean articulatory boundaries. Instead, we call them hypothetical boundaries (hypo- boundaries) and refer to the segments split by hypo-boundaries as hypo-segments. Note that there are other similar methods such as Kempel [1999]. We chose Tanaka-Ishii (2005) because it performs well not only for English but also for Chinese, which has many one-character words. Emergent languages might also have such words. With this algorithm, Tanaka- Ishii and Jin [2008) reported F-score = 83.6% for word boundary detection from phonemes in English and F-score = 83.8% for word boundary detection from characters in Chinese. They are considerably high scores for unsupervised settings.
108
109
111
113
115
4 'The term "branching entropy" is from Tanaka-Iste Ishii and Jin 2008 but the definition per se is quite basic. AAthoough this is called hypothesis in Tanaka-Ishh Ishil 2005 Tanaka- Ishii and Jin (2066] and Tanaka Ishii and [2007] we refer to as scheme following the recent publication [Tanaka- Ishi 2021].
Ishii
The original algorithm involves another parameter maxlen to ensure < maxlen for practical reasons. We
omit it because the message length in emergent languages is fixed in this paper (see Section 3).
3

---

116
3
Emergent Language Arising from Signaling Game
We have define environments, agent architectures, and optimization methods for language emer- gence simulations. This paper adopts the framework of |Chabbonni et al. [2020). We define an environment in Section 3.1, specify the agent architecture and optimization methods in Section 3.2
118
119
120
and also give an explanation of the compositionality of emergent languages in Section 3.3
121
3.1
Signaling Game
122
An environment is formulated based on Lewis's signaling game [Lewis]]] 1969.. A signaling game G consists of a quadruple (I, .M, S, Rt), where I is an input space, .M a message space, : - M is sender agent, and R .M- + I is receiver agent. The goal is the correct reconstruction
123
125
i = R()(i)) for all E I. While the input space I and the message space M are fixed, the agents S. R are trained for the goal. An illustration of signaling game is shown in Figure Following
127
Chaabouni et al. [2020], we define I as an attribute- attribute-value value set Diatt (defined below) and M as a set
of discrete sequences of fixed length &o over a finite alphabet A:
I
Diiate Reat
M == A = {ai ... .x ag E A}.
(4)
129
Attribute- Value Set Let Batt: Teatt be positive integers called the number of attributes and the number of values. Then, an attribute-value set Diatt is the set of ordered tuples defined as follows:
130
Peal {(11, Dr By E {1...., Rool! }}
(5)
131
This is an abstraction of an attribute- value object paradigm [e.g.. Kottur et al., 2017] by Chaabouni et al. [2020]. Intuitively, each index j of vector (uI,... ..j..... Un is attribute (e.g., color),
133
while each 2y is an attribute value (e.g., blue, green, red, and purple)
134
3.2
Architecture and Optimization
135
We follow
Chaabouni et al. 20200 as well for the architecture and optimization method.
196
the sender decoder and the receiver encoder are based on single-layer GRUs [Cho et al. 2014), while the sender encoder and the receiver decoder are linear functions. Each element i Diaate has to be vectorized so that can be fed into or output from the linear functions. Formally, each i (Vi,..., Una.)) is converted into the Natt X -ima--imensional vector which is the concatenation of natt one-hot representations of U.. During training. the sender samples messages probabilistically. During the test time, it samples them greedily so that it serves as a deterministic function. Similarly, the receiver's 's output layer, followed by the Softmax, determines Nhttr categorical distributions over values {1,..., Nmat] during training. During the test time, Natt values are greedily sampled from the distributions.
Architecture
Each agent is represented as an encoder-decoder model (Figure Ty:
137
138
139
140
141
142
145
145
The agents are optimized with the stochastic computation graph (Schulman et al.. 2013) that is a combination of REINFORCE (Williams) 1992) and standard backpropagation. The sender is optimized with the former, while the receiver optimized with the latter.
Optimization
148
147
3.3
Compositionality of Emergent Languages
149
An attribute- value set Diaat by Chaabouni et al. (2200) is an extension of an attribute- value setting KKttur et al., (0177) introduced to measure the compositionality of emergent languages. While the concept of compositionality varies from domain to domain, researchers in this area typically regard it the disentanglement of representation learning. Kottur et al. (017). for instance, set up an environment where objects have two attributes: color and shape, each of which has several possible values (e.g., blue, red, ... for color and circle, star, ... for shape). They assumed that if a language is sufficiently compositional, each message would be a composition of symbols denoting the color value and shape value separately. This concept has been the basis for subsequent studies Li and
151
152
153
154
155
156
157
Bowling
2019 Andreas 2019 Ren et al. 2020 Chaabouni et al. 2020
7 AAthhough the game is extremely simple, it is suitable to avoid some pitfalls. Lowe et al. 2019 pointed out that agents may not communicate effectively in more complex games than in a signaling game Bouchacourt and Baroni (0188) suggested that agents fail capture conceptual properties when I is a set of images.

---

158
Topographic Similarity Topographic Similarity (TopSim) [Brighton and Kirby 2006 Lazaridou
159
ct al., 2018 the de facto compositionality measure in emergent communication literature. Suppose we have distance functions dz, dM for spaces I, M. respectively. TTopSim is defined as the Spearman correlation between distances (1 12) and (S( (i2)) for all i1, i2 E I s.t. in * 12. This definition reflects an intuition that compositional languages should map similar (resp. dissimilar)
160
161
162
165
inputs to similar (resp. dissimilar) messages. Following previous work using attribute-value objects ge.g.. Chaabouni et al. 2020), we define dz the Hamming distance and dm the edit distance. Because this paper is about message segmentation, we can consider two types of edit distance. One is the "character" edit distance that regards elements a E A as symbols. The other is the "word" edit distance that regards repo-segments as symbols. Let us call the former C-TopSim and the latter
165
186
167
W- TopSim.
Entropy
4
Problem Definition
169
Q1: H decreses monotonically?
170
The purpose of this paper is to study whether Harris's articulation scheme (HAS) also holds in emergent languages. However, this question is too vague to answer. We first divide it into the following:
Q2: h jitters?
172
173
Q1. Does the conditional entropy H decrease monotonically?
174
Q2. Does the branching entropy repeatedly fall and rise?
175
Q3. Do hypo-boundaries represent meaningful boundaries?
Message
176
is the same as the original question, except that Q3 is slightly more formal. However, we have to answer Q1 and Q2 beforchand because HAS implicitly takes it for granted that H decreases mono- tonically and h jitters. Although both Q1 and Q2 generally hold in natural languages, neither of them is trivial in emergent languages.
Q3
178
Hypo-segments
180
Figure 2illustrates Q1 Q2 and Q3
Q3: Are They Meaningful?
182
is straightforward to answer Q and Q2 as we just need to calculate
Figure 2: Illustration of ques- tions.
H and h. In contrast, Q3 is still vague to answer, since we do not
184
have prior knowledge about the boundaries of emergent languages and do not even know if they have
185
such boundaries. To mitigate it, we posit the following necessary conditions for Q3 Let G be a game
DDaa AF, S, R). If the answer to Q3 is yes, then:
186
187
CI. the mean number of hypo-boundaries per message should increase as Nat increases,
188
C2. the size of the vocabulary (set of all hypo-segments) should increase as nea! increases,
C3. W-TopSim should be higher than C-TopSim.
190
An attribute-value set Aval. was originally introduced to measure composi- tionality. Composiiionality in this context, means how symbols in message separately denote the components of meaning. In our case, each segment, or word, can be thought of as a certain unit that denotes the attribute values, that the number of words in message should increase as the corresponding attributes increase. Therefore, if the answer to Q3 is yes, then CT should be valid. Likewise, the size of the vocabulary should be larger in proportion to the number of values neall motivating C2] Here, we mean by vocabulary the set of all hypo- bypo-segments. Note that the message length is fixed, because otherwise the number of hypo-segments would be subject to variable message length as well as (Natt, Reaa)) ), and the implication of results would be obscure.
A bout C1 and C2
192
194
195
196
198
About C comes from the analogy of the linguistic concept called double articulation Martinet 1960). In natural languages, meanings are quite arbitrarily related to the phonemes that construct 200
them. In contrast, the meanings are less arbitrarily related to the words. The phonemes do not denote meaning units but the words do. In our case, for example, the attribute-value object (RED, CIRCLE) seems less compositionally related to the character sequence "r.e.d.c.i.r..Ilee,, while it seems more compositionally related to the word sequence "red,circle." " This intuition motivates C3
202
204
Based on conditions CI and C3 Q3 is restated as follows: (Q3-1) Does the mean number of
206
hypo-boundaries per message increase as Natt increases?
(Q3-2) Does the vocabulary size increase (Q3-3) Is W-TopSim higher than C-TopSim?
as nepa increases?
5

---

Experimental Setup
208
209
5.1
Parameter Settings
210
Input Space
Phatt and Deal have to be varied to answer Q3-1
Q3-2 and Q3-3 while the sizes of
the input spaces [I] (Nhv
M Mart must be equal to each other balance the complexities of games.
212
Therefore, we fix [Z| = 4096 and vary (Natt, Real)) as follows:
(natt, Mlva)) E (096) (2, 64), (3, 6). (4,8), (6, 4), (12,
Message Space The message length & and alphabet A have be determined for message space
M A We set & = 32, similarly to previous work on ZLA CChaboouni et al. 2019 Rita et al. 2020 Ueda and Washio 2021] that regards each A as "character." Note that 32
214
215
set much
216
onger than those of previous work on compositionality [Chaabouni et al., 2020) Ren et al.
2020 Li
and Bowling, 2019] that typically adopts k = Nott as if each symbol a E A were "word. We set
A = 21,2,.. 8J. Its size |A| should be as small as possible to avoid the problem of data sparsity 218
219
when applying boundary detection, and ensure that each symbol E serves as a "character."
220
In preliminary experiments, we tested [A| E {2, 8, 16} and found that learning is stable when
221
8.
222
2020 for agent arthitectures and pptmization methods. The hidden size of GRU [Cho al. [0144] set to 500, following Chaabouni
Architecture and Optimization We follow Chaabouni et al.
224
ct al. 2020]. All data from an input space I Diatt are used for training. This dataset is upsampled
225
100 times following the default setting of the code Chaabouni al. 20200. The learning rate
226
set to 0.001, which also follows Chaabouni et al. [2020] Based on our preliminary experiments to
227
explore stable learning, sender and receiver are trained for 200 epochs and the coefficient of the entropy regularizer is set to 0.01.
228
Boundary Detection Algorithm The boundary detection algorithm involves parameter threshold. 230
Since the appropriate value of threshold is unclear, we vary threshold as follows:
threshold E {0, 0.25, 0.5, 0.75, 1.25, 1.5, 2..
5.2
Implementation, Number of Trials, and Language Validity
232
We implemented the code for training agents using the EGG toolkit [Kharitonov al. 2019 EGG also includes the implementation code of Chaabouni et al. (2020). which we largely refer to. They are published under the MIT license. For now, our code available on Anonymous GitHub For each
233
234
Ratt, Dvat) configuration, agents are trained 8 times with different random seeds. Each run took few hours with a single GPU In the following sections, an emergent language with communication success rate of more than 90% is called successful language.
236
237
Results
239
As result of training agents, we obtained 7, 8, 6, 7, and 6 successful languages out of runs for configurations (Matt, Rivat) = (1, 4096), (2, 64), (3. 16). (4, (), (6, 4), and (12, 2), respectively.
240
6.1
Conditional Entropy Monotonically Decreases
To verify Q1 we show the conditional entropy H(n) (Bq. in Figure 3 In Figure 3
the conditional entropies of the successful languages (solid red lines) decrease monotonically This confirms Q1 successful languages. Interestingly, the conditional entropies of emergent languages derived from untrained senders do not necessarily decrease, shown as dashed blue lines in Figure 3 The monotonic decrease in conditional entropy emerges after training agents.
243
244
245
248
10
"NVIDIA A100.
One might think that the conditional entropy cannot increase by its definition. However, this is not the case our setting (see Appendix A for more details).

---

3.0
Emergent language befare training
25
M21
2,5
Successful language after training
NJ...4.5
2.0
N4..... 4.81
1.5
1.0
0.5
00
0.0
0
30
S0n60toon boundary
Figure 3: Conditional entropy Hn). Dashed Figure 4:
Example transition sequences of the branching entropy message "3.8,4,...
blue lines represent (n) of languages from un- trained agents that finally learned successful lan- in successful language for (Natt, Nyal)) guages, while solid red lines represent (n) of 64).
successful languages.
6.2
Branching Entropy Repeatedly Falls and Rises
248
Next, to answer Q2 we computed the branching entropy h(s) (Eq. 1) of the successful languages and applied boundary detection. As an example, we show few actual transitions of h(s)) in Figure in which y-axis represents the value of h(s) and x-axis represents a message "3,8.4..... 4,4,, The message is randomly sampled from successful language when (Ratt, Nyat)
249
250
251
64). The boundary detection algorithm with threshold = 1 yields three popo-boundariess that are represented as dashed black lines (Figure 4 Blue, yellow and green lines with triangle markers represent the transitions of h(s) that yield hypo-boundaries. Note that the (i 1)-th transition of hs)) does not necessarily start from the i-th hypo-boundary, due to the definition of the algorithm. For instance, second transition overlaps the first hypo-boundary. While the conditional entropy decreases monotonically
252
253
255
257
as shown Figure 3] the branching entropy repeatedly falls and rises Figure Moreover, we show the mean number of hypo- aopo-boundaries per message in Figure 5) Figure 5 indicates that for any (natts uyat) configuration, there are srpo-bounnaries threshold repeatedly falls and rises. These results validate Q2
259
2, i.c., the brancing entropy
6.3
Hypo-Boundaries May Not Be Meaningful Boundaries
262
Next, we investigate whether Q3-1 Q3-2 and Q3-3
hold in successful languages. The results in may not be true: hypo-boundaries may not be
the following sections falsify all of them. Thus, Q3
meaningful boundaries.
264
285
Mean Number of Hypo-Boundaries per Message See Figure 5 again. The figure shows that the mean number sypo-boundaries per message does not increase as natt increases. does not decrease, either. This result falsifies Q3-1 as other configurations.
266
Even when
1, there are as many hypo-boundaries
268
289
Figure shows the mean vocabulary sizes for each (Natt. Reat). The vocabulary size does not increase as Neat increases, which falsifies Q3-2) However, focusing on (Natt. Neal (2, 64), (3, 16), (4, 8), (6, 4)} and 0.25 < threshold < 1, there is weak tendency to support C2
Vocabulary Size
270
It
272
suggests that hypo-segments are not completely meaningless either.
shows C-Topsim and W-Topsim for each (Natt Real)) and threshold Note that C-TopSim TopSim with "character" edit distance and W-TopSim TopSim with "word" edit distance. In Figure threshold = oc corresponds to C-TopSim, while the others
C-TopSim vs W-TopSim Figure 7
274
275
Note that TopSim only be defined when Natt

---

threshold =
7000
dhreshold 0
threshold = 0.25
threshold 0.25
Mcan Ilypo bounddbuddrriss per
threshold 0.5
6000
threshold = 0.5
15
threshold 0.15
threshold 0.75
threshold
4000
threshold =
threshold = 1.25
3000
threshold
threshold - 1.5
mhrehoold 1.5
5
threshold = 1.75
2000
threshold = 1.75
threshold
1000
threshold =
(1, (.46666
(2,64)
(3.16)
8)
(12.2)
(1, (1,4096)
(2,64)
(3,16)
(4.8)
(6.
(6.
(12.
Datu Drall Configuration
(ngg. Configuration
Figure 5: Mean number of hypo-boundaries per message in successful languages. threshold varies according Eq. 7 Each data point averaged over random seeds and shaded regions represent standard error of mean (SEM).
Vocabulary size in successful lan- gauges. threshold varies according to to[Eq 7 Each data point averaged over random seeds and shaded regions represent one SEM.
Figure 6:
0.12
Mett Ava) = 12, 64)
(Namt fival) - (2, 64)
(ean, Roal = 13, 16)
0.10
[nat, 64) trandom boundary)
(eat, (4,8)
(Mat, Aowl
0.08
TopSim
(flat, Powa) = (12.2)
TopSim
0.06
0.2
0.1
0.0
0.00
0.5
1
15
0.0
0.5
1.0
1.5
2.0
tireshokl
threshoid
Figure C-TopSim and W-TopSim in successful languages. threshold -oo corresponds to C- TopSim, while other threshold correspond W- TopSim. Each data point is averaged over random seeds and shaded regions represent one SEM.
hypo-boundary-basee W-TopSim com- pared to handombboundary-ae successful languages for (Matt: Teeal) = (2, 64). Each data point is averaged over random seeds and shaded regions represent one SEM.
Figure 8:
W-TopSim in
correspond to W-TopSim. 13 Our assumption in Q3-3 was C-TopSim W-TopSim. On the contrary,
277
Figure shows a clear tendency for C-TopSim W-TopSim, which falsifies Q3-3 Hypo-boundaries may not be meaningful. However, they may not be completely meaningless, either. This is because the W-TopSim is higher than the random-bounde bou-daby-dddraeeeee W-TopSim in Figure Here, mean by random boundaries the boundaries chosen at random in the same number as hy nyoobboundaries each message. Other (Natt, nvat) configurations show similar tendencies (see Appendix B).
279
hypo-boundary-basee
281
6.4
Further Investigation: Word Length and Word Frequency
284
The results so far are related to compositionality of emergent languages [e.g. Kottur et al. 2017]. In this section, further associate results with previous discussions Zipf"s 's law of abbreviation (ZLA) in emergent languages [Chaabouni et al. 2019 Rita et al. 2020 Ueda and Washio 2021). ZLA is known statistical property in natural languages that the more frequently word is used,
286
If we were to apply boundary detection with threshold = oc. would regard every data point in message as boundary. In other words. W- TopSim with threshold = -oo would be identical to C-TopSim. We adopt this notation in order represent C-TopSim and W-TopSim in unified manner in single figure.

---

288
By considering hypo-segments as "words," we can check whether Figure 9 shows the hypo-segment lengths sorted by frequency rank for (Rate, (nal) (1, 4096) 14 If hypo-segments follow ZIAA ideally, they should show monotonic increase. The distribution of the lengths of the hypo-segments shows a clear ZL.A-like tendency for threshold E {0, 0.5). although the tendencies are less clear for the other threshold [5
the shorter it is Zipi
1935
289
hypo-segments follow ZLA.
291
292
It means that nypo-segments follow ZLA with an appropriate threshold value. Other (Ratt: Ryal) configurations show similar tendencies (see Appendix C).
293
7
Discussion
Semessegnaa Length
10'
296
In Section 6.1 we showed that the conditional
297
entropy H (n) decreases monotonically in emer-
298
gent languages, confirming QT In Section 6.2.
299
we demonstrated that the branching entropy h(s) repeatedly falls and rises in emergent languages,
10a
300
10a
10
102
10a
which confirms [Q2 It is an intriguing result, considering the discussions of Kharitonov al.
Frequency Rank
n/al) 40%)), threchoodd
302
303
2020), who showed that the entropy decreases
(Nam nual) - [1, 4096), threshold - 0.5
304
to the minimum for successful communication
(Man, nual) = (1, 40 96),), threshold 1.5
(Mat. P/g)) 4096), threshold =
305
if the message length & 1. In contrast, our
306
results suggest that the (branching) entropy does
not simply fall to the minimum when the mes- sage length is longer. However, in Sectioo 6.3 results indicate that the hypo- bonndooussssess may not be meaningful since Q3-1 Q3-2 and were falsified.
Hypo-segment lengths sorted by fre- quency rank for (Natt. Meat) (1,4..)). Each data point averaged over random seeds and shaded regions represent one SEM.
Figure 9:
308
310
312 Nevertheless, hypo-bounddaris may not be completely meaningless either. This because the W-TopSim is higher than the mindom-bounddry-baae W-TopSim. It suggests that HAS- boundary detection worked to some extent. addition, the hypo-segments show ZLA-like tendencies with certain threshold values. This is a suggestive result because neither imposed a length penalty on messages [Chaabouni et al. 2020), modeled the laziness/imppatinnee of agents [Rita et al. 2020). nor modeled short-term memories Ueda and Washio 202T]. Of course, it is important to note that may be just an artifact, analogous to the fact that even a monkey typing sequence divided by the "white space" follows ZLA [Miller 1957].
314
316
318
320
This paper showed that there is a gap between emergent and natural languages in terms of word segmentation. There are some potential methods to bridge the gap. For example, several methods have been proposed facilitate the compositionality of emergent languages, such as iterated learning Ren et al. 2020]. the ease-of oftechhfn paradigm [Li and Bowling, 2019), and concept game [Mu and
322
323
324
Goodman 2021]. The regularizations for ZLA mentioned above might also help for this purpose.
These are left for future work.
8 Conclusion
In this paper, investigated whether Harris's articulation scheme (HAS) also holds in emergent languages. Emergent languages are artificial communication protocols emerging between agents, while HAS is statistical universal in natural languages. HAS can be used for unsupervised word segmentation. Our experimental results suggest that although emergent languages satisfy some prerequisites for HAS, HAS-basce word boundaries may not be meaningful. Our contributions are (1) to focus on the word segmentation of emergent languages, (2) to pose verifiable questions to
328
329
330
331
332
333
answer whether emergent languages have meaningful segments, and (3) to show another gap between emergent and natural languages. It is left for future work to bridge the gap between emergent and natural languages in terms of HAS.
334
395
We picked up only threshold e (0.5, 1.5, and adopted log log graph for readability. 1he plot shows the zigzagging behavior for threshold = 1.5 and most of the hypo segment lengths hit the message length & 32 for threshold = 2.

---

References
336
337
Jacob Andreas. Measuring compasitionality in representation learning. In 7th International Confer- ence on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019. httppsst////ppeeeeeii
338
339
www..eee//ffoeooo HJ220000kk
340
Timothy C. Bell, John G. Cleary, and Ian H. Witten. Text Compression. Prentice-Hoe Hall, Inc., 1990.
341
Diane Bouchacourt and Marco Baroni. How agents see things: On visual representations in an emergent language game. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ichi Tsujii, editors, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, Brussels, Belgium, October 31 November 4, 2018, pages 981-988 Association for Computational Linguistics, 2018. URL https:t/ttii.orrrrt/eeeeoo org/10....... wtt..///.....11..1/////////111 11l.1/6///v11-11111
342
343
344
345
346
Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artif. Life, 12(2):229-242, 2006. http
347
artl. 22.0612.2.00. 12.1.2.2222 229
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Beygelzimer, Florence Allché-Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, neurips cc/ httpp/ppp//11/////eeee pages 6290-6300, 2019 URL att.aaableeeeee0--trrr........ Abstractt.... htmml
350
Hanna M. Wallach, Hugo Larochelle, Alina
352
354
$1ca0ca71184bbab3dd (ee777222255ae220.
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACI. 2020, Online, July 5-10, 2020, pages 1427-4442 Association for Computational Linguisties, 2020. https//ts://doi. org/ 10. 1ww.31811111000 acl-main
357
359
360
361
407
362
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qutar, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1724-1734 1734. ACL, 2014. URL htttt::t////oo org 11/ 1ttg///0.111///111414111
383
364
385
366
387
388
Thomas M. Cover and Joy A. Thomas. Elements of Information Theory (Wiley Series in Telecommuu- nications and Signal Processing). Wiley- bilyyInntrsscienc, 2006.
389
370
Gautier Dagan, Dieuwke Hupkes, and Elia Bruni. Co-evolution of language and agents in referential games. In Paola Merlo, Jörg Tiedemann, and Reut Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, EACL 2021, Online. April 19 - 23, 2021, pages 0099990004000 Association for Computational Linguistics, 2021. URL
371
372
373
374
http:://///aaathloggo.o org/ 2021. eacl main. 260/
375
Jakob N. Foerster, Yannis M. Assael, Nando de Freitas, and Shimon Whiteson. Learning to com- municate with deep multi-agent reinforcement learning. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Advances in Neural Information Processing Systems 29: Annual Conference on Neural Information Processing Systems 2016, December 5-10, 2016, Barcelona, Spain, pages http://///aaaaaaeeoofaaaa////ee0
378
377
378
379
2016. http://bbbbf.-Abtbaaa.... html
380
neurips.
381
Katerina T. Frantzi and Sophia Ananiadou. Extracting nested collocations. In 16th International Conference Computational Linguistics, Proceedings of the Conference, COLING 1996, Center for Sprogteknologi, Copenhagen, Denmark, August 5-9, 1996, pages 41 41-46, 1996. URL https
383
//aclanthologe .rgg/ C0//CC96--0 C
10

---

385
Laura Graesser, Kyunghyun Cho, and Douwe Kiela. Emergent linguistic phenomena in multi-agent communication games. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMN..P-ICCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3698-37787 3708. Association for Computational Linguisties, 2019.
386
386
389
390
381
Zellig S. Harris. Distributional structure. WORD, 10(23):146-162, 1954.
392
7ellig S. Harris. From phoneme to morpheme. Language, 31(2):190-22) 1955. URL http:
393
// www. www.jstor.oo. wwwpjjsstrosossapi.. /raarrrrrg/saabe/110666
394
André Kempe. Experiments in unsupervised entropy-based corpus segmentation. In Miles Oshorne and Erik F. Tjong Kim Sang, editors, Proceedings of the 1999 Workshop on Computational Natural Language Learning, CoNI.199,, Held in cooperation with EACL '99, Bergen, Norway, June 12, org 19t5//99-070 0702/
385
396
387
1999, pages 7-13. ACL, 1999. URL ttttpp::/aacaattoolll
396
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. EGG: a toolkit for research on emergence of language in games. In Sebastian Padó and Ruihong Huang, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019 System Demonstrations, pages 55 60. Association for Computational Linguistics, 2019. URL https://doi. org wwwp/....//0 (0/655//v//19--111
400
402
404
Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimization in emergent languages. In Proceedings of the 37th International Conference on Machine Learning. ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pages 5220-5230. PMLR, 2020. URL http:://proceedingg// .mm.er mlr. press/v119/ kharitonov20a htmll
405
406
407
409
Simon Kirby. Spontaneous evolution of linguistic structure-an iterated learning model of the emer- gence of regularity and irregularity. IEEE Trans. Evol. Comput., 5(2): 110- 2001. URL https://doi lrt.dooo..ogg//// 10. 1109/4235... 428449143339
411
412 Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Proceedings of the 2017 Conference on Empirical Methods in Natural Longuage Processing, EMNLP 2017, Copenhagen, Denmark, September 9-11, 2017, pages 2999622267.... Association for Computational Linguistics, 2017. htt.ttpp:e/doiioogg/10.0o0.
414
416
1321
417
Angeliki Lazaridou and Marco Baroni. Emergent communciioo in the deep learning era. CORR, abs/2006.024 024119, 2020. URL btttpp:://axxiv.oo//aaa////////
418
024199
419
Angelkki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https:
421
422
net/
423
forum?id-HJGv12-- AW
424
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel R. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 10, 2020, pages 7663 7674. Association for Computational Linguistics, 2020. URL httlpss/:///o:: org 00g/10.110 18865/
425
426
427
428
v1 v1/2020. acl maii...in.emm 685
430
David K. Lewis. Convention: A Philosophical Study. Wiley-Blackwell, 1969.
431
Fushan Li and Michael Bowling. Gasee-of-teaching and language structure from emergent commu- nication. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Bus Buc, Emily B. Fox, and Roman Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 188255-1835,,,01 2019. URI /
432
435
435
neurips. wwtpppcr/ppacccccccccccccce 2019/ http:///........./////2222222220 www.:aaagstassatttttttaaa. html

---

437
Ryan Lowe, Jakob N. Foerster, Y-Lan Boureau, Joelle Pineau, and Yann N. Dauphin. On the pitfalls of measuring emergent communication. In Edith Elkind, Manuela Veloso, Noa Agmon, and Matthew E. Taylor, editors, Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '19, Montreal, QC, Canada, May 13-17, 2019, pages 693-701. International Foundation for Autonomous Agents and Multiagent Systems, 2019. URL
438
440
442
http: acm. org/ httg/citatioon.. c3m717=3331177
André Martinet. Eléments de linguistique générale. Armand Colin, 1960.
444
George A. Miller. Some effects of intermittent silence. The American Journal of Psychology, 70(2):
445
311-314, 1957. URL http:t//:wwwt jstorr wwtg//tgg/l.aoo sttp///aalleeeeeeeeeeeeeeeeeeeee 1/te//4441466
Igor Mordatch and Pieter Abbeel. Emergence of grounded compositional language in multi-agent populations. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors. Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 1495-1502. AAAI Press, 2018. URL https: / //::::/wwww www. aaai. org/ocs/ owwo/ciix...i/////eeeeeeeemi.i.e iinexxndxx..... php/ /At///AA/AAAA1//PPa
447
449
450
451
452
view 17007
453
Jesse Mu and Noah D. Goodman. Emergent communication of generalizations. In Marc' Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, Advances Neural Information Processing Systems 34: Annual Conference on Neu- ral Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual, pages 17994- 18007 2021. URL https: http://eppceeee..ceeeepp/o peg.ocediingsnnuuipp...... wwtpc/pppp///////eeeeee 9557363e4416335766er/7 http://666666672222244422222226 Abstract html
454
455
457
459
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional languages emerge in a neural iterated learning model. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020.
480
461
462
URL httpp:://ppenreiie// http://ppenereeeeeeeeefrrrrrrnep hwtp:/.nnn/noone////e/e hett://ooruumm.ddHeePeptPhooC
Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. "lazimpa": : Lazy and impatient neural agents learn to communicate efficiently. In Raquel Fernández and Tal Linzen, editors, Proceedings of the 24th Conference on Computational Natural Language Learning, CoNLL 2020, Online, November 19-20, 2020, pages 335-343, Association for Computational Linguistics, 2020. URL https://doet org/ 10. 10186533v12200 conll- 1.11-11.66 226
464
486
468
John Schulman, Nicolas Heess, Theophane Weber, and Pieter Abbeel. Gradient estimation using stochastic computation graphs. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett, editors, Advances in Neural Information Processing Systems
469
470
471
28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, www.rroeeeddnnnnn...e.rpporp neurips Montreal, Quebec, Canada, pages 3528 3536, 2015. -eeabbbdd4Abbbtraae .ttmtttrrt.t.tl..........
472
473
cc/ cc/paper/2011 http///01///hhh////// fee03beffeed9da55f550 htw.://33e9aeeeaaaaaaabaeeeeeee.
Kumiko Tanaka-Ishii. Entropy as an indicator of context boundaries: An experiment using web search engine. In Robert Dale, Kam-Fai Wong, Jian Su, and Oi Yee Kwong, editors, Natural Language Processing- IJCNLP 2005, Second International Joint Conference, Jeju Island, Korea, October 11-13, 2005, Proceedings, volume 3651 of Lecture Notes in Computer Science, pages 93-105. Springer, 2005. /ttp:://://// 10tp///1000000/11221144101
475
476
477
478
479
Kumiko Tanaka-Ishii. Articulation of Elements, pages 115-124. Springer International Publishing, Cham, 2021.
480
org/ 10. 10111100777---300-5500000
481
Kumiko Tanaka-Ishii and Yuichiro Ishii. Multilingual phrase-based concordance genera- tion in real-time. Inf. Reir, 10(3):275-295, 2007.
482
URL https:://oii ott.ffii..rg110.1000.7....
483
s 8079910006-902--
484
Kumiko Tanaka-Ishii and Zhihui Jin. From phoneme to morpheme: Another verification using a corpus. In Yuji Matsumoto, Richard Sproat, Kam-Fai Wong, and Min Zhang, editors, Computer Processing of Oriental Languages. Beyond the Orient: The Research Challenges Ahead, s International Conference, ICCPOL 2006, Singapore, December 17-19, 2006, Proceedings, volume
486
12

---

488
4285 of Lecture Notes in Computer Science, pages 234-244. Springer, 2006.
URL https
489
//doi.rrg/ 10. 100.100090001900....
490
Kumiko Tanaka-Ishii and Zhihui Jin. From phoneme to morpheme - another verification in
english and chinese using corpora- Studia Linguistica, 62(2):224-248, 2008. URL https
432
//doi org/ 1111/j 1467- 9582. 2007000138...... 001000018.... x
498
Ryo Ueda and Koki Washio. On the relationship between zipf's law of abbreviation and interfering noise in emergent languages. In Jad Kabbara, Haitao Lin, Amandalynne Paullada, and Jannis Vamvas, editors, Proceedings of the ACL-IJCNLP 2021 Student Research Workshop, ACL 2021, Online, JUli 5-10, 2021, pages 60-70. Association for Computational Linguistics, 2021. URL ptttps://aclanttologg org/ wttg/202..cclaa acl-alll.
495
486
487
498
Oskar van der Wal, Silvan de Boer, Elia Bruni, and Dieuwke Hupkes. The grammar of emergent languages. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the
499
500
2020 Conference on Empirical Methods in Natural Language Processing. EMNLP 2020, Online, November 16-20, 2020. pages 3339 3359. Association for Computational Linguistics, 2020. URL
501
https://doi. orgo 10. 18653/v1/20000 emnlp- mommmmain.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionis reinforcmmetttt learning. Mach. Learn., 8:229-256, 1992. htt.::ttt////oiior///00000 10tp://00000000/00099900000000
504
505
George K. Zipf. The psycho-biology of language. Houghton Mifflin, 1935.
Checklist
506
507
1. For all authors...
508
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
510
(b) Did you describe the limitations of your work? [Yes] See SeetSon 7
511
(c) Did you discuss any potential negative societal impacts of your work? [No] The main point of this paper is analysis of artificially emerging languages without human participants. It is hard to imagine that it has a direct negative impact on society. However, our paper may have to do with energy issues, since we used GPU. We wrote about the GPU resource in [Section 5.2]
512
513
515
516
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
518
2. If you are including theoretical results...
519
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
521
3. If you ran experiments..
522
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material
URL)? [Yes] Our implementation code is uploaded on Anonymous Github. Seesiicioon 5.2
524
525
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Seesioction 5.1
526
527
(c) Did you report error bars (e-g., with respect to the random seed after running experi- ments multiple times)? [Yes] Error bars are necessary in Figure 5 Figure 6 Figure 7
$29
Figure 8| and Figure and are reported in all of them.
530
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See/Section 5.2)
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
533
(a) If your work uses existing assets, did you cite the creators? [Yes] Our implementation
534
code depends on the EGG toolkit [Kharitonov et al. 2019] See SSction 5.2
535
(b) Did you mention the license of the assets? [Yes] The EGG toolkit is published under
MIT License. See Section 5.2)
13

---

537
(c)
Did you include any new assets either in the supplemental material or as a URL? [Yes] Our implementation code is uploaded on Anonymous Github. SeefSection 5.2
538
539
(d) Did you discuss whether and how consent was obtained from people whose data you' rr using/curating? [No] We only used artificial data for experiments.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [No] We only used artificial data for experiments.
542
543
5. If you used crowdsourcing or conducted research with human subjects...
544
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you deseribe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
547
548
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
519
14


Reviews:
[{"review_id": "bRbC4h8JCUY", "reviewer": "Reviewer_RfVg", "summary": "Harris' hypothesis (or articulation scheme, \"HAS\"; Harris, 1955) suggests that the linguistic boundaries of words can be detected by a count that is unrelated to meaning. However, it is concerned with natural human languages. In this paper, the authors explore if emergent languages between agents share this property with natural languages. \n\nThey do this by using HAS for unsupervised word segmentation, i.e., the segmentation of text (in emergent languages) into smaller units. Then, analyzing the results, they ask: Do emergent languages have meaningful segments?\n\nThey find that emergent languages somewhat follow HAS, but not in all aspects. Similar to natural languages is that: (1) the conditional entropy decreases monotonically and (2) the branching entropy repeatedly falls and rises. However, in contrast to natural languages, HAS-based boundaries in emergent languages aren't always meaningful. Overall, this is a mixed, but nevertheless interesting result.", "questions": "- Do you believe your results are generally applicable to all/most emergent languages? If not, this might be worth clarifying in the relevant passages.\n- Except for the above, where do you see the limitations of your work?", "limitations": "There are no obvious negative societal impacts of this work. The authors do not explicitly address limitations.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This is a very clear paper with an easy-to-understand yet interesting research question. \n- The experiments are solid and suitable to answer the research question.\n\nWeaknesses:\n- The experiments are limited with regards to the settings in which the emergent languages emerge, but the paper is written quite general. This might be worth clarifying.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "OtUPhwoPKCk", "reviewer": "Reviewer_KNRm", "summary": "This paper presents a new paradigm for evaluating the degree to which artificial emergent languages resemble natural languages. They test the predictions of Harris' articulation scheme (HAS) on the messages produced by converged agents trained in a Lewis signaling game, where a speaker agent is trained by reinforcement learning and a listener agent is trained by supervised learning.\nThe authors develop a simple word segmentation algorithm designed on the assumptions of HAS. They also propose several predictions which they would expect to hold of the resulting vocabulary / utterance structures based on the nature of the reference game. They find only weak partial support for these predictions in the resulting emergent languages, and claim that this demonstrates a gap between the structure of emergent language and natural language.", "questions": "1. What is the baseline / foil for these experiments? The authors claim that HAS is a \"statistical universal in natural language,\" but greedy unsupervised segmentation algorithms based on entropy alone are known to be insufficient for word segmentation tasks / lexicon induction tasks. (The authors cite some sub-100% F-scores themselves in L112-114). This means it's unclear what to expect from this procedure even if we were to stumble upon an emergent language with natural language structure. If humans played the same reference game posed to the communicating agents, what would we expect the outputs of the analysis to look like?\n2. What is the intuition behind the ZLA result? This makes sense in systems where the speaker has a length penalty, but not otherwise to me.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Significance: I am not certain of the significance of the weak negative results of the paper. Some ways that I could imagine this paper would be more significant: 1) if there were a more concretely demonstrated gap with human natural language (see question #1); 2) if there were a positive result (e.g. resulting from implementing one of the ideas in L322-324); 3) if there were adequate qualitative examination (e.g. demonstrating that the TopSim positive result is meaningful, or showing the emergent meanings of hypo-words grounded in the reference game, or showing qualitative examples of compositional utterances).\n\nClarity: the paper provides *too much* detail, I think -- specifically, I don't see a reason for all of the figures and results to be repeated across different threshold settings. If there is one threshold setting/range that yields intuitive results across analyses, then it'd be best to present just that. If there isn't, that might suggest deeper problems with the procedure! But it seems like 0 < threshold < 0.5 would be sufficient for all the main results?\n\nQuality+originality: Good! Clear and sensible operationalization of abstract ideas in experimental predictions (though see questions).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NTMM85uhkZj", "reviewer": "Reviewer_CeDT", "summary": "The paper investigates whether HAS also holds in emergent languages, which are languages emerging among artificial agents in a simulated environment. HAS is a universal property in natural languages which says that articulartory boundaries can be obtained from statistical information of phonems without knowing what words mean. The experiments show mixed results---emergent languages satisfy some preconditions for HAS, but the boundaries are not necessarily semantically valid.", "questions": "Do you think such results will extrapolate to emergent languages in settings other than Lewis's signaling game. Are there better ways to quantify how \"meaningful\" the word boundaries may be?", "limitations": "Seems fine to me", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Originality: I am not super familiar with this research area but the idea seems quite original. It also seems like a scientifically interesting research topic to me to find the linguistic differences between natural and emergent languages.\n- Quality: The experiments seem to be well-motivated, and the background is explained with good attention to details. \n- Clarity: The paper writing is quite good.\n- Significance: The paper presents extensive experiments. While the experimental results do not fundamentally change the field of NLP, it is a nice focused contribution.\n\nWeaknesses:\n- It could be even better to explore emergent languages in multiple settings in addition to Lewis's signaling game. \n- Scope of the paper is quite narrow but I do not think this takes away from the merit of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject