Semantics and Spatiality of Emergent Communication
Authors: Rotem Ben Zion
Boaz Carmeli
Orr Paradise
Yonatan Belinkov
Venue: NeurIPS 2024 poster (2024)
Keywords: emergent communication
Lewis' games
Primary area: natural_language_processing

Abstract: When artificial agents are jointly trained to perform collaborative tasks using a communication channel, they develop opaque goal-oriented communication protocols. Good task performance is often considered sufficient evidence that meaningful communication is taking place, but existing empirical results show that communication strategies induced by common objectives can be counterintuitive whilst solving the task nearly perfectly. In this work, we identify a goal-agnostic prerequisite to meaningful communication, which we term semantic consistency, based on the idea that messages should have similar meanings across instances. We provide a formal definition for this idea, and use it to compare the two most common objectives in the field of emergent communication: discrimination and reconstruction. We prove, under mild assumptions, that semantically inconsistent communication protocols can be optimal solutions to the discrimination task, but not to reconstruction. We further show that the reconstruction objective encourages a stricter property, spatial meaningfulness, which also accounts for the distance between messages. Experiments with emergent communication games validate our theoretical results. These findings demonstrate an inherent advantage of distance-based communication goals, and contextualize previous empirical discoveries.

Semantics and Spatiality of Emergent Communication
Rotem Ben Zion'
Boaz Carmeli'
Orr Paradise2
Yonatan Belinkov'
Technion Israel Institute of Technology
UC Berkeley
rotm@campus.c technien.cc...cc ac orrpseecs berkeley edu belinkov@ttechiion. ac iil
boaz. boaz.carmel: carmell@campus.e www.ccaapss...ccniiioooooooo ac.
Abstract
When artificial agents are jointly trained to perform collaborative tasks using communication channel, they develop opaque goal-oriented communication protocols. Good task performance is often considered sufficient evidence that meaningful communication taking place, but existing empirical results show that communication strategies induced by common objectives can be counterintuitive whilst solving the task nearly perfectly. this work, we identify aonn-sic prerequisite to meaningful communication, which we term semantic consistency, based on the idea that messages should have similar meanings across instances. We provide formal definition for this idea, and use it compare the two most common objectives the field of emergent communication: discrimination and reconstruction. We prove, under mild assumptions, that semantically inconsistent communication protocols can be optimal solutions to the discrimination task, but not to reconstruction. We further show that the reconstruction objective encourages a stricter property, spatial meaningfulness, which also accounts for the distance between messages. Experiments with emergent communication games validate our theoretical results. These findings demonstrate an inherent advantage distance- based communication goals, and contextualize previous empirical discoveries.
Introduction
Humans use language in multi-agent social interactions. Pressures of synchronization and collabo- ration play central role in shaping the way we communicate. Motivated by this observation and the goal of creating artificial agents capable of meaningful communication, the field of emergent communication (EC) employs multi-agent environment jointly trained to accomplish a task that requires active transmission of information. The agents utilize messaging channel that usually takes the form of a discrete sequence of abstract symbols, resembling the structure of human language. Successful optimization results in synchronized agents operating newly developed communication protocol tailored the objective. We study a type of EC setup inspired by Lewis games [32 where sender agent describes given input and receiver agent makes prediction based on that description. The game objective is designed to make the receiver demonstrate knowledge of the original input, which in turn compels the sender to encode relevant information in the message. There two common objectives used in this type EC setup (illustrated in Figure T):
Reconstruction In the reconstruction task 34 44 45).
the original input is re-generated based solely on the message. We are specifically interested in reconstruction objective that quantifies distance between prediction and target, forming a discrete version of autoencoding [18
Discrimination In the discrimination task [1 candidates, incentivized by negative log-likeliih likelihood.
28 33
the original input retrieved from a set of
38th Conference on Neural Information Processing Systems (NeurIPS 2024).

---

2
Message
Message
Sender
Receiver
Sender
Receriver
7
2
2
loss =
2_
2
2
loss = -lg D.7
3
(a) Reconstruction game.
(b) Discrimination game.
Illustration of the reconstruction and discrimination tasks. The discrimination receiver is given the candidates in addition to the message.
Figure 1:
central goal in EC, which motivates our work, is understanding how different factors in the environment affect the emergent protocol, and specifically developing agents and objectives that create Tools and experiments have been developed to evaluate the proximity to natural language by testing for properties such as 77. Unfortunately, many of these empirical methods show great dissimilarity to human communication. One particularly surprising experiment [3] revealed that protocols created via the discrimination task can generalize extremely well to random noise data, suggesting that they do not signal human-recognizable (meaningful) properties of the input.
protocols with similar characteristics to natural language 22 25 40 41 45
compositionality (6] or efficiency
In this paper, we identify fundamental property of any meaningful communication protocol, and thus prerequisite for similarity to natural language. We observe that the discrete bottleneck forces EC protocols be many-to-one mappings, i.e.. that messages likely represent more than one input. With this in mind, we claim that inputs mapped to the same message should be semantically similar, as is the case with human language. We formalize this idea with mathematical definition that we term semantic consistency. We further develop a stricter version of this definition, spatial meaningfulness, which also accounts for distances between messages, and therefore better suited to indicate similarity to natural language.
Armed with these definitions, we analyze theoretical solutions the two common EC environments. In the reconstruction setting, under mild assumptions, we prove that every optimal solution is semantically consistent. With different set of assumptions, we also show that reconstruction- induced communication protocols are spatially meaningful. In sharp contrast, we surprisingly find that the discrimination objective does not guarantee semantic consistency, i.e., communication protocol can be optimal in a discrimination environment but still not semantically consistent nor spatially meaningful. In fact, we show that uniformly random messages can lead globally optimal discrimination objective value, meaning that the relationship between inputs mapped to the same message potentially arbitrary despite optimally solving the task. Our results provide theoretical support to previous empirical findings, such as the discrimination generalization to noise. We further analyze several common variations of the discrimination game from the EC literature.
To support our findings, we run experiments Shapes [27 and MNIST 30 The empirical results agree with most of our theoretical findings. However, observe gap between theory and practice regarding the level of channel utilization, which we leave for future work to further investigate.
Background and related work
2.1 Emergent communication with Lewis games
large portion EC research, based on Lewis signaling games [28] 32], defines two-agent coop- erative tasks where one agent, Sg ("sender"), receives an input X' ("target") and generates message m E M, and the second agent, R. ("receiver"), takes that message and outputs a prediction. The two most common such tasks reconstruction and discrimination.
In the reconstruction task, the receiver tries to predict the target directly from the message. This generative method, so it can naturally be modeled with receiver that outputs distribution over the input space 7
i.e., R2 M -> A (x)). this case, the objective usually negative log- likeeeilloood or accuracy. However, this approach has two major flaws: First, explicitly outputting a distribution is not always feasible, e.g., over images. Thus, this game is mostly implemented on simple synthetic datasets. Second, the incentive of likelihood or accuracy does not measure how

---

close the receiver's output is to In this work, we assume a different approach to reconstruction. inspired by autoencoders [8 34 where the receiver outputs an object in the input space rather than a distribution, i.e., R₂ M + X. Thus, the loss function in our analysis is straightforwardly the distance between prediction and target.
In the discrimination task, the receiver tries to identify the target within a set of candidates [17 28 It can be modeled as R.p M X + A.,,..., d3. We explore some alternative formulations in Appendix F) This setting introduces two major choices: the number of distractors (candidates other than x) and how to select them I2 29 These parameters have been shown to
29 33 44
have a major effect on the resulting protocols [T5 29 The results in the main body of this paper concern the vanilla version of the discrimination task, where candidates are chosen independently. We analyze other discrimination versions in Appendix A
2.2 Towards interpretable human-lite communication
A central line of study aims to develop EC systems that create protocols with high proximity natural language. Some have tried aligning the two explicitly, by using pretrained language models [1 43 or training translation systems 45 46). In this work, we focus on properties of the EC protocol itself, as induced by the objective rather than exposure to text. The most studied such characteristic is compositionality, which refers the ability of agents to assemble complex units (e.g., sentences) from simpler units (e.g., words). Compositionality is hard to measure 5
6 24 37], and while some claim that is correlated with generalization [2 40 42, the opposite has also been argued R [ [3]. In any case, emergent languages are often not compositional [25). inefficient 41) and generalize to noise [3, suggesting a mismatch between common EC objectives and the evolution of human language. In this work, we advance the goal of creating human-like communication by interpreting properties of natural language into formal constraints, and analyzing whether the EC objectives create protocols that follow them. A commonly used compositionality measure, topographic similarity (topsim) [5], is related to the definitions presented in this work. It evaluates the correlation between distances in the input space and the corresponding distances in the messages space. Notably, topsim considers the relationship between every pair of inputs, whereas our definitions only consider pairs that correspond to similar messages. The latter follows an intuitive asymmetry: inputs with similar messages are expected to be similar, but inputs with dissimilar messages do not have to be different (for example, the same image often be described by several distinct human captions).
2.3 Input information encoded in the message
The game objective dictates the information that needs to be stored in messages. In this work, we focus on spatial information by considering the geometric relationship between inputs mapped to the same message. To the best of our knowledge, no previous work has theoretically analyzed this idea in EC literature. A related concept is the mutual information (MI) between messages and inputs, which
MI is strongly related to the infoNCE objective (38) often used in implementations of the discrimination game. MI can be used to measure the complexity of the communication channel [44 45), and is correlated with task performance [23 45 and even compositionality [.]. Finally, The mutual information gap [9] [8] [3], which is also related to compositionality. However, Guo et al. [15] show that MI is not sufficient to indicate transfer- learning generalization, and connect that to pairwise distances of inputs mapped to the same message, effectively testing for semantic consistency. The first part of this work, combined with their findings, would suggest that languages developed by agents in the reconstruction setting should be more expressive and generalize better to different tasks.
has been studied both empirically [ 23 35| 45) and theoretically [9 21 42).
can be used to measure disentanglement
On the theoretical side, a relevant work is Rita et al. 42 where the global discrimination objective is decomposed into two interpretable components, a co-adaptation loss and MI. The latter is equivalent to our Lemma A.I In this work, we use such equivalent objectives for both the reconstruction and discrimination tasks to investigate the behaviour of optimal solutions, and specifically whether the tasks encourage semantic consistency.
'We refer to this game as "global discrimination" and analyze it in Appendix A. When the input space is finite, it is equivalent special case of the discrimination game.
3

---

Input distribution
X ~ fx(x), E &'
EC setup
Message space
Agent parameters Objective
sender O, receiver +
M
Training
Optimal solutions
0 q
600, x))
Figure 2: Notation for the emergent communication (EC) setup.
Notation and task definitions
3.1 General notation
We begin by introducing general notation for the EC setup; see Figure 2 for illustration. The input space modeled by a real-valued random variable X (X, where X' S R is the bounded set of possible inputs (e.g., images), and fx: X' + R4 is prior distribution. Denote by M = {m,, m2,...... ....................... C Rad the finite or countably infinite set of possible messages We assume that the minimal distance between pair of messages is attained, i.e., that min ||
m2 l| (0, oo).
The sender agent Sy: X M, parameterized by 0 E θ, maps each imput to a message. Sender So defines the communication protocol; we compare EC setups based on which optimal sender agents they induce. The receiver agent Rgg. maps the message and optionally additional input to a prediction. It is parameterized by Φ. During EC training, the goal is to learn hypotheses (0, sp) that minimize a loss described by a function é: value. To group all of the above notation into single definition, denote EC setup as a sextuple fx, O, E). An EC setup induces a set optimal agents by
+ IR, which maps a pair of agents and an input to a real
0", 1p" E argmin E E(0, (00,6.2). 2).
0EO x
Note that every part of the setup can affect the set of optimal agents. We also define notion of conditional optimality, where one of the agents is fixed at potentially sub-optimal hypothesis; sender Sa is synchronized with receiver Ryp if the pair is optimal for the EC setup (X' Jx. M, θ, {P), E). The same definition applies the receiver agent respectively. Note that pair of agents can be synchronized in both directions without being optimal.
3.2 Task definitions
In the reconstruction setting, the receiver agent predicts the original input (the "target") given only the message. In the discrimination setting. the receiver is given the message and set of candidate inputs containing the target in a random position, and attempts to predict that position based the message. A visual illustration can be seen in Figure I
We now present formal definitions of these setups.
The receiver maps a message Sg (z) to prediction in the input space. The loss the Euclidean distance between that prediction and the target
Reconstruction.
M,X T reconstruction is defined as the set of all functions the form R: M - X.
reconsttuucton (0, p, x) = [R (Sa (Sa(()) (æ)) x11².
Definition An EC setup (x, M, θ Φ, E) is a reconstruction game if $ C T M,X recomstruction and Prscostraction
fecconstractonn It has an unrestricted receiver hypothesis class if M.X
Discrimination. In addition message Sa(x), the receiver sees set of candidates {21 1......, Ta). which contains the target at a random position t, i.e. I₁ = x, and the rest are d - independently sampled distractors. The receiver outputs probability distribution over the candidates. The loss is the negative log-likelihood of the correct position according to this distribution, averaged over the target position and distractors.
"The description M a vector space flexible. As example instantiation, classic communication channel where each message a sequence of symbols from a fixed vocabulary can interpreted as a concate- nation of one-hot vectors. In that case, we would have das = V L where V is the vocabulary size and L the message length. The Euclidean distance between messages becomes their Hamming distance.

---

T
M,X.d discrimination is defined as the set of all functions of the form R: M x X - ... d}.
x.d discrimination (0, 9, x) =
E
log P(R (Sa 21 T.) t).
Definition 2.
An EC setup (X, M
is a d-candidates discrimination game if $ S It has an unrestricted receiver hypothesis class if & M. discrimination
&.M..d discrimination and gx.d discrimination"
In this vanilla version of the discrimination game, the candidates are sampled independently. Several other choice mechanisms have been explored in EC literature. In Appendix A we define and analyze the most common such variations:
Global discrimination In Appendixx A. we analyze a version of the discrimination game where the receiver outputs a distribution over the entire data rather than small set of candidates. We find that optimal communication protocols in this setting maximize mutual information between the inputs and messages, as shown Rita et 42
Supervised discrimination Appendix A.2 explores variant of the discrimination game that in- corporates labels by selecting distractors with labels different from the target. We find that optimal communication strategies in this setting both diverse (the sender encouraged to output messages uniformly) and pure (labels distribute with low entropy given a message).
Classification discrimination In Appendix A.3] we consider a format of the discrimination game where the target is excluded from the candidate set, requiring the receiver to identify candidate that matches the target's ' label. We find that optimal solutions in this setup maximize mutual information between messages and labels.
4 Semantic consistency: prerequisite to meaningful communication
In many EC environments, the message is a latent representation of the input, and is often directly analogous to the corresponding latent vector from continuous training setups (c.g. autoencoding, contrastive learning). That said, the discrete bottleneck in EC offers an additional perspective the concept of message; when the message space is smaller than the input space, the communication protocol is forced be a many-to-one to-one mapping, meaning that each message represents set of inputs.
We consider inputs mapped to the same message as equivalent with respect to the communication protocol, thus defining a of equivalence classes that partitions the input space. With this perspective in mind, we wish to understand whether this partition signals meaningful properties of the inputs. In other words, we ask the question: Is there a meaningful relationship between inputs mapped to the same message? This motivation is illustrated in Figure
mi
m2
m1
m3
m3
(a) Input space.
(c)
Semantically consistent mapping. Semantically inconsistent mapping. Figure 3: message describes set of inputs. Note: the shapes and colors are not part of the input.
The semantics of message can be interpreted as the set of properties shared across all inputs in its equivalence class. We would like to design a theoretical test that evaluates whether meaningful properties are encoded this way in an emergent communication protocol. We term this notion semuntic consistency, based on the idea that inputs with shared properties are more semantically similar, as illustrated in Figure 3] We use the same idea develop formal definition: We define communication protocol to be semantically consistent if random pair of inputs mapped to the
5

---

same message is, in expectation. more similar than two completely random inputs. Formally, for communication protocol Sg. consider the following inequality:
| - x211? Se(z1) = S0
IE
|| - 22 221l?]
(I)
Equation (1) is simplified in Appendix C.1 into the following formal definition:
Definition 3. A communication protocol Sg semantically consistent if
VVr [X Sa(X) = ml] < Var [X].
(X)
The difference between the two expressions Definition 3 is the explained variance, given by Varm=~S((X) [ [X] (x) = ml]. Thus, communication protocol is semantically consistent if explains some of the variance X.
Remark. the message space is larger than the input space, a sender could map each input to unique message and lose no information. Any such lossless protocol is trivially semantically consistent.
5 Are EC systems semantically consistent?
Having defined the two common EC environments in Section 3.2 now apply Definition 3 to assess whether semantically consistent protocols are induced by reconstruction and discrimination tasks. Formally, answer the following question for each of the tasks:
every optimal emergent protocol semantically consistent?
Under the assumption of an unrestricted receiver hypothesis class, we find that:
In the reconstruction game, the answer is yes. We show that equivalent inputs in reconstruction- induced communication protocols are clustered together in input space, matching Figure 3b)
In the discrimination game, the answer is no. The relationship between equivalent inputs in iettiiiinationinnducee protocol can be arbitrary, matching Figure 3c
All proofs are given in Appendix
5.1 Reconstruction results
Idcally, would like examine closed-form set of optimal communication protocols. Unfortu- nately, there is no closed-fon solution to the set of optimal agent pairs in a general reconstruction setting. That said, we do have such a solution to the synchronized receiver, i.c., the optimal receiver for a fixed sender S (the following arg min set contains single item):
R" (m) arg min E rEN
[|]r 211 S(x) m| = E E S(X) = m] x~X
By assuming that unrestricted, we can plug the closed-form synchronized receiver back into the loss function. This yields transformed objective given in the following lemma.
Lemma 5.1. [proof in page [6] Let fx, M, O, E) be reconstruction game. Assuming & is unrestricted, sender Sg is oplimal if and only if it minimizes the following objective:
Σ P(So(X) =m)) Var [X Sx(X) = m)]
m)
MEM
This equivalent objective clearly shows the connection between inputs mapped to the same message: In every optimal solution, these inputs will have high proximity. (In fact, this is the objective function of k-means clustering [36]. as elaborated (pppendix B) Moreover, this formula is the unexplained variance, so minimizing it will maximize the explained variance, leading to the following theorem.
Theorem 5.2. [proof in page 17 Lei (X, //xx,MM M, θ $ E) be a reconstruction game. Assuming & is unrestricted and @ contains at least semantically consistent protocol, every optimal communica- tion protocol is semantically consistent.
We assume that Euclidean distance in the input space entails semantic information. In some types of data, like image pixels, using pretrained embeddings may improve the validity of that assumption.

---

5.2 Discrimination results
In similar fashion to the reconstruction setting, we have a closed form solution to the synchronized discrimination receiver
Rt (m, 21,...,...................... = -P (t 21s.... Zd. m)
Explicitly (see Lemma
5.3 proof):
P P (m, . ra) = =) Kz 2a) (m- 1{5(x;) = m}
Applying this receiver, we get the following discrimination loss
Lemma 5.3. [proof in page Let (x, ; fx. M, O 2, be a d-candidates discrimination game. Assuming & is unrestricted, a sender Se is optimal if and only if it minimizes the following objective:
Σ ((Sa(X)= m) m)) MEM
log (1+ Binomial dd P(So(X) =m))
And when = (a single-distracte game). this simplifies into: ΣmEM P(So(X) = =m)²
This equivalent objective reveals the interesting disparity between reconstruction and discrimination. Note that this objective is solely function of the 'sizes' of equivalence classes (the probability for each message). Thus, while the above formula incentivizes the sender distribute the messages uniformly, it does not impose any constraint on their content; the connection between inputs mapped the same message could be arbitrary. In formal terms, prove the following corollary:
Corollary 5.4. [proof in page 19 If the of possible messages is finite, any sender agent that assigns them such that their distribution is uniform can be paired with some receiver to obtain a globally optimal loss for the discrimination game.
Using this corollary, we prove that optimal solutions can be inconsistent in the discrimination setting:
Theorem 5.5. (proof in page 20 There exists a discrimination game (X, fx. , M, & E) where O is unrestricted and & contains at least semantically consistent protocol, in which not all of the optimal communication protocols are semantically consistent.
6 A missing dimension: spatiality in the message space
While the notion of semantic consistency successfully differentiates between the two EC frameworks, it has major shortcoming: is does not take into account distances between messages. Spatiality in the message space is integral to concepts like compositionality (5) and case of teaching [33]. A hypothetical communication protocol that maps similar messages to very different meanings may satisfy Definition 3] but is fundamentally diffcrent from natural language. We now present a stricter version of semantic consistency, spatial meaningfulness, which does consider distances in the message space, and therefore better indicates an inherent similarity to natural language. In addition, eliminate the assumption that the receiver's 's hypothesis class is unrestricted, in favor of simplicity and non-degeneracy conditions, which are more realistic in the context of natural language.
To introduce distance between messages, we replace the message equality with similarity in Equa fion (1) Let Eg denote a threshold under which messages are considered similar. To avoid reverting back to Definition 3] require that Eg be greater than equal to EM. defined the minimal distance between messages: EM := min || [m - m21
mizmze
Definition 4. For so 2 EM. communication protocol Sg Eo-spatially meaningful v0 S Eo
E #1.22~X
[|
121 [1))(z1) So(2))
E
[[|=1
22
a1 -1-22~x
A communication protocol Sp is spatially meaningful if this definition holds for Eo = EM-
'Negative log-likel log-likelihood is proper scoring rule; see www Lemma 5.3 proof.
'The expectation is taken with respect the binomial distribution.
This is well defined due to our assumption in Section

---

Note that this definition is stricter than semantic consistency. Informally, Definitio Infiitonly,Diiiitiion 4requires the protocol be semantically consistent when combining pairs of close messages.
With this new definition, we are able to reach similar results to the original semantic consistency:
In the reconstruction game, for any receiver that satisfies conditions, we show that every synchronized sender is spatially meaningful.
In the discrimination game, given the same conditions, we show that a synchronized sender is not necessarily semantically consistent (and hence nor spatially meaningful).
We now define two conditions on the receiver.
Simplicity constraint. The first condition limits the receiver's complexity, and introduces the effect of distance between messages. We do that by bounding the rate of change in receiver's s output, similarly to Lipschitz constant. The bounding term depends on the variance of the input distribution and the chosen distance threshold. Formally:
Definition 5. Receiver Rsp is (X, M, -simle if Vr 22 E domain Rp):
V2
[R((11) (22(1)) S k || x2/1,
where k:
Var
250
Non-degeneracy constraint. The second condition prevents the receiver from being too weak. After applying the first condition, no longer have closed-form solution for the synchronized receiver. Instead, we take a non-optimal receiver, but add condition to ensure it is strictly better than any constant function. Formally, let & be receiver that always outputs C, and let C" arg min E E(O, x~x
"(0,s0 Sp, ) be the optimal such constant. (Note the sender does not matter here.)
Definition 6. Receiver Rip is non-degenerate if
sup E(O", 9,, x) S 1 E E(0, (0, xt) x>x
=EX
for every sender 0* that is synchronized with Rig
6.1 Results
With these definitions, we have the following theorems:
Theorem 6.1. Iproof in page Let fx, O 2, E) be reconstruction game, let EM and let & such that Rsg is (X, M,..) -oi-simple and non-degenerate. Every sender that is synchronized with R.g is Eu-spiiillyy meaningful.
Theorem 6.2. [proof in page 22 There exists a discrimination game (X. fx. M, M, EM and a receiver & & which is (X, M, Eu) s.)--ussmmlee and non-degenerate, where a synchronized sender matching R.p is not e-spatially meaningful for any e.
Experiments
To support our theoretical findings, we run experiments on two datasets: (i) the MNIST dataset
30 contains images of single hand-written digit; (ii) the Shapes dataset (27 contains images of
an object with random shape, color and position. We train reconstruction and discrimination (40 distractors) agents using communication channel of vocabulary size 10 and message length (no EOS token), optimized with Gumbel-Softmax 22]. In the MNIST experiments, we use the digit labels to train another set of agents with the supervised discrimination task (40 distractors). Further data and training details are found in Appendix D.I
7.1 Semantic consistency, compositionality and task performance
To evaluate semantic consistency empirically, we develop the empirical message variance measure (in Appendix D.2) which calculates the empirical variance of each equivalence class and takes
Our code is available at uulttps://githuu com/Rotem http:://..ffftaaooooooooootttcta

---

weighted average. We measure task performance using discrimination accuracy over 41 candidates (40 distractors), where the referential prediction of reconstruction -tranner agents defined as the closest candidate to the reconstructed target. We also report Topographic Similarity [5] the most common compositionality measure from literature. See (ppenddix E. for results with other compositionality measures (PosDis [8]. BosDis [ and Speaker-PosDis 13 Additional information on each metric is given in Appendixx D.2)
Table 1 and Table 2 show results over the Shapes and MNIST validation sets, respectively (showing standard deviation). While all setups achieve good task performance over MNIST, the discrimination game yields higher discrimination accuracy over the more difficult Shapes dataset, As expected. Interestingly, the discrimination trainneddtdddddddeeeon agents utilize less of the communication channel's capacity, as shown by the number of unique messages in both tables. This presents a gap from the theoretical analysis, as the discrimination setting should benefit from maximal message diversity (this can be seen in Lemma 5.3). To enable a fair comparison, we establish individual random baselines that maintain the number of inputs mapped to each message but randomize their assignment. Thus, the baseline's message variance shows the level of message diversity induced by the task, whereas the improvement over the baseline indicates semantic consistency. Over both datasets, the reconstruction task induces significantly more semantically consistent protocols both in absolute value and improvement over the baseline, as predicted by our theoretical findings. Furthermore, TopSim strongly favors the reconstruction setting, suggesting connection between semantic consistency and compositionality. We note that the difference measured by TopSim is more significant over the Shapes dataset, which is itself more compositional, as objects have several attributes. These results, and the correlations reported in [Appendix F. indicate little to no correlation between the evaluated compositionality and the discrimination objective performance. In fact, within the set of reconstruction runs Shapes, their correlation is negative (-0.24). This illustrates the counter- intuir--n nature of this objective, which we discuss in Section 5.2
Table 1: Empirical results on Shapes, averaged over five randomly initialized training runs.
Msg Var 1
ECC setup
Uniqe Msgs
Disc. accuracy f
TopSim f
Trained
Rand
Reconstruction
306.60 1:28.52 52
31.64 12.51
0.34 10.02
1334.38 178.05
2554.77 1108.19
Discrimination
251.60 +-29.33
61.96 +4.78
0.09 +0.01
2280.24 157.38
2793.65 +115.45
Table 2: Empirical results on MNIST, averaged over three randomly initialized training runs.
Msg Var 4
EC setup
Uniqe Msgs
Disc. accuracy f
TopSim f
Trained
Rand
Reconstruction
2523.66 +:3000
88.00 -0.6
0.365 +0.042
371.63 12.1
1029.57 113.
Discrimination
402.00 170.4
78.76 1104
0.360 10.036
1226. 14 144.4
1784.59 124.3
Supervised disc.
287.33 +28.5
87.10 #5.4
0.269 -0.044
1381.72 +41.6
1821.53 +12.2
7.2 Message purity
Since the MNIST dataset has only the digit label, we expect meaningful communication protocols to signal digits in their messages. To evaluate this, we calculate the percentage of images that have the majority digit label of their equivalence class. This can be interpreted as the highest attainable accuracy of classifier mapping messages to digits. Figure 4 shows the most significant improvement over the baseline in the supervised discrimination game, as expected by Lemma A.2 The reconstruction task induces nearly perfect digit description protocol as well, despite being unsupervised, providing evidence of its ability to unveil meaningful properties, in agreement with our findings. On the other hand, message purity is much lower in the unsupervised discrimination setting, again showcasing that task success does not guarantee an intuitive communication strategy.
In Appendix E.2 we perform a similar analysis over the Shapes dataset, by evaluating message purity with respect to each attribute individually or to a maximum aggregation. We find similar but weaker results.

---

100
D Trained Random
80
Purity
60
40
20
Reconstruction
Discrimination
Supervised disc.
Figure 4: Average message purity, comparing trained models to random baselines.
7.3 Spatial meaningfulness analysis
We develop evaluation method for spatial meaningfulness, which term cluster variance, based on the message variance measure. While we do not find a decisive result using this method, we suggest several ideas for further investigation; see (ppeendii E.3 for the full analysis.
Limitations
Our analysis compares the two frameworks via their optimal solutions, and often assumes unrestricted hypothesis classes. In reality, the complexity of the agents is limited by the chosen architectures. Optimization-erron affects the emergent protocol well, especially when it is discrete, as regular differentiable training not possible. Future work may further analyze these assumptions from theoretical side, or evaluate their effect empirically. This limitation is evident in empirical results (Section 7, where channel utilization in the discrimination game behaves differently than expected.
Our main results regarding the discrimination game deal with simplistic version, where candidates are chosen independently. While we analyze some common variations the appendix, many other ideas have been proposed and shown to have significant benefits, including multiple-target games
6 37 different-view candidate representations 10 16] and multi-modality
14
Our results assume, as illustrated in Figure 3, that proximity in the input space entails semantic information, and specifically that Euclidean distance broadly indicates the level of semantic similarity. Future work may investigate other distance metrics or evaluate the validity of this assumption.
Conclusion
Based on properties of natural language, we have defined notions of semantic consistency and spatial meaningfulness, meant to evaluate meaningfulness in emergent communication protocols. Using these definitions, we found that communication protocols generated to solve the reconstruction objective have messages with inherent meaning, while the discrimination objective can be solved by unintuitive systems. Our findings provide insight into known empirical results in EC, such as the ability of agents in the discrimination game perform well unseen random noise. The theoretical tools that we have proposed can be used for future investigation and design of EC environments. As main takeaway, we conelude that distance-based EC environments have promising potential in the prospect of inducing characteristics of natural language, whereas probability- objectives must be more carefully designed.
10

---

Acknowledgements
This research was supported by grant no. 2022330 from the United States Israel Binational Science Foundation (BSF), Jerusalem, Israel, the Israel Science Foundation (grant no. 448/20), an Azrieli Foundation Early Career Faculty Fellowship, and AI Alignment grant from Open Philanthropy. OP was funded by Project CETI via grants from Dalio Philanthropies and Ocean X; Sea Grape Foundation; Virgin Unite and Rosamund Zander/Hansjorg Wyss through The Audacious Project: collaborative funding initiative housed at TED.
References
Jacob Andreas. Measuring compositionality in representation learning. In International Conference on Learning Representations, 2018.
[2] Michal Auersperger and Pavel Pecina. Defending compositionality in emergent languages. NAACL 2022, page 285, 2022.
[3] Diane Bouchacourt and Marco Baroni. How agents things: On visual representations in emergent language game. Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pages 981-985, 2018.
[4] Nicolo' Brandizzi. Toward more human-like communication: review emergent communication research, IEEE Access, 11:142317-142340, 2023.
[5] Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial life. 12(2):229-422, 2006.
[6] Boaz Carmeli, Yonatan Belinkow, Ron Meir. Concept-best-matching Evaluating compositionality in emergent communication. arXiv preprint arOiv: 1477 2024.
[7] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems, 32, 2019.
[8] Rahma Chaabouni, Fugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and generalization in emergent languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4427-4442, 2020.
[9] Ricky Chen, Xuechen Li, Roger B Grosse, and David K Duvenaud. Isolating sources of disentanglement in variational autoencoders. Advances in neural information processing systems, 31, 2018.
[10] Edward Choi, Angeliki Lazaridou, and Nando De Freitas. Compositional obverter communication lcarning from visual input. In 6th International Conference on Learning Representations, ICLR 2018.
International Conference on Learning Representations, ICLR, 2018.
[11] Kevin Denamganai and James Alfred Walker. On (emergent) systematic generalisation and composi- tionality in visual referential games with straight gumbel-softmax estimator. arXiv preprint arXiv:2012, 10776, 2020.
[12] Kevin Denamganal and James Alfred Walker. Referentialgym: nomenclature and framework for language emergence grounding in (visual) referential games. arXiv preprint arXhv:2012.09486, 2020.
[13] Kevin Denamganai, Sondess Missaoui, and James Alfred Walker. Visual referential games further the emergence of disentangled representations. arXiv preprint arXiv:2304. 14511, 2023.
[14] Katrina Evtimova, Andrew Drozdov, Douwe Kiela, Kyunghyun Cho. Emergent communication a multi-modal, multi-step referential game. International Conference on Learning Representarions, 2018.
[15] Shangmin Guo, Ren, Kory Wallace Mathewson, Simon Kirby, Stefano Albrecht, and Kenny Smith. Expressivity of emergent languages is a trade-off between contextual complexity and unpredictability. In International Conference on Learning Representations, 2021.
[16] Kamal Gupta, Gowthami Somepalli, Anubhav Anubhav, Vinoj Yasanga Jayasundara Magalle Hewa, Matthias Zwicker, and Abhinay Shrivastava. Patchgame: Learning to signal mid-level patches in referential games. Advances in Neural Information Processing Systems, 34:26015-26022 2021.
[17] Serhii Havrylov and Ivan Titov. Emergence of language with multi- agent games: Learning to communicate with sequences of symbols. Advances in neural information processing systems, 30, 2017.
11

---

[18] Geoffrey E Hinton and Ruslan R Salakhutdinov. Reducing the dimensionality of data with neural networks. science, 313(5786):504-577, 2006.
[19] R Devon Hjelm, Alex Fedorov, Samuel Marchilna Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. Learning deep representations by mutual information estimation and maximization. In International Conference Learning Representations, 2018.
[20] Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparametrization with gumble-softmat In Interna- tional Conference Learning Representations (ICLR 2017). OpenReview. net, 2017.
[21] Natasha Jaques, Angeliki Lazaridou, Edward Hughes, Caglar Gulcehre, Pedro Ortega, DJ Strouse, Joel Z Leibo, and Nando De Freitas. Social influence intrinsic motivation for multi-agent deep reinforcement learning. In International conference on machine learning, pages 3040-3049. PMLR, 2019.
[22] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Egg: toolkit for rescarch emergence of language in games. EMNLP-IJCNLP 2019, page 55, 2019.
[23] Eugene Kharitonov, Rahma Chaabouni, Diane Bouchacourt, and Marco Baroni. Entropy minimization in emergent languages. In International Conference on Machine Learning, pages 5220-5230 PMLR, 2020.
[24] Tomasz Korbak, Julian Zubek, and Joanna Raczaszek-Loonardi Measuring non-trivial compositionality in emergent communication. arXiv preprint arXiv: 1505, 2020.
[25] Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally' in multi-agent dialog. Proceedings the Processing, pages 2962222222222 2967, 2017.
the 2017 Conference on Empirical Methods in Natural Language
[26] Harold W Kuhn and Alhert W Tucker. Nonlinear programming. In Traces and emergence of nonlinear programming, pages 247-258. Springer, 2013.
[27] Alexander Kuhnle Ann Copestake. Shapeworld new test methodology for multimodal language understanding. arXiv preprint arXiv:1704.0455 1704.04517, 2017.
[28] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence (natural) language. International Conference on Learning Representations, 2016.
[29] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Conference on Learning Reprexentations 2018
[30] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- based learning applied to document recognition. Proceedings of the IEEE, 86(11):227) 2324, 1998.
[31] Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing /EMNLP-IJCNLPP pages 4385 4395, 2019.
[32] David Lewis. Convention: A philosophical study. John Wiley Sons, 2008
[33] Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent communication. Advances in neural information processing systems, 32, 2019.
[34] Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning to ground multi-agent communication with autoencoders. Advances in Neural Information Processing Systems, 34: 15230-15522, 2021.
[35] Ryan Lowe, Jakob Foerster, Y-Lan Boureau, Joelle Pincau, and Yann Dauphin. On the pitfalls of measuring emergent communication. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 693-701, 2019.
[36] JB MACQUEEN. Some methods for classification and analysis of multivariate observations. In Proceedings of the 5th Berkeley Symposium on Mathematical Statistics & Probubility, volume 1, pages 281-297. University of California Press, 1967.
[37] Jesse Mu and Noah Goodman. Emergent communication of generalizations. Advances in Neural Informa- tion Processing Systems, 34:17994-18800 18007, 2021.
[38] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arxiv:1807.03748. 2018.
12

---

[39] Gustav Grund Pihigren, Fredrik Sandin, and Marcus Liwicki. Improving image autoencoder embeddings with perceptual loss. In 2020 International Joint Conference Neural Networks (IJCNN), pages 1-7. IEEE, 2020.
[40] Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B Cohen, and Simon Kirby. Compositional languages emerge in neural iterated learning model. 8th International Conference on Learning Representations. 2020.
[41] Mathieu Rita, Rahma Chaabouni, and Emmanuel Dupoux. "lazimpa": Lazy and impatient neural agents learn to communicate efficiently. In CONLL 2020-The SIGNLL Conference on Computational Natural Language Learning, 2020.
[42] Mathieu Rita. Corentin Tallec, Paul Michel, Jean-Bastien Grill, Olivier Pietquin, Emmanuel Dupoux, and Florian Strub. Emergent communication: Generalization and overfitting in lewis games. Advances in Neural Information Processing Systems, 35:1389-188 1404, 2022.
[43] Shane Steinert- Threekeldd Xuhui Zhou, Zeyu Liu, and CM Downey, Emergent communication fine- fine-tuning (ec-ft) for pretrained language models. In Emergent Communication Workshop at ICLR 2022, 2022.
[44] Mycal Tucker, Roger Levy, Julie A Shah, and Noga Zaslavsky. Trading off utility, informativeness, and complexity in emergent communication. Advances in neural information processing systems, 35: 22214 22228, 2022.
[45] Mycal Tucker, Roger Levy, Julie Shah, and Noga Zaslavsky. Generalization and translatability in emergent communication via informational constraints. NeurlPS 2022 Workshop Information- Theoretic Principles in Cognitive Systems, 2022.
[46] Shunyu Yao, Yu, Yang Zhang. Karthik Narasimhan. Joshua Tenenbaum, Chuang Gan. Linking emergent and natural languages via corpus transfer. International Conference Learning Representa- tions, 2022.
13

---

A Analysis of discrimination variations
In this section, we extend our analysis of the discrimination game to three common variations from EC literature.
A.1 Global discrimination
In this variation, rather than sampling small set of candidates, the receiver must discriminate between the entire of data points, which does not have be finite nor countable. In related work this game is often referred to the reconstruction game generative task. However, since it uses the negative log-likelihowe objective and not a distance-based one, call it global discrimination. Note that when the set of data inputs X' is finite, this is similar to a special case of the regular discrimination game, where the number candidates is set to | (and candidates sampled without replacement). This equivalence is also mentioned by [[2]. Denote:
42 44 as it can be interpreted as
o M,X' to be the set of all functions of the form R: M A(X).
Eglobal (0, (p, == log (R x)
Definition 7. An EC setup (X, fx M, θ Φ E) is a global discrimination game if & C M. global and
= fglohal.
Lemma A.1. [proof page 24) Let fx. M, O D, E) be a global discrimination game, Assuming ① is unrestricted, sender Sa is optimal if and only if it minimizes the following objective:
[[(; Sx (X))
We that optimal protocols maximize mutual information between inputs and messages. This corresponds to popular idea in representation learning literature [19 38) However, similarly to the objective shown in Lemma 5.3) for the regular discrimination game, this communication goal does consider distances between inputs. fact, mutual information can be calculated for non-numerical
input spaces.
A.2 Supervised discrimination
In this variation, labels are incorporated into the game via the candidate choice mechanism. The distractors (candidates that not the target) are guaranteed to not have the same label as the target. This means that the receiver will never have to tell apart two inputs from the same class.
Let y be finite set of labels, and let label X' - y be (deterministic) function mapping input to label. Denote the distribution over labels Y label (X). We assume that this random variable has uniform distribution, i.c., that the classes balanced. The supervised variation the same receiver functions as in Definition 2 and a slightly different loss function:
supervised 4p, x)
log (S Fa) =)
6- F.=F P(X #lbbl((=))
Definition 8. For < 1/1 EC setup (X. fx, M, θ o, E) is d-candidates supervised discrimina-
tion game if + C z. M.X.d discrimination and "supervised"
Lemma A.2. [proof in page[4) Let (x, fx.xxxMM. M, e, $ be a 2-candidates supervised discrimination game. Assuming & is unrestricted, sender Sn is optimal if and only if minimizes the following objective:
Σ P(Sa(X)= = m)² MEM
Σ ΣΣ MEM yEy
P(So(X) m, mm,labe(XX) =
This result is interpretable too. The first expression is the unsupervised objective that encourages uniformity message probabilities, which we encountered in[Lemma 5.3, The second expression encourages non non-diversity of labels within each equivalence class (optimal if all inputs mapped to a message share the same label). This means that optimal messages would have high mutual information with the labels, which in turn could introduce semantic consistency. However, within each class, the communication protocol would not necessarily be able distinguish between inputs; perfect accuracy can be achieved with just | unique messages.
14

---

This result presents another key implication, regarding complexity in the discrimination game [15] We can consider candidate choice mechanism that makes sure all distractors have the same label as the target (the opposite of the classic supervised game). This task will require the receiver to tell apart similar inputs, resulting in high contextual complexity. Interestingly, this task incentivizes the sender to map same-label inputs to different messages, in sharp contrast from Lemma A.2] Therefore, messages learned to solve this high-complexity task will have low mutual information with the labels, i.e., this game is likely to result in counter-intuitive communication protocols.
A.3 Classification discrimination
The classification variation builds on top of the supervised version, by swapping out the target from the set of candidates, and replacing it with random input that has the same label. In other words, the receiver attempts to guess the candidate with the same label as the target. This game is structured such that every class has exactly to the object-focused referential game [00 12).
representation in the set of candidates. This is somewhat related
Let y = {I..... n be finite set of labels, and let label X' + y be (deterministic) function mapping input to label. Denote the distribution over labels = label(X). The classification variation uses the same receiver functions as in Detiiiion 2] and the following loss function:
X.YY
classification (0.).
E (a4- P(XIY 0]
llog (())(S5(()) rn) = label(x))
Definition 9. An EC setup (X, fx, M, 0, classification discrimination game if $ C M.N
discrimination and pX,YY classification"
Note that if X' is finite, we could choose = | and label(a;;) = i, i.e. define labels as the identity mapping, and end up with the global discrimination game.
Lemma A.3. [proof in page 25 Let fx. M, O 2, be a classification discrimination game. Assuming is unrestricted, a sender So is optimal if and only if it minimizes the following objective:
-I()))((X))
Optimal communication protocols in the classification game maximize mutual information with the labels. If the labels signal meaningful properties of the inputs, protocol induced by this game will reflect that, and be semantically consistent. However, similarly to the supervised discrimination game, optimal solutions can use just [ different messages, as they do not have describe any properties of the input other than its label.
B Reconstruction and k-means clustering
Lemma 5.1 shows that optimal communication protocols in the reconstruction setting minimize the objective function of k-means clustering (36), where each equivalence class corresponds cluster. In this section we show a stronger connection between the two environments. Namely, we show that X is uniform distribution with finite support, both θ and & unrestricted, and the message space M is finite, the reconstruction game can simulate k-means clustering with k: = [M] (the equivalence classes correspond clusters). This happens if we alternate between each agent's optimization:
Assignment step When the receiver is frozen and sender is optimized, the best solution (synchro- nized sender) assigns each input its projection on Receiver's Image:
R()) R(arg min I| R(m)ll) x' x'Elmg(It)
arg min ||x--/1 x/1--
mEM
In other words, the sender chooses the message (cluster) with the closest receiver output (centroid) to its input.
Update step When the sender is frozen and receiver is optimized, the best solution (synchronized receiver) sets each output to be the mean over the relevant inputs:
Rd (m) arg min E [|r
S(x)
m|
E S(X) m]
rEX
am
In other words, the receiver updates its output (centroid) to be the mean over inputs mapped to the given message (cluster).
15

---

C
Proofs
For ease of notation, we often write S² R instead of Sg, Sø-, R.p. Re when the hypotheses are clear from context. Additionally, we use square brackets as a simple notation of message's equivalence class, i.e.,
[m] = (mlo = ST (m) = fxr E E:: So(a) m)
C.1 Simplification of the semantic consistency definition
simplification of Equation (1) Recall Equation (1)
E 21202
| =02l? So(x1) = S0(x2)
E
||
x2ll?
21.23~
Note that since X' is bounded, these values as well as the expectation and variance of X are finite. x²1l²| [Vrr the left term is equal to
Using the identity
21.22~ [||
| - x2l12 S0(11) = =
2
|1:2 x221?
S0(11) = 50(x2) S0(x)) = ml
||21 x2l13
Se(x1) = m. S0(())) = m
m 82
-2-
Var [X] (X) ml]
and the right term is simply 2ar X). Applying this to Equation (1) yields Definition 3
C.2 Semantic consistency proofs
Reconstruction
Let (x, M, 0,0, be reconstruction game. Assuming & is unrestricted, a sender So optimal if and only if minimizes the following objective:
Lemma 5.1
Σ P(S (X)+ =m) Var [X So(X) m]
Let Xjmj denote the distribution ( X [m]) The synchronized receiver for sender S is defined by:
Proof of Lemma 5.7
Rt
arg min E (R(((z)) x|| Re
Since assumed be unrestricted, we get
(m) = arg min E.
[]]r- [[lr- x:| S(x) = m]
rex xxx
arg min E ||r-
rex x-mXpm
E [[mm]]
We can apply this synchronized receiver back in the reconstruction loss function without changing the set of optimal sender hypotheses, since sender is optimal if and only if when paired with
16

---

synchronized receiver, the pair is optimal.
Note that [m] = 5 (m) depends on sender. We get:
S arg See
migmin min
E |I
x||?
a-x
= arg min
E
E [[s(s))]
x=1²
see
x-X
arg min See
P(X [ml)-
E
E X [stta]]
- ==1??
MEM
arg min See
Σ
P(X [m])
E x-X(m
E
X [mi =01(")
mEM
arg min
Σ
P(X [m])
Var [mm
See
meM
The third line is the law of total expectation, where the events are the assignment to message m.
Theorem 5.2 Let (X, fx. M, O, E) be a reconstruction game. Assuming & is unrestricted and ① contains least semantically consistent protocol, every optimal communication protocol is semantically consistent.
Proof of Theorem 5.2
Following Lemma 5.1
let 8" be optimal reconstruction sender, i.e.,
8" E arg min Σ
P(X E (m]o) Var [X] X E [m]o]
mEM
arg min E Var X (X)]]
arg max Var X (x
8
arg Var [X] E Var [X] So(X))
0
The final formula is non-negative by variance decomposition. Since @ contains a semantically consistent protocol, this expressions can be greater than zero, since 0* maximizes it, we get
Var [X] [Var [X]
Sx(X)]] >>
Which proves that Definition 3 is satisfied.
Discrimination
Let (X, fx, M, M, &b, E) be d-candidates discrimination game. Assuming & is unre- stricted, sender Sq is optimal if and only minimizes the following objective:
Lemma 5.3
Σ P(Sa(X) m) E log [ Binomial (d 1, P(So(X) m))
mEM
And when d = 2 (a single-distractor game), this simplifies into
Σ P(So(X) m) MEM
Proof of Lemma 3.3 Given message and candidates (I .2), such that t is the index of the target (S(xz) = m), denote C = { _z} n [m]. Since is sampled uniformly and only candidates in C have non-zero likelihood of being the target, have for all j E {1.....d) d}
Fa. m) = fx (II,..., ...) P = j II, xa) a)
(S(x.) = m 21,
Fdd = j)
P = = 31,
fx (ZI...., xa) P (S(x.) m -1,.... x.)
((S(x)) = =m)
11z C
c
17

---

Denote this probability Pj. A synchronized receiver for sender S is defined as
R' E arg min [(0, R.
ReD
2~X
arg min E
E Rimm te- 1-
log P(R(S()), 2I...,,
REP x-x
=,=2
arg min
E Biamaga (.....
log P(R(S(()), F1, ..,...,, rd)
Re
Since & is assumed be unrestricted, get
RR (m, Fi...... , Ta) E
arg min
E
[- log d: ,1....., R m, (1x)) m)
arg min
Σ Pj log qj
Waimma9a) (a) (1...dd
lower bound on the objective (Gibbs' inequality):
d
Pj log qi Σ J=1
Pj logP;
==1
This lower bound is achieved by selecting 9, Pý (negative log- log-likelihood is proper scoring making a legal output for R. We now apply this
rule). Note that py 2 Vj and Σソ= Pj receiver back in the loss function, similarly to the proof Lf 5.1] Recall that ,1,-.., are the candidates, and given the target's ss index L, the other candidates ( ((+1))))........................ F.) are called distractors. Also note that [m]] = (m) depends on sender.
S* E arg min See
E E(S, R ,x)
x~x
arg min SEe
E
log 2I.
Ti t-Uf
arg min SEO
E Fa to..f.....df
log
[ mal [S(x))]]
arg min SEO
E
log (al, [S(x,)]
B1 Ea (L....
arg min SEO
Σ P(X E [m])
E Ba
log (ai,..., [S(x))] S(xx) = m
MEM
arg min SEE
Σ P(X [m]]
E
log (I (21, ( ( za) n [ml])
x tr-.11.....
mEM
Σ P(X [m])
E log [ + distractors n [mll)
arg min
X 13
MEM
arg min See
Σ
P(X
[m]) E log 1 Binomial (d 1, P(X E [m]))
mEM
d = 2 (a single-distractor game), the expression simplifies to:
min
Σ
[m]) E log (1 (in [m])) P(X
See
mEM
arg min SeB
Σ
P(X e [m]) (p(X E [m]) log 2 + (1 P(X E [m])) log
mEM
arg min SeB
Σ
P(X [m]]²
M
18

---

Corollary 5.4 If the set of possible messages is finite, any sender agent that assigns them such that their distribution is uniform can be paired with some receiver to generate globally optimal loss for the discrimination game.
Proof of Corollary 5.4 Let = |M] be the finite number of available messages. Denote the vector p (p), .......PP Pn) such that p; = P(X E [m:]). For [0, 1J. denote the function
((x x E log(1 + Binomial 1,
Following Lemma 5.3 we can formulate the task as a constrained optimization problem on the simplex:
n
F(P) = Σ (p:)
min PER
==1
111=1 s.t.
p20 0
Note that the distribution of X may create additional constraints. We will prove that f is convex, meaning that this is convex problem, so any KKT solution global minimum (26). The uniform solution p* ( +) is indeed KKT: assigning À 0 and μ ( we have
VF((") VV v(p"1 1)(p") = f
It is left to prove that f is convex over [0, 1]. For simplicity, we replace d 1 with d E N+:
f(x) x Eloog(1 + Binom(d,
This convexity proof is adjusted from mathoverflow (link). Denote
o(k) log(1 k)
and
( k=0 k
d x²(1 -x)e 7 [(x) = x (x))) g(k)
By (x) = Elg(Binom(d,zz))
Σ
We have
Bald (x) (x)
= Σ g(k)
d')
Aax--((1 101 xx)
g(k) d
) (d k)a* (1 2)
k
k=
km
dd)
d
(d)
(d)
Σ 9(b)
Σ g(b)
(d b)r** (1 x21
k)
k:
k=
Using the identities (K) = d((=t) and (i) (d k) = dl (())))) we get
(d-13 1) x² -1 (1 - --) Σ g(k)d
d
d - 1) at (1 - d-)-k-1
Bd(()) (x) Σ
s())d
k -
h.
1=1
k=U
d-1
d 1) x² x²(1- --ا- d Σ g(bb) (d-1) x² (1 - =)=)dd--
Σ
g(k + +1)
d-1
(o(k 1) s(k)) (d-1) x²
( - ddd-1-
k
k=0
d 1 (x:)
where Ag(k) g(k 1) g())
We can apply this again to get the second derivative of 3
Bdd (a) d. (x) d(d - -
19

---

We can now write the second derivative of f:
f*(x) 2B (x) + x- Bg (x)
2dd BBg (x) + x d(dd - 13) (x)
d-1 (d-1)
x² (
d-2
(d-2)
=2d Σ (k(
z)d-1- d(d 1)
Σ
A
g(k))
rht1(1 22)0
k:
k
Since (d-2)
(d-1), the second term can be written as
d-1
1(d-1) Σ g(x))
d-2
(d=
2-2 2
x +1 - 20)a-2 = =idd - 1) Σ
d-1
2 g(k: 1: - 1) h: - -1 x² x²(11 - =z)dd--
11(8-1
k
k=0
k=1
d-1
(d-1) x x))-1-
Agg(k- - 1)k
d-1
(d-1) 1) kk (1 - x) x)--1-
(l: 1)k
k=0
Thus, we have
d-1
(d-1)==
)) a² (1 22)=2)a-1k + d
d-1 1
A >²g(k: ())( d - 1)
k 1)at a² 1a 2) (1-2)a)
(x(x) 2d
Σ (a(k)
k=
k=
d-1
(2Ag(k) + A2g 1)6b) (d=1)==
dd)
x* (1 x/) -
To finish the proof, we show that the term h(k) = =2a((k) A3g(k: 1)k is positive, proving that is convex.
(2g(k) -
g(k: - 1() (g(k + -0
+ (kg(k) + g2g( g(k 1)k 2g((k (g)())))))())+++++ k(g + +1) + +)) - 1) - -g(())
(k ()++2)-g(2 + 1)+ + g(k - 1) 12kk 1) g(k)
We get
l(k)) (k) + A2
1)k- = (A: + 2) log(k: + 2) + k- log(k) - 2(k + 1) log (A: + 1)
For k 0, the function é flogt t is convex, thus h(k) > 0 by Jensen's inequality. Also, h(0) =
2 log 2 > 0.
Theorem 5.5 There exists a discrimination game (X, fx, M, M, E) where & is unrestricted and t contains at least one semantically consistent protocol, in which not all of the optimal communication protocols are semantically consistent.
Proof of Theorem 5.5 X is a uniform distribution over a finite support, then as shown by Corol lary 5.4, Sender is optimal if it maps the same number of inputs to every message. Thus, for example, |x| = 2- M (both finite), any Sender that maps exactly two inputs to each message is optimal, and we can define such an agent by mapping the least similar pairs of inputs together, creating an optimal Sender that does not induce meaning-consistent latent space. For a concrete example, see the proof for Theorem 6.2
20

---

C.3 Spatial meaningfulness proofs
Theorem 6.1 Let (x, M, (. be a reconstruction game, let en 2 &M and let E such that Rip is (X, M,,oo-simple and non-degenerate. Every sender that synchronized with R2 is eo-spatially meaningful.
Proof of Theorem 6.1
the reconstruction task, the optimal constant receiver
C arg min
E E(0, "
arg min
E ||C - x xm
E x
=
Let O* be sender synchronzze with Rip Denote L = supzex ,,,,) and L -= E 11 X112. Since Rp is strictly better than constant, we have
22
L
E
2
Denote the receiver's prediction (Sp- (x) L.p. Note that when the sender's hypothesis class + is unrestricted, this becomes the projection on Ry 's image:
Rip (So- (a:) = Ryp (arg min || (m()))l) arg min | - x'1
Proj (x).
MEM
a'Elng(R.) Rg)
Img(Rpp)
By the definition of reconstmuctio. we get
W || fipll 2 <
Vx EX
|| fupll S VL.
Let 1R
12-1
250 v Var[X] be the constant from Definition 5
For any and T1, X such that
S| (Ia) So (122) Se
we have, since Rip is (X, M, simmle,,, that
@ 12 || RRp (m1)) - R.p(So < IR 8
Thus:
|||] x2ll || l1 + || 82pll + 1|02 szpll < 2 I. In
We now show that every S Eo holds 2/L + +R- < 2/L. First, note that
V2-1
Var X
(1- 12) v Var[X]
Eo
LR
lR
In addition, since L < J.
T.
V VL.
V = (1- VE
v2
so for any < Eo. get:
(1
/
1 L
2(1
1 L
2(VT. VT.)
Eo
<
la
IR
IR
Which finally leads to
2 W <
21

---

Thus, for any E X that hold || (x1) Sg- (22) S S Eu, we get:
11 - x21l & (2VL + lR =)²
< <2V L)² = 4L
2. E || - X12
x-X
E3 || X|2 + 13 || X -2
E 1/1 xwx
X1"
E 1x
X
=-x
xmx
xm
E
[[ XX) - (x2 - x)1
"1
E
11 x2ll
21
Taking expectation yields the desired result for Definition 4
E 2
|1| x2l1?
[|Sa(z1) S0(x2))l S
E 11
|| x21??)
There exists discrimination game (X, M,,,M,00000 én 2 SM and a receiver o which (X, M, 6o -simlee and non-degenerate, where synchronized sender matching Rup is not e-spatially meaningful for any 8.
Theorem 6.2
We propose the following counter example. Let M = (1,2,8,4,5,6) and let X the uniform distribution over the finite support set {1...., 6),0-,,........ -6). = E 1. We let both hypothesis classes be unrestricted, set the number of candidates d to 2 single distractor), and denote the following sets:
Proof of Theorem 6.2
A {1, -1
A2 [2, -
A3 {3. -3]
A {4. -4]
Aş {5. 55
As = {6, -6}
We construct receiver function that optimally tells apart members from different sets, but is unable differentiate within each one. Formally:
R&p cannot tell apart inputs within each set:
vk {l....,6) .6. Vzi, x2 Ag, Vm M R(m,x1,22) (0.5,0.5)
R&p is optimal for inputs from different sets; Given the message m k where E [I....,6) 6), R2 outputs the true probabilities of each input being the target conditioned on the event X E Aş:
0.5)
x1 A₁ and 22 A
(k,
(1,0)
22 $ Ax 41 & A and 32 E A
#1 E A and
10,
The largest possible distance between R2 outputs is / and the minimal distance between messages M 1. Thus,
1
[z((1)) (x2))
5
15
|| -
x2
12
/2
Note that
2
2
91
1
VVar[X]
>
25p
2
6
/2
22

---

Which means Rp is (X, =o We now show that the sender that maps each input to the index of its set, I.c.,
if E A1
if E A2
if E A3 if E A if E As
Sy(1)
if x E As
is synchronized for R.p. Let æ E X' belong to set Ag. The loss function is bounded by:
if = 1 (S,(),,,, 2, 12)1
((0,9,2) = (1) x2~
log
i = 2 RR (So (i), 22, x)2
R& (Sg
if 1
= P( E Ax)
E
log
(1.2
Ri
ift =
22 1 14
(R,( x, 02)1
if =
+(X & As)
E
log
(1,)
(Sp( 02,
if if 2
22~xXxx1Ax
(-108 2)
-log (5)
ift = 1 ift= 2
P(X E An)
+ P(X AR)
E (-,2
(R,(So()),22
xx1A
2 P(X E Ax) 2)og2)+ + P(X (-)
= =(0,g,,))
This means that is synchronized for Rgg. In fact, Sğ is optimal by Corollary 5.4 simplifying the expression to show that R2 is strictly better than constant:
We can continue
e(0, x) = P(X E Ax) - (log 2) + P(X & As) (- log 1)
= 6
log
log 2
And indeed, the average loss of the optimal constant is:
Rgo x,)
ift
E(O, -0.
E
log
(07)1
Rope (Sa(i), x)2
if = 2
Baro
log [C
ift
E
(0( X
1--C C
if 2
(- log C²) + 1 (-log(1 C))
2
(log + loogll -c))
2
1
1
1
VE(0,00,00 (x)
1-c
C
2
C
1
E(0,
0
x))
2 log log
xmN X
2
Finally, can show that S does induce a spatially meaningful latent space. We make the observation that
E X
S5(X) m] = Vm Img(S;)
23

---

This is constant function in mh which means Var(E [ S = m) = 0. By variance decompo-
sition:
Var [X] = E Var X
S2(X)
m] ) Var [ Sx(X) = =m]
Var [X] ((Var Var X
Sx(X) ml
E
||00] 32 x2
E
E
||01 x2 11
01 01,22200
m mn u2 Xma1
I X
||01 x22l?]
E 22
| - x221?
Sa
= So
This equality contradicts Definition 4 that requires strict inequality (for any 6o > 0, the requirement will hold with min(Fo, 0.9).). showing that S3 is neither meaning consistent nor spatially meaningful.
C.4 Proofs for variations of discrimination setups
Lemma A. Let [X, M, o, E) be global discrimination game. Assuming & is unrestricted, sender Sq is optimal if and only if it minimizes the following objective:
-I(;; (x())
Proof of Lemma A. Since negative log- -iikliikhoodd is proper scoring rule and & is unrestricted, the synchronized receiver for sender S is
R (m =
= =m)
Which means that S* is optimal if and only if:
S E arg min E ((S R² ,.) See x~X
arg min
E
log fx(# S(X) = S(x))
SEA
x~X
arg min H HXX S(X))
See
= arg min See
Lemma A.2
Let (X, fx, M, Φh P) be 2-candidates supervised discrimination game. Assuming is unrestricted, a sender Sg is optimal and only if it minimizes the following objective:
Σ WEM
P(S. = m)² MEM BEY
E Σ P(Sx(X) = m, label(X) = =
Proof of Lemma A.2 Since negative log-likelihood is proper scoring rule (shown in the proof of unrestricted, the synchronized receiver for sender S is
Lemma 5.3) and
(m, 31, 21, x2) P((t æi x2, m))
By the game construction, we have
P(t = m) fx(x,) fx(13-) Y label(z,)) 1(5(z,) m)
fx(x
1(5(z,) = =m}
P(Y # label(z,))
Since is assumed to have uniform distribution, P(Y label(z,)) = |-|-
D which means
|
P(t = j, 42, m) = fx
-1-1 1(S(z,) m)
24

---

Thus, the synchronized receiver is P(R"(m, (1.
5) = P(t = 21, 22. m)
P(t = 11 I2 m)
P(t = 21 2,,, m)) + P(t = D2,, m)
1(5(2;) = m]
1[5(z,) m) + ((S(xs-)) =m]
Applying this synchronized receiver to the loss function, we get that 5* is optimal if and only if:
S* E arg min
E x~x
(()
See
{ {
x)) 1), ift=1 } (( = 2). if }
arg min E See
E [1.2
- -log
T-X
I N P(X (lmbel(a))
1
arg min E
E See 2~Xx P(X y label(
log ((( = S())
log(1 + arg mig E P(X & (5(x)) Y lah)l(x:)) log - P(X e [8(x) Y * label(x:) log
arg min E See 2~Xx' = (1)))l(a))
E
S(x))
See 2~X
arg min
E P(X E [5(x)) Y label(x:))
SEe x~X
arg min See
Σ
E
P(X [m], Y y) E P(X [S(z)] Y * (bbel(z){((X)) m. Y y)
mEM yEy
arg min see
Σ
Σ
P(X E [m],Y = = P(X [m] Y +))
mEM yEy
arg min See
Σ
P(X E [m]. Y = y) P(X e [m], Y z3)
MEM WEY!
arg min See
Σ
Σ
P(X E [m]. Y y) (P(X E [ml) - P(X E [m]. Y = =y)
mEM yEy
arg min SEe
Σ
P(X [m]) Σ P(X E E (m), = -) Σ Σ
P(X E [m], y) y)
MEM
yEY
mEM vEY
arg min See
Σ
P(X E [m])²
ΣΣ [m)
VEM
Lemma A.3 unrestricted, sender So is optimal if and only if it minimizes the following objective:
Let (X, M,
0, E) be a classification discrimination game. Assuming & is
(o()))
Proof of Lemma A.3 Since negative log- likelihoooh is a proper scoring rule (shown in the proof of Lemma 5.3) and & unrestricted, the synchronized receiver for sender $ is
(m, 21, x2) P(t æ1,..., En. m)
By the game construction, we have
P(t = j. m) P(X E [m]. Y = =) II fx
Thus, the synchronized receiver is
P( (m, (I..,,,,,
mm)= =) = P(t = Ii,..., Xn,
P(X E [m]. Y =
P(X E [m])
P(Y = j x {mm)
25

---

Applying this synchronized receiver to the loss function, we get that 5" is optimal if and only if:
S E arg min (S, R σ)
See
a~x
arg min E Sge 2~ (m: (xXx)((())))))
E
log
FI, (I label()))
arg min E - log P(Y - label(x)) | X [5(x)))
Sge z~X
= arg min H(Y[S(X))
See
arg min ((X)(() SEe
D Details on empirical work
D.1 Training details
Data. We use version of the Shapes dataset [22]. where each input is image of dimensions (3, 64, 64) containing a single object. The object has a random shape out of (square, rectangle, triangle, pentagon, cross, circle, semicircle, ellipse), and random color out of (red, green, blue, yellow, magenta, cyan, gray). In addition, the object has a random position, rotation, size and distortion. See the Shapes github repo and our code for exact details. We generate ~00K data samples, split into 9088 (142 batches of 64) training samples and 896 (14 batches of 64) validation samples. The MNIST dataset 30 contains (1, 28, 28) images of hand-writted digits, split into 54K training samples and 5440 validation samples.
Agent architectures and loss functions. We use the format (Shapes value //MMNIST value) to indicate values that differ between the datasets. The sender consists of two parts: an image encoder and a message generator. The image encoder receives an input image of dimensions (3, 64, (4) 28, 28) and outputs a latent vector of dimension 100. The exact encoder architecture is given in Figure 5a / Figure 6a A trainable linear layer then maps this output to vector of dimension 150 which serves as input to the message generator, which is implemented by single-layer GRU with hidden dimension 150. Its input is used as an initial hidden state, together with a learnable BOS vector. After the GRU cell, the resulting hidden state is mapped by a linear layer to the vocabulary size (10). We use the Bumbel-Softman relaxation to approximate a symbol sampling, then an embedding matrix is used on the relaxed on-hot vector and the result is fed to the next generation step (along with the current hidden state).
The receiver has different architectures for reconstruction and discrimination. Both receivers start with message embedder, implemented by a singlc-layer GRU with hidden size 150. the reconstruction setup, the final hidden state from that network is given to an image decoder implemented by transpose- convolutions, see Figure 5b) Figure 6b) for full details. The loss in the reconstruction setting is simply the Mean Squared Error (MSE) between the receiver's s output and the target image. In the discrimination setting, the receiver uses another image encoder (with the architecture from Figure 5a/ Figure 6a) and applies it to every image in the set of candidates. The resulting vector is further processed by a Multilayer perceptron (MLP) with two layers, hidden dimension 150 and relu activation. The output of this MLP is a representation for each candidate. We take dot product of this representation with the final hidden state of the message embedder (both normalized) to get the score for each candidate. The discrimination loss is then the Cross- Cross-Entropy classification loss with the target position as label (also called the infoNCE objective). Visual illustrations of the two tasks given in Figure
Figure 8.
Training procedure. While the architectures described above can be jointly trained from scratch, the discrete optimization with Humbel-Sofftax is much slower and less stable than continuous training. We account for that by employing a two-stage training procedure. In the first stage, we train a continuous autoencoder, which is concatenation of an image encoder and an image decoder. The trained image encoder is used by the sender agent in both setups, and also by the receiver in the discrimination game. The trained decoder used by the receiver the reconstruction game. Both of these pretrained modules are kept frozen during the second stage of training, in which the discrete
26

---

game is played out. This means that during EC training, the sender only optimizes the message generator (and linear layer), and the receiver only optimizes the message embedder (and the MLP in the discrimination setting).
Training parameters. We train (during the EC phase) for 200/100 epochs with batch size 64. We vocabulary size 10 and message length 4. Every message is full length do not special end-of-sentence token. We use a fixed temperature value 1.0 for the Gumbel-Softmal sampling method.
Computation resources. The main training stage (EC) takes 4 hours on Shapes, 1.5 hours on MNIST using NVIDIA GeForce RTX 2080 Ti. The continuous training takes < 10 minutes.
Output
Conv2d(32, 3, 3x3, stride=1, pad=1)
LeakyReLUr0.01)
Output
BatchNorm2d(32)
Conv2d(256, 25, 3x3, stride=2, pad=1)
Conv Ton/Traaspose22132..0 32, 3x3, stride=2, pad=1, pad=l
LeakyReL.U00.01)
LeakyReLUr0.00)
2atchNorm24(256)
BatchNorm2d(32)
Conv2d(128, 256, 3x3, stride=2, pad=1)
Con/Traaspose21164. 32, 3x3, stride=2, pad=1, pad=)))
LeakyReLUU00.0)
LeakyReLU10000)
RatchNorm24(1255
RatchNorm2d(64)
Conv2d(64, 128, 3x3, stride=2, pad=l)
Com/Transpose2((12, 64, stride=2, pad=l, pud_pad=l)
Lca0yReLU(0.01)
LeakyReLU(0.00)
BatchNorm2d(64)
BatchNorm22(123)
Conv2d(32, 64, 3x3, stride=2, pad=!)
Con/Transposc24/266, 128, 3x3, stride=2, pad=1, out put_pad==)
LeakyReLU(0.01)
LeakyReLU(0.00)
BatchNorm2d(32)
ButchNorm2d(256)
Conv2d(3, 32, 3x3, stride=2, pad=l)
Conv Com/Transpose2d(25, 256, 3x3, stride=2, pad=l, out_pad=l)
Input
Input
Encoder architecture.
(b) Decoder architecture.
Figure 5: Encoder and Decoder architectures used on Shapes.
27

---

Output
(1024, 784)
Conv2d(32, 3x3, stride=1, pad=l)
LcakyReLU(0.00))
Output
BatchNorm22(322)
Conv2d(128, 25, 3x3, stride=2, pad=
ConTTranppse22132,333 32, 3x3, stride=2, pad=1, pad=1)
Le0.)))LU(0.00)
LcakyReLU(0.00)
BatchNorm2d(128)
BatchNorm22(32)
Conv2d(64, 128, 3x3, stride=2, pod=1)
Con/Transpose2d(64. 32, 3x3, stride=2, pad=1, out_pad=11
LeakyReLU(0.00)
LeakyReLUro..0))
BatchNorm2d(64)
BatchNorm22r64)
Conv2d(32, 64, 3x3, stride=2. pad=1)
Con/Traansooee2((18,, 64, 3x3, stride=2, pad==, pad=)))
LeakyReLU(00.0)
LeakyReLU((0.01)
BatchNorm2d(32)
BatchNorm22((123)
Conv2d(1, 3x3, stride=2, pad=1)
Con/Traanposeed115,, 128, 3x3, stride=2, pad=1, pad=
Input
Input
(a) Encoder architecture.
(b) Decoder architecture.
Figure 6: Encoder and Decoder architectures used on MNIST.
D.2 Metrics
Discrimination accuracy. We sample 40 distractors independently, so the receiver predicts the target out of 41 given candidates. Every image in the validation set is used as target exactly once, and the reported accuracy is the precentage of images that were correctly predicted as targets. reconstruction setting, the prediction is defined by first passing the given message the receiver, which outputs reconstruction of the target, and then picking the candidate which minimizes the loss function (Euclidean distance) with respect to that output.
28

---

Chacte vessAge D
Channerennin U.S.A.
digg KAl DEaged
Picanoracid
Chared mauage 14 5.5.1
Chennimmangg 636...5
Eusling
220
SET
Figure 7: Reconstruction (top) and discrimination (bottom) examples Shapes
Charnel
4
4
Charnd
car Stees NOYE
3
2
3
2
0
5 S06
2
5
Figure 8: Reconstruction (top) and discrimination (bottom) examples on MNIST
Message variance. The unexplained variance is E [Var[X S(X)] Emen P(m) Var [mm]]
where [m] is the set of images mapped to message Using the empirical measures
Var [[m]] 2
Σ
|| 22l
[[m]]
22 [m)
[[m]]
P(m)
N = 896
N
get
[m
Σ P(m)
Var [m]]]
Σ
Σ
| 2211
2
[m]] 2 N
MEM
MEM
1
Σ
22
2N
[m]]
MEM
21,72E(m)
We calculate message variance using this formula. lower value indicates more semantic consistency (less unexplained variance). See Sce/Algorithm for the full calculation procedure. Instead of calculating
29

---

Euclidean distance between raw pixels 4 10 which might be uninformative, we use embeddings
created by the continuous encoder (perceptual loss 39
Algorithm Compute message variance
Require: Dataset X, Set possible messages M, Sender function S +
1: Initialize dictionary equét_classee classes with keys from M and empty lists values
2: for each input x E X do
3: message t- (())
4: Append x to equiv_classse
5: end for
6: Initialize pairwise__ee _8m f- 0
7: for each message M do
8: Initialize local_soe sum + 0
9: for pair (Ii, 2zj ) in equii__llaseeeeeeee messagel] do
10: local_sune + local_ssuun + distance 2zj)
end for
12: if len(equiv ciciiv__lasse_eeeee message))
then
local _uu__ssumm-a local cocoolssmmmmi sum/len(ee (onnleen(equiis (equiv paaases/messsge
14:
pairwise_sure + pairwise_ sum + local_ssu end if
16: end for
+- size of X
18: result +- poirwise_sum/P/ x N)
19: return result
Random baseline. The distribution of messages induced by the trained sender significantly affects some of the metries. To facilitate an insightful comparison between the two EC setups, establish a random baseline per-protocol which randomizes the assignment of images while preserving the number of inputs mapped each message. Comparing the outcomes across different baselines allows us to examine the extent to which channel utilization impacts the results.
Topographic similarity. TopSim 5 is defined as the (Spearman) correlation between pairwise distances in the input space and corresponding distances in the message space. We use edit distance in the message space, and Euclidean distance between embeddings in the input space. The embeddings generated by the frozen encoder trained in the continuous stage (see Appendix D.I).
Bag-of-symbols disentanglement. BosDis 8 calculates per-symbol the gap between the highest and second highest mutual information to an attribute, normalizes by the symbol's entropy and sums over the vocabulary. We consider only two attributes (shape and color), so the mutual information gap is simply the difference between the two.
Positional disentanglement. PosDis S operates similarly to BosDis, but considers positions within the message rather than symbols.
Speaker-centered topographic similarity. S-TopSim 13 operates similarly to PosDis, but rather than summing over positions, the mutual information gap is calculated per attribute. This is more straightforward application MIG (
Message purity. The purity of message with respect an attribute is the percentage of images in its equivalence class that have the majority value. The minimal purity value is achieved by uniform distribution over all of the attribute's 's possible values; the maximal value, 100%, reached only if all images mapped to the message share the same value (e.g., if the attribute is shape and all images mapped to that message contain squares). We take weighted average of message purity over messages. The final measure can be interpreted as the highest attainable accuracy by a deterministic classifier of messages to the attribute. easier version of this measure, which we term max-purity, defines the purity of each message the maximum purity over all attributes.
30

---

E Additional experiments
E.1 Full results: Compositionality, semantic consistency and game performance
We consider three additional compositionality measures from literature (BosDis, PosDis and S- PosDis), Appendixx D.2 for their definitions. Some of these measures require multiple attributes (based on the mutual information gap), so we report results on Shapes. Table showss these results, and Table 4) shows the correlation between each pair of measures, over three subsets of the collected measurements: the joint set (10 runs), reconstruction only (5 runs) and discrimination only (5 runs). We note that the correlations calculated over the joint and often show opposite values to the individual tables. Furthermore, the results regarding S-PosDis somewhat noisy due its low values reported in Table 3 A key observation is that the correlation between compositionality and game performance (disc. accuracy) is negative or nonexistent across all compositionality measures in both the reconstruction subset and the discrimination subset, other than the S-PosDis value in the discrimination table (which we believe is noise, see previous note). This further corroborates our findings in /eeciion
heavily affected by the two distributions
Table 3: Empirical results with compositionality measures from literature.
TopSim f
BosDis f
PosDis
S-PosDis f
Disc. accuracy t
Msg Variance
Reconstruction
0.34 10.02
0.11 10.01
0.04 10.03
0.01 10.01
0.32 10.03
1334.38 178.05
Discrimination
0.09 10.01
0.32 10.19
0.19 20.13
0.01 10.00
0.62 10.05
2280.24 1157.38
E.2
Message purity over the Shapes dataset
Figure 9 presents the color purity, shape purity and max purity in each game type. See Appendix D.2 their definitions. Table 5] shows the baseline values well. The reconstruction-traile agents generate messages that convey more information about each attribute. However, the generated distribution of messages (which defines the random baseline) seems have significant effect on the purity results. We hypothesize that higher multiplicity of attributes may better differentiate the two tasks.
0.6
0.57
0.55
0.54
Purity
0.5
0.5
0.49
0.49
0.45
1.42
0.4
Purity-Max
Purity-Colon
Purity-Shape
Reconstruction Discrimination
Figure 9: Message purity per attribute and game type.
31

---

Table Correlation Matrices for All Data, Reconstruction, and Discrimination Groups
(a) Correlation Matrix (All Data)
TopSim
BosDis
PosDis
S-PosDis
Accuracy
Msg Variance
TopSim
1.00
-0.63
-0.63
0.49
-0.97
-0.97
BosDis
-0.63
1.00
0.98
-0.26
0.65
0.61
PosDis
-0.63
0.98
1.00
-0.29
0.61
0.62
S-PosDis
0.49
-0.26
-0.29
1.00
-0.40
-0,42
Accuracy
-0.97
0.65
0.61
-0.40
1.00
0.92
Msg Variance
-0.97
0.61
0.62
-0.42
0.92
1.00
(b) Correlation Matrix (Reconstruction)
TopSim
BosDis
PosDis
S-PosDis
Accuracy
Msg Variance
TopSim
1.00
0.33
-0.37
0.76
-0.24
-0.61
BosDis
0.33
1.00
0.00
0.17
-0.37
-0.93
PosDis
-0.37
0.00
1.00
-0.02
-0.61
0.26
S-PosDis
0.76
0.17
-0.02
1.00
-0.03
-0.28
Accuracy
-0.24
-0.37
-0.61
-0.03
1.00
0.35
Msg Variance
-0.61
-0.93
0.26
-0.28
0.35
1.00
(c) Correlation Matrix (Discrimination)
TopSim
BosDis
PosDis
SiPosDis
Accuracy
Msg Variance
TopSim
1.00
0.61
0.50
0.82
0.10
0.00
BosDis
0.61
1.00
0.99
0.07
-0.04
-0.23
PosDis
0.50
0.99
1.00
-0.07
-0.14
-0.17
S-PosDis
0.82
0.07
-0.07
1.00
0.26
0.12
Accuracy
0.10
-0.04
-0.14
0.26
1.00
-0.80
Msg Variance
0.00
-0.23
-0.17
0.12
-0.80
1.00
Table 5: Purity measures compared with random baselines.
Purity-Max
Purity-Color
Purity-Shape
Trained
Random
Trained
Random
Trained
Random
Reconstruction
0.57 +0.03
0.54 +0.02
0.54 +0.03
0.49 +0.02
0.49 +0.02
0.48 =0.03
Discrimination
0.50 +0.05
0.45 +0.03
0.49 +0.05
0.43 10.04
0.42 +0.03
0.42 =0.03
E.3
Spatial meaningfulness empirical analysis
Recall that spatial meaningfulness requires close messages to have similar meanings. We propose an evaluation method that alters the message space such that is well clustered, and evaluates the semantic consistency of message clusters rather than individual messages. To do so, we partition the 10 vocabulary symbols to five groups: {0, 1). {2, 3). (4, {6, 7). and (8.9). 9). During training, agents are restricted to using messages composed of symbols from single group. This is implemented by letting the sender generate the first symbol freely, and then masking out all symbols except those belonging to the same set as the initial symbol for the remainder of the message generation. Consequently, messages such as (3, 2,3, 2) and 77 7, 7) are possible, whereas message like (3, 4, 5, 9) can never be generated. There are 16 possible messages within each cluster, and only messages from the same cluster can have Hamming distance smaller than (the maximum). We then calculate spatial meaningfulness as the empirical unexplained variance (Algorithm 1) of these
32

---

clusters; lower value indicates more spatially meaningful messages. Table and Table present this measurement over the validation sets of Shapes and MNIST respectively. While the reconstruction setting leads to better improvement over the random baseline on Shapes, as expected by our findings, the results do not demonstrate high levels of spatial meaningfulness any of the trained protocols. We hypothesize that the size of each cluster (16 messages) is too large, and perhaps different partitioning method will provide more insight.
Table 6: Cluster Variance Shapes.
EC setup
Trained Cluster Var 4 Random Cluster Var
Improvement
Reconstruction
3742.19 19 +64.87 3804.09 +84.73
1915.19 +70.52
4.41% %%
Discrimination
3919.11 +73.73
2.94%
Table Cluster Variance on MNIST.
EC setup
Trained Cluster Var +
Random Cluster Var
Improvement
Reconstruction
1790.88 +29.88
1921.78 +0.17
6.81%
Discrimination
1745.86 +:2.477
1922.23 +0.43
9.18%
Supervised disc.
1783.60 1:4.155
1922.06 +0.35
7.20%
F
Alternative formulations of the discrimination game
Recall that we formally model the discrimination receiver as a function R: M x X' - A(1.....
(Definition 2}. In this setting, assuming an unrestricted hypothesis class, show that the condition- ally optimal receiver outputs a uniform distribution over the subset of candidates that belongs to the given message's equivalence class. This functionality is demonstrated in Figure 10
In this section, we tackle two alternative formulations which provide stronger resemblance common implementations of the discrimination game, at the cost of added complexity. The first alleviates the receiver's s ability to consider more than one candidate at a time; the second explicitly models the prediction using dot product in latent space. With the assumption of an unrestricted receiver hypothesis class, our results are persistent.
S( Candidate1)
Candidate1 w.p.
0.5
S( Candidate2) ) * m
Candidate2
w w.p.
S( Candidate3 ) + m
Rs (m, candidates)
=
Candidates
w.p.
S( Candidate4) m
Candidate4
w.p. 0.5
S( Candidates) + m
Candidate5
w.p.
Figure 10: The conditionally optimal discrimination receiver, assuming it is available in Ø.
Candidate-unaware discrimination
In this formulation, rather than directly outputting a distribution over the candidates, the receiver predicts score for each of the candidates independently, and those scores are normalized to create the final probabilities. Referred to as the descriptive-only variation by [12. Formally, the receiver takes the form
R: X 10.1]
and the game objective is
R(S(2z), log E R(S(I)),
ESS, R, x)) =
E
33

---

The key difference is that each score only depends on the corresponding candidate and the message. However, the conditionally optimal receiver from Figure 10 can still be achieved, for example by the following receiver:
f
if S(x) m
R (m, x))
otherwise
By assuming unrestricted Φ, this receiver is available and therefore optimal. Hence, the set of optimal communication protocols is the same in this setting as the original one.
Latent similarity discrimination
This formulation further restricts the receiver by generating the scores via cosine similarity in latent space, and using the softmax operator to create the final probabilities. This formulation closely resembles common implementations of the discrimination game. Formally, let be the latent dimension, the receiver now consists of two parts: an input encoder Rx: X - IR and a message encoder Ry: M IR The game objective becomes
log exp(cos(Ras (x()))))
E(S, R, =
1- =
(alloos(Raa(SCeee Rx (x())))
Note that the softmax output strictly positive in every entry, so the conditionally optimal receiver from Figure cannot be achieved in this alternative formulation. However, we make the observation that the optimal imput encoder outputs the same latent vector as the corresponding message encoder:
exp(cos(RMr (S(x.)))((()))))))))))))) Rx (x))))
log
2i=1
RM exp(cos(Rx (S(x)))) Rx(xx)))
Eier (RM(coo((M((((a)))) Rx(x))
=-log log
1
exp(cos( RM (S (z.))) Rx (2+))) + li exp(cos( (aa///osssMMMM RM (S(x))) Rx (x:)))
Eipr exp(cos (S(x.)) Rx (x:)))
2 log
1
e+ Eiet (SM Rx (x.)))
and this lower bound can be achieved by selecting
(x) (S()).
In other words, the conditionally optimal receiver needs map equivalent inputs to the same latent vector (up to a multiplicative constant). In the case of unrestricted receiver hypothesis class, this still allows the relationship between equivalent inputs be arbitrary. However, the representations of different messages are incentivized to be well separated, meaning that if similar inputs are mapped different messages, their encodings by Rxx will be far apart. Restrictions on It* may render this infeasible, opening the way for semantic consistency in the discrimination setting. We leave further investigation of this idea to future work.
34

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main results outlined in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Discussion of this work's limitation is given in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
35

---

Assumptions are properly defined and stated for each theorem. Complete proofs are given in Appendix C
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We give a rigorous explanation of the training setup and procedure Ap pendix D.I] and release code with reproducibility instructions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
36

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Exact commands for reproducibility are given with the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iiss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httaa//gggiiee//eeuuebiisisssss /Cc////oddiiisssseeeeeee ssiiiiiisiiiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Training details, including data splits and hyperparameters, are given in Appendix D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the mean and standard deviation our results, see Section 7
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
37

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Compute information is given in Appendix D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research presented in the paper adheres to the NeurIPS Code of Ethics in all aspects.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This theoretical work may improve interpretability of learned representations in the long run, but has no direct societal impacts that we foresee.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
38

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This is theoretical work and does not include released data models, there is no risk of this paper's contributions being misused.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the most relevant code package (EGG) in Section 7
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
39

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Training details are given in Appendii D.I The code contains documentation and usage instructions for reproducibility.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
experiments in this paper involve human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
No experiments in this paper involve human participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
40

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
41


Reviews:
[{"review_id": "0O0x60CKfl", "reviewer": "Reviewer_cajw", "summary": "This paper investigates and analyzes the emergent communication protocols developed by agents during collaborative tasks that necessitate message transmission to solve given problems. The authors contend that traditional performance measures, including task performance and properties like compositionality and efficiency, are insufficient for ensuring that emergent communication is meaningful or interpretable, especially in the context of human communication evolution. To address this gap, they introduce the concepts of semantic consistency and spatial meaningfulness, inspired by observations of human communication and the properties of emergent communication (EC) protocols influenced by the discrete bottleneck. The authors propose theoretical solutions to assess the similarities of emergent protocols to natural language and conduct empirical tests to support their theory. Their study revolves around an EC setup inspired by Lewis’ games, where a sender agent describes a given input, and a receiver agent makes predictions based on that description. The authors focus on reconstruction and likelihood-based discrimination objectives used to train the agents and develop theories based on these objectives. Experiments are conducted using the MNIST dataset, and results are reported based on various metrics measuring semantic consistency and spatial meaningfulness.", "questions": "- Could the authors elaborate on the interpretability and usefulness of the explained variance?\n- As mentioned earlier, most discussions and results are closely tied to the MNIST dataset. Could the authors discuss how their theory and measures would generalize to other datasets?\n- How do you think such setups and measures would fare in the presence of LLMs? What are some examples where these measurement concepts would be useful, considering either the agents themselves can use LLMs or an LLM interpreter could make sense of the emergent protocols?\n- Are the theoretical results tied specifically to two-agent games, or would they scale with the number of agents?\n- As highlighted in the weaknesses, many results appear to be artifacts of the setup where the sender serves as an input encoder and the receiver acts either as a decoder (reconstruction) or predictor (classification), with messages representing the latent state space of inputs. Given this, could the authors explain the motivation and significance of their theoretical results? Is there some aspect of this setup and definition of semantic consistency that makes these results non-trivial?\n- It is unclear if the study of spatial meaningfulness provides any useful insights, as both theoretical and empirical results seem incomplete and less rigorous. Could you elaborate on why no significant differences were found? One possible reason might be the reliance on the threshold value $\\epsilon_0$. Have the authors studied the effect of changing this threshold?\n- It may also be useful to explore how these measures relate to already established measures such as compositionality and efficiency. For instance, do semantically consistent messages lead to more efficient protocols or the emergence of a more compositional language?", "limitations": "The authors have discussed the limitations at length which is a plus point of the paper and remaining limitations have been described in the weakness section.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- In the field of emergent communication, there is indeed a need to introduce concepts that capture the similarities with human language evolution and measures such as compositionally and efficiency have limitations in achieving this. In that regard, the topic of the paper is interesting and it useful to see semantic consistency being studied towards this goal. \n- The paper is well-written, organized, and easy to read.\n- The authors provide rigorous theoretical analysis and formal proofs regarding semantic consistency and spatial meaningfulness, which are robust and well-founded.\n- The discussion of the approach's limitations is thorough and highly appreciated.", "weaknesses": "- While the introduction of a concept to analyze the likeness of emergent communication to the natural language (especially in terms of its meaningfulness or interpretability) is an interesting direction, there are several shortcomings of the current exposition with respect to its novelty. significance and clarity of the contributions.\n- Semantic consistency definition seems highly contrived as it depends on distance between inputs which is only applicable to some modalities. The connection to human communication is weak and often far-fetched. The authors' claim that \"inputs mapped to the same message should be semantically similar\" is valid in some cases but does not universally apply to human language, making this motivation less useful.\n- It is not clear why the difference observed in semantic consistency between reconstruction and discrimination task is surprising or an interesting result. The way it is discussed and formulated in this work, the implication on distance based similarity between inputs mapping to the same message (which are essentially latent representations of inputs learned with different objectives) seems to fall out from classical results on classification vs generative modeling.\n- Along the similar lines, the spatial meaningfulness appears to be a fancy name for clustering of messages in the latent space. While the authors have overall weaker results for spatial meaningfulness, any difference in the structure of this latent space and resulting clusters also seem to fall out of classical deep learning concepts. \n- While the authors already acknowledge various assumptions and simplifications required to present the analysis in this work as a part of limitations, the theorems on spatial meaningfulness requires even further stricter assumptions such a as simplicity and synchronization which limits its applicability. \n- Related work: While the authors have covered a series of related literature, some important ones are missing. For instance, semantic meaning and other measures have been studied in several studies such as [1], [2] and [4]. Also, there is nice survey on this topic that is useful to position this work [6].\n- A major drawback of this paper is that it considers a very simple and small setup and dataset for it to provide significant impact to the community where already emergent protocols are being studied at high scale [5] and complexity [3].\n- The empirical results provide modest support for the theory, except for semantic consistency in the reconstruction task, as measured by message variance. However, the interpretation of message variance in this context is unclear.\n- The omission of large language models (LLMs) as tools to measure the closeness of emergent protocols to human-like communication is a significant oversight. LLMs could serve as both evaluators and agents capable of generating more human-like communication protocols.\n\n\n[1] Emergent Discrete Communication in Semantic Spaces, Tucker et. al. NeurIPS 2021\n\n[2] Interpretable agent communication from scratch, Dessi et. al. NeurIPS 2021\n\n[3] Emergent Communication for Rules Reasoning, Guo et. al. NeurIPS 2023\n\n[4] Emergent communication for understanding human language evolution: what’s missing? Gelke et. al. ICLR 2022\n\n[5] Emergent Communication at Scale, Chaabouni et. al. ICLR 2022\n\n[6] Models of symbol emergence in communication: a conceptual review and a guide for avoiding local minima, Zubek et. al. 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this insightful review. We are glad you have found the paper to be rigorous and well written. Following your feedback and others, we have prepared some major modifications to be added in the next revision.\n\n**Question 1**\n\nThe explained variance relates to the average proximity of inputs with identical messages. The absence of explained variance (which we refer to as semantic inconsistency) means that inputs mapped to the same message are not closer to each other (in expectation) than two randomly selected inputs. This would indicate an inherently unintuitive communication protocol.\n\n**Question 2**\n\nSince our main contributions are theoretical, we used MNIST to generate proof-of-concept results. That said, given your question, we have conducted additional experiments on a more sophisticated dataset—a special version of the Shapes dataset [1] where each input is an image of an object with several sampled attributes (shape, color, position, rotation, and size). The results are appended to the general author rebuttal (and will be added to the paper in the next revision).\n\nThe results show a significant gap in semantic consistency (lower message variance is preferable), on par with the previous evaluation on MNIST. Additionally, topographic similarity strongly favors the reconstruction setting (0.37, versus 0.08 for discrimination), confirming a positive relationship between semantic consistency (as measured by message variance) and compositionality (as measured by topsim). We also report correlations between every pair of measures.\n\n**Question 3**\n\nInteresting question. In general the field of EC as we know it did not catch up with the advance of LLMs. One might try to use LLMs for EC interpretation and decipherment, but this seems quite challenging. This idea is empirical in nature and not entirely in the spirit of our paper. In any case, we will add discussion of these possible connections in a revision. \n\n**Question 4**\n\nOur analysis only considers two-agent setups. The generalization of Lewis' games to multiple agents is not straightforward but could provide interesting insights.\n\n**Question 5**\n\nThe sender-receiver EC setups are indeed built to resemble encoding and decoding, which is why we can consider messages as latent representations. What distinguishes EC setups from general representation learning is the discreteness of the communication channel, which is key to our analysis. The discreteness induces the many-to-one nature of the communication protocol, which is why the perspective of equivalence classes makes sense in this context. As for the significance of our (theoretical) results: a greatly desired goal in EC literature is the characterization of communication properties that separate human language from EC protocols. We discover, precisely define, and rigorously support two such properties, which can be used in future research. For example, our analysis led us to question the generative vs. discriminative division of objectives, in favor of a distance-based vs. probability-based distinction. This move is reflected in our choice to rebrand the probabilistic reconstruction variant (which is generative) as \"global discrimination.\" These results are novel and non-trivial. \n\n**Question 6**\n\nThank you for this insightful question. With regard to the empirical results, we admit that our proposed metric (cluster variance) for spatial meaningfulness does not yield interesting results. We hypothesize that the size of each cluster (16 messages) is too large, and perhaps a different partitioning method will provide more insight. This hypothesis will be added in the next revision. On the theoretical front, for the next revision we have improved our results by modifying the simplicity condition (ln 271) so that theorem 6.1 applies to any desired threshold (previous version applies to the minimal threshold $\\varepsilon_M$). The interest behind the spatial meaningfulness definition is twofold: a) it allows an analysis that takes into account distances between messages, and b) the analysis does not assume an unrestricted hypothesis class, which is a major shortcoming of the theorems in section 5.\n\n**Question 7**\n\nThere is an interesting relationship between our definitions and the most common compositionality measure, Topographic Similarity (topsim): topsim evaluates the correlation between distances in the input space and the corresponding distances in the message space. Notably, topsim considers the relationship between every pair of inputs, whereas our definitions only consider pairs that correspond to similar messages. The latter follows an intuitive asymmetry: inputs with similar messages are expected to be similar, but inputs with dissimilar messages do not have to be different. This paragraph will be added to the paper in the next revision. In addition, we performed additional experiments comparing message variance to existing compositionality measures from the literature. See the response to question 2.\n\n**The implication <...> seems to fall out from classical results on classification vs generative modeling.**\n\nAn illustration of why this is not the case can be seen in the \"global discrimination\" game, which is generative but equivalent to a special case of the discrimination game (on the classification side).\n\n**Spatial meaningfulness appears to be a fancy name for clustering of messages in the latent space.**\n\nIf anything, spatial meaningfulness describes the clustering of clusters! Recall that the message space M is given and constant; the sender only determines the mapping to messages. Each message corresponds to a set of inputs, and spatial meaningfulness requires those sets of inputs to be close (in the input space) when their message vectors are close.\n\n[1] Kuhnle, Alexander and Copestake, Ann. \"ShapeWorld - A new test methodology for multimodal language understanding\". arXiv preprint arXiv:1704.04517 (2017)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cFLTcHBcN", "reviewer": "Reviewer_XigD", "summary": "This paper explores the properties of communication protocols that emerge when artificial agents are trained to perform collaborative tasks through a communication channel. The authors identify a key prerequisite for meaningful communication, termed \"semantic consistency,\" which demands that messages with similar meanings should be used across different instances. They provide a formal definition for semantic consistency and use it to compare two common objectives in emergent communication research: discrimination and reconstruction. The paper proves that under certain assumptions, semantically inconsistent communication protocols can be optimal for discrimination tasks but not for reconstruction. It introduces a stricter property called \"spatial meaningfulness,\" which considers the distance between messages and aligns more closely with natural language characteristics. Experiments with emergent communication games validate the theoretical findings, showing an inherent advantage in communication goals based on distance rather than probability.", "questions": "See weakness", "limitations": "- The paper mentions a two-stage training procedure involving a continuous autoencoder, but it does not delve into the computational efficiency of the proposed methods. Including a discussion on the scalability of the approach and its computational requirements would provide a more comprehensive understanding of its practical applicability.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The formal definitions and theoretical proofs are innovative and contribute to the literature by setting new standards for what constitutes meaningful communication in EC systems.\n- The paper is well-structured, with clear definitions, hypotheses, and theorems that build logically upon one another.", "weaknesses": "- The empirical validation using the MNIST dataset is a good start but may not fully capture the complexity of real-world applications. The paper could benefit from additional experiments with more diverse datasets or more complex tasks to further validate the theoretical findings.\n- The paper introduces the concept of spatial meaningfulness, which considers distances between messages. However, the empirical analysis of spatial meaningfulness using message clusters may not fully explore the nuances of this concept. Further investigation into different methods of evaluating spatial meaningfulness could provide more actionable insights.\n- The author assume that Euclidean distance broadly indicates the level of semantic similarity. However, depending on the context, other distance metrics might be more appropriate. The paper could be improved by discussing the limitations of using Euclidean distance and the potential impact of using alternative metrics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this insightful review. We are glad you have found the theoretical part innovative and well structured. Following your feedback and others, we have prepared some major modifications to be added in the next revision. We would now like to address your comments one by one.\n\n\n**Comment 1: The paper could benefit from additional experiments with more diverse datasets or more complex tasks to further validate the theoretical findings.**\n\nIndeed. While our key contributions are theoretical, we agree more experiments can help validate our findings. While the simplicity of MNIST has its benefits, we have recently conducted additional experiments on a special version of the Shapes dataset [1], where each input is an image of an object with several sampled attributes (shape, color, position, rotation, and size). The multiplicity of attributes allowed us to calculate compositionality/disentanglement measures from the literature, and the results are appended to the general author rebuttal (and will be added to the paper in the next revision).\n\nThe results show a significant gap in semantic consistency (lower message variance is preferable), on par with the previous evaluation on MNIST. Additionally, topographic similarity strongly favors the reconstruction setting (0.37, versus 0.08 for discrimination), confirming a positive relationship between semantic consistency (as measured by message variance) and compositionality (as measured by topsim). We also report correlations between every pair of measures, taken over 10 runs (5 for each task). These correlations should be considered with caution as they are affected by the inclusion of both tasks for calculation. We may report separate tables for each game type to avoid this issue.\n\n**Comment 2: The empirical analysis of spatial meaningfulness using message clusters may not fully explore the nuances of this concept. Further investigation into different methods of evaluating spatial meaningfulness could provide more actionable insights.**\n\nInvestigating other estimation methods of spatial meaningfulness would indeed be an interesting addition. We wish to emphasize that the major contributions of this paper are theoretical, and the empirical analysis is intended to be a complementary proof-of-concept supporting our findings. With that said, we admit that our proposed metric (cluster variance) for spatial meaningfulness does not yield interesting results, unlike its counterpart (message variance). We hypothesize that the size of each cluster (16 messages) is too large, and perhaps a different partitioning method will provide more insight. This hypothesis will be added in the next revision.\n\n**Comment 3: The paper could be improved by discussing the limitations of using Euclidean distance and the potential impact of using alternative metrics.**\n\nWe do mention this limitation on page 6 (footnote) and in the limitations section, but perhaps not elaborately enough. In some types of data, like images, Euclidean distance on raw pixels might not be indicative of desirable properties. Note that all our theoretical results hold anyway, and that some of our results do not depend on distance (e.g., Corollary 5.4). Furthermore, a solution that we implement in our experiments is the use of embeddings generated by a pretrained continuous model. Within that embedding space, the meaningfulness of Euclidean distance becomes an easier assumption. This discussion will also be added to the next revision.\n\n**Comment 4: Including a discussion on the scalability of the approach and its computational requirements would provide a more comprehensive understanding of its practical applicability**\n\nAppendix D is written precisely to supply technical information regarding the experiments. The first stage of training is a continuous autoencoder training procedure, which is very quick (especially on MNIST). It takes approximately 10 minutes using a standard GPU. This detail will be included in the next revision.\n\n[1] Kuhnle, Alexander and Copestake, Ann. \"ShapeWorld - A new test methodology for multimodal language understanding\". arXiv preprint arXiv:1704.04517 (2017)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t4qoYxsIfv", "reviewer": "Reviewer_TFDu", "summary": "The authors consider a collaborative multi-agent (2 agent) setting with inter-agent communication, where the communication protocol is learned by the agents in order to maximize their common objective, which can be either a reconstruction or discrimination task. However, the authors note that this communication protocol that maximizes their common objective need not be meaningful. The authors define meaningful communications as those that have semantic consistency across different objectives or tasks. The authors show that under some technical conditions that semantically consistent protocols optimize discrimination tasks but not reconstruction tasks. They also show that reconstruction tasks require spatial meaningfulness as well, which is defined as a stricter property than semantic consistency. Also, the authors show that all optimal solutions to the reconstruction task are semantically consistent and also spatially meaningful (under some different assumptions), however semantic consistency is not a necessary condition for optimal protocols for discrimination tasks and neither is spatial meaningfulness. They validate these theoretical results with some numerical experiments using MNIST.", "questions": "1. Since the setting considered is a collaborative multi-agent system, why don't the agents communicate the raw information that they have to each other? Is this due to communication channel or cost related limitations? A related question: what leads to the \"discrete bottleneck\" mentioned in line 45?\n2. Why are only reconstruction and discrimination tasks considered? What communication protocols emerge in multi-agent games such as football (for example, multi-agent communication in https://github.com/LARG/HFO?tab=readme-ov-file , which is a simulated soccer environment) ?\n3. What does \"...protocols created via the discrimination task often generalize extremely well to random noise data,...\" in Line 42 mean? Is it that such protocols allow a receiver to accurately classify a random noise input or does it imply that the receiver is robust to noise? Why is this property compared with \"human-recognizable properties\",  since human communication is not usually fully collaborative and the objectives are also more complex than discrimination and reconstruction?\n4. Why is similarity to natural language a desired property for such inter-agent communication protocols? Is it for human-in-the-loop scenarios or for debugging the system by a human?\n5. Shouldn't the EC setup objective given immediately after Line 142 also include $M$, the message space as a conditioning variable?\n6. With the statement: \"Note that a pair of agents can be synchronized in both directions without being optimal.\", do the authors imply that constraining the $\\theta$ and $\\phi$ space might lead to sub-optimal protocols or unilateral (independent or disjoint) optimization by the sender and receiver will lead to sub-optimal protocols?\n7. Is the communication channel assumed to be noiseless?\n8. Have the authors considered comparison with Vector Quantized auto-encoders as well?\n9. The additional input given in the Discrimination task is not shown in Figure 1. Is there a reason for this?\n10. How are the independently sampled distractors communicated to the receiver agent in the discrimination task? Don't channel limitations apply here?\n11. In Figure 3, are the colours and shapes indicative of the input class or are they (partial) input properties? If so, can this be added to the caption?\n12. The equation below Line 213 is not clear. The arg min operator is set-valued, so $\\in$ is more appropriate than $=$.\n13. Does the synchronized receiver setup lead to some loss of optimality?\n14. Can the authors define the \"Binomial\" function and their notation in Lemma 5.3?\n15. Can the condition given in Definition 5 be ensured/verified apriori, in practice?\n16. What is the motivation/intuition behind Definition 6? Specifically why is the $\\frac{1}{4}$ term chosen?\n17. In Page 14, what is $i$ in $S(i)$ the third line of the last set of equations? Shouldn't this be $S(x)$?\n18. What are the differences in the proof of Lemma 5.1 with reference [33]?\n19. Typo: Line 541, \"for\" is not needed here.\n20. Can the $\\cdot$ (dot) symbol imply in the equation in last line on Page 15 be defined and brackets added here to make reading this expression easier?", "limitations": "The authors adequately address the limitations in a separate section in the main paper itself.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is very well written and all the theorems and proofs are clearly written. The problem is interesting and the theoretical tools presented by the authors will be useful in several other problems as well.", "weaknesses": "The authors define two properties for meaningfulness of messages, but in their empirical evaluation, they are unable to show the importance of their more restrictive property - spatial meaningfulness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this insightful review. We are glad you have found the novel theoretical tools useful. Following your feedback and others, we have prepared some major modifications to be added in the next revision.\n\nYou have mentioned as a weakness the empirical results with regard to spatial meaningfulness (the \"cluster variance\" metric\"). We would like to emphasize that the key contributions of our work are theoretical, and that the empirical analysis is intended to be a complementary proof-of-concept supporting our findings. With that said, we admit that our proposed metric for spatial meaningfulness does not yield interesting results, unlike its counterpart (message variance). We hypothesize that the size of each cluster (16 messages) is too large, and perhaps a different partitioning method will provide more insight. This hypothesis will be added in the next revision.\n\nWe now like to address your questions.\n\n1. The agents cannot communicate raw information due to the limited capacity of the communication channel, i.e., the \"discrete bottleneck\". In most EC setups, the message space is constructed as sequences of tokens and is therefore often small in comparison to the input space. For example, in our MNIST experiments, there are 10K possible messages, which is smaller than the size of the dataset.\n2. There are several benefits of simple two-agent communication games. To name a few: optimization is easier, language analysis is more straightforward (there is only one speaker), and each message functions as a latent representation of the input, so concepts from representation learning literature can be applied to study the setup.\nWith that said, EC with many agents is also studied frequently, but outside the scope of this paper.\n3. The experiment that revealed the agents' ability to perform well on random noise [1] trained discrimination agents on natural images (imagenet) and then performed inference where both the target and the (single) distractor are just sampled noise. Surprisingly, the agent were able to play the game (i.e., the receiver was able to detect the target) with high accuracy. This result indicates that the features perceived by the agents are not high-level classes or objects (\"human-recognizable properties\"), illustrating the potential for counterintuitive communication protocols to emerge.\n4. The goal of inducing characteristics of natural language in communication protocols is what often sets apart EC from related fields such as Multi-Agent Reinforcement Learning (MARL), which focus on long-term expected reward. Creating learning environments that induce protocols similar to natural language is desirable in several aspects: it may improve our understanding of the underlying deep learning architectures, advance the goal of human-AI interaction, and even provide insights into language evolution.\n5. It doesn't have to, because M is embedded into the functions S and R.\n6. A situation where both agents are synchronized is a situation where neither agent can 'do any better' given the fixed state of the other (Nash equilibrium). This pair of agents does not necessarily minimize the expected loss. This kind of suboptimal equilibrium can happen with an alternating optimization method (e.g., local minimum at the end of kmeans).\n7. Yes. Investigating the effect of noise in the communication channel is an interesting idea for future work.\n8. Quantization is a viable discretization method for EC research. However, it often defines very large message spaces, which means that most used messages likely represent a single input. For that reason, it does not fit the required setting for our analysis.\n9. Sorry for the confusion. We avoided additional arrows to maintain an elegant figure. To clarify this, we added an explanation in text below the figure.\n10. The distractors are given to the receiver by the game, as independent input. The only thing going through the communication channel is a single message, which the sender generates based on seeing only the target (the \"correct\" candidate).\n11. Right, this is an important clarification. The input itself is only the location of each object (a vector), not the attributes themselves. We used shapes and colors to illustrate how distance between inputs can be indicative of their semantic similarity. This clarification has been added to the caption for the next revision.\n12. Thank you for the observation, we fixed it.\n13. It does not. By definition, a synchronized receiver minimizes the loss for a specific sender. As a result, when we plug in the closed-form synchronized receiver into the loss function, we get the best achievable loss value for the sender (as a function of the sender agent).\n14. We have added a clarification note, the word \"Binomial\" refers to the binomial distribution, over which the expectation is taken.\n15. We are not sure how one should go about verifying this assumption in practice. While the constant k can be empirically estimated, one would have to bound (from above) the norm of the receiver's gradient to ensure that this definition is satisfied.\n16. The motivation behind the non-degeneracy definition is to prevent a situation where the fixed receiver in theorem 6.1 is constant, which means that the sender agent doesn't affect the game performance, and therefore isn't incentivized to use a spatially meaningful communication protocol. Our proof for theorem 6.1 requires the receiver to be strictly better than constant and this bound is performed twice, so any constant smaller than ½ can replace the ¼ in definition 6.\n17. Indeed a typo. Thank you for the observation, we fixed it.\n18. We use slightly different settings and notations, so we chose to include the proof for completeness.\n19. Thank you for the observation, we fixed it.\n20. Of course. Done.\n\n[1] Bouchacourt, Diane and Baroni, Marco. \"How agents see things: On visual representations in an emergent language game\". arXiv preprint arXiv:1808.10696", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YdT9VXklF5", "reviewer": "Reviewer_o4TS", "summary": "# Problem :\n\nEmergent Communication (EC) protocols have been shown to be counterintuitive even when they enable agents wielding them to solve their related task.\n\n# Contributions :\n\nRather than assuming that meaningful communication is taking place when a goal-oriented communication protocols enables the goal to be fulfilled by the agents wielding it, the paper identifies and formalises two goal-agnostic prerequisites to meaningful communication, the first one being coined **semantic consistency**, which relies solely on the claim that ‘inputs mapped to the same message should be semantically similar as is the case with human language’, and the second one entitled **spatial meaningfulness**, which is stricter for ‘it also accounts for distances between messages’ in the language space.\n\nThe paper then proceeds to analyse the two common EC environments (reconstruction vs discrimination settings) with those concepts, showing both theoretically and experimentally (with the MNIST dataset) that:\n\n1. in the reconstruction setting, every optimal solution is semantically consistent ;\n\n2. providing (i) a 'bounding [of] the rate of change in receiver's output' is applied and (ii) the receiver does not degenerate, then optimal reconstruction-setting solutions are also spatially meaningful, which is opposed to …\n\n3. … the fact that optimal discrimination-setting solutions need not be semantically consistent nor spatially meaningful.\n\nThe later (3) is proven thanks to the interesting insight (in Corollary 5.4) that a uniformly random communication protocol can be a globally optimal _discrimination-setting_ solution, thus meaning that its input-to-message mapping can be arbitrary, i.e. meaningless.", "questions": "Please see within the strenghts and weaknesses, but mainly the following:\n1) Please see WC3.\n\n2) Please see WC4.\n\n3) Please see WS3 and WS4.\n\n4) Please see WS5.\n\n\n# References:\n\n[paper 7]: Chaabouni, Rahma, et al. \"Compositionality and Generalization in Emergent Languages.\" ACL 2020-8th annual meeting of the Association for Computational Linguistics. 2020.\n\n[1] : Denamganaï, Kevin, and James Alfred Walker. \"ReferentialGym: A Nomenclature and Framework for Language Emergence & Grounding in (Visual) Referential Games.\" arXiv preprint arXiv:2012.09486 (2020), 4th EmeCom Workshop @ NeurIPS 2020.\n\n[2]: Cogswell, Michael, et al. \"Emergence of compositional language with deep generational transmission.\" arXiv preprint arXiv:1904.09067 (2019).\n\n[3]: Lazaridou, Angeliki, et al. \"Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input.\" International Conference on Learning Representations. 2018.\n\n[4]: M. Krifka. Compositionality. The MIT encyclopedia of the cognitive sciences, pages 152–153, 2001.\n\n[5]: Denamganaï, Kevin, and James Alfred Walker. \"On (emergent) systematic generalisation and compositionality in visual referential games with straight-through gumbel-softmax estimator.\" arXiv preprint arXiv:2012.10776 (2020). 4th EmeCom Workshop @ NeurIPS 2020.\n\n[6]: Denamganaï, Kevin, Sondess Missaoui, and James Alfred Walker. \"Visual Referential Games Further the Emergence of Disentangled Representations.\" arXiv preprint arXiv:2304.14511 (2023).\n\n[7]: Choi, Edward, Angeliki Lazaridou, and Nando de Freitas. \"Compositional Obverter Communication Learning from Raw Visual Input.\" International Conference on Learning Representations. 2018.\n\n[8]: Kottur, Satwik, et al. \"Natural Language Does Not Emerge ‘Naturally’in Multi-Agent Dialog.\" Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. 2017.\n\n[9]: Resnick, Cinjon, et al. \"Capacity, Bandwidth, and Compositionality in Emergent Language Learning.\" Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems. 2020.\n\n[10]: Santoro, Adam, et al. \"Symbolic behaviour in artificial intelligence.\" arXiv preprint arXiv:2102.03406 (2021).\n\n[11]: Denamganaï, Kevin, Sondess Missaoui, and James Alfred Walker. \"Meta-Referential Games to Learn Compositional Learning Behaviours.\" arXiv preprint arXiv:2207.08012 (2022).\n\n[12]: [https://github.com/google-deepmind/3d-shapes](https://github.com/google-deepmind/3d-shapes)\n\n[13]: [https://github.com/google-deepmind/dsprites-dataset](https://github.com/google-deepmind/dsprites-dataset)\n\n[14]: Xu, Zhenlin, Marc Niethammer, and Colin A. Raffel. \"Compositional generalization in unsupervised compositional representation learning: A study on disentanglement and emergent language.\" Advances in Neural Information Processing Systems 35 (2022): 25074-25087.\n\n[15]: Brandizzi, Nicolo. \"Toward More Human-Like AI Communication: A Review of Emergent Communication Research.\" IEEE Access 11 (2023): 142317-142340.", "limitations": "## Limitations & General advice:\n\nI think the paper’s experiment section is the main issue of the paper for it is too shallow and not performed on a relevant dataset.\nBy shallow, I mean that it fails to provide insights with respect to previous works, which limits its overall impact onto the community.\nI have highlighted in the weaknesses related to significance how this could be addressed, as far as I am concerned.\n\nOverall, my appreciations of the paper is currently only at 5 because of those limited evaluation, but if this can be addressed then I would raise my overall rating to 8, and my contribution rating to excellent.\n\nIt will be necessary to find some space to include some of those considerations.\nThus, I would advise the authors to replace most of their \\subsection with \\textbf{Subsection Title.} without line jump, for it would gain about 3 lines for each subsection titles, without hurting clarity.\n\nFigure 3 is great for insights but it occupies a lot more space than necessary, I think. Moving it all into a vertical \\wrapfigure should be helpful to gain space, without impairing clarity. Subsequently, moving the subcaptions into the main caption and referring to each part as \\textbf{Top/Middle/Bottom} would be helpful in gaining some space too.\n\nln203-208’s itemize could be replaced with normal text and ‘(i)’ and ‘(ii)’ before each contributions, for instance, to gain 3 lines of content probably. Similar treatment for ln261-266’s itemized can be applied.\n\nLemma 5.1’s equation could enable one extra line by removing the assumpted line jump.\n\nLn 252 and the associated replication of Equation 1 could be spared by simply referring to equation 1, which would gain about 3 lines of space..\n\n# POST REBUTTAL UPDATE :\n\nMost of my concerns have been adressed by the rebuttals, with the exception of some clarifications around WS2/WS4 and some possible further revisions regarding WS5, therefore I am increasing my overal score to 7.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "## Originality :\n\nTo my knowledge, the contributions made here on a theoretical standpoint are novel and valuable.\n\n## Quality :\n\nSQ1: Acknowledgement of the ‘global discrimination’ context (in footnote 1) and discussion around the difficulty to measure the quality of the receiver’s reconstruction is very insightful. It might be worth expending and/or emphasising further even (possible citation of [1]’s section 2.7 parag. 2, where the Task & Talk game [2,8] is highlighted as an instance of it), by bringing the nuance made in ln446-447 back into the main text, for instance.\n\nSQ2: Section 2.3 is very insightful and provides a lot of important connections to understand the current work and its stakes.\n\nSQ3: The definition of an EC setup as a sextuple that allows representing both discrimination and reconstruction settings in a common framework is very insightful and a worthwhile contribution to the field, I think.  \n\n## Clarity :\n\nSC1: Well-written introduction that provides context and stakes with clarity.\n\n## Significance :\n\nSS1: I think the EC setup (cf. SQ2) is of great significance to the EmeCom field.\n\nSS2: I think the semantic consistency is a valuable theoretical contribution, but it is unclear how it relates to other aspects in practice and therefore I would hope for the authors to revise the paper towards providing more significance for it (cf. weaknesses below with some proposal improvements).", "weaknesses": "# Weaknesses :\n\n## Originality :\n\nWO1: Missing citation in Section 2.2 of [15]\n\n## Quality :\n\nWQ1: possible missing citations regarding the introduction of choices for the discrimination task [1: section 2.6.1. highlighting [3]] and the impact on the resulting protocols [3: section 3.7].\n\nWQ2: ln 92-94 : missing citations for the definition of compositionality, e.g.  “the meaning of a complex expression is a function of the meaning of its immediate syntactic parts and the way in which they are combined” [4] ;\n\nln94-95 : possible missing citations regarding the argument that compositionality does not correlate with generalisation: [5] and [6], especially in visual domain (as opposed to [paper 7] being constrained to symbolic domain).\n\nWQ3: following SQ2, I would be inclined to argue for the latter end of section 2.3 to be further detailed in order to make it more accessible to a wider audience, possibly by defining MI in the context of EmeCom already here, maybe? My point is that the discussion is citing a lot of papers (which is great for attribution and significance), but it does not help the not-specialist reader to understand the stakes concretely. A possible solution could be to separate background and related works into two sections and push the related works section at the end of the paper, in order to allow it to reference back at the important concepts within the paper, with the benefit of the reader having hopefully acquired some familiarity with the work at large by the time they reach this section.\n\nWQ4: ln220 after Lemma 5.1 provides some valuable insights, especially with respect to the ‘unexplained variance’, but the statement feels too shallow without a concrete equation defining what is ‘unexplained variance’ for instance. I would like to urge the author to make it more concrete in order to increase the quality of that insight and the corresponding lemma.\n\n## Clarity :\n\nWC1: meaningfulness and ‘similarity to natural language’ are often talked about together (e.g. 44-45, 50-51), without having been clearly related to each other before. I understand that it may seem obvious that the adjective ‘meaningful’ means ‘meaningful to human beings or from the comparison with human languages’ and therefore a ‘meaningful’ emergent language should have high similarity with human/natural languages, but the general audience might appreciate a warning on that association. It also offers an opportunity for the paper to address concepts and related work around (human) alignment, which I assume would increase its significance. This is a rabbit hole that can go as far as the work of [10] (and [11] that instantiates it in the context of EmeCom).\n\nWC2: ln178-179 : the paper states that in EmeCom papers 'the discrete message space is usually much smaller than the input space', which I think is a misrepresentation and it also loses an important nuance regarding the impact of the communication channel's capacity, as studied in [5,8,9] for instance, where [8,9] finding that a ‘restrictive’ communication channel  is a driver of compositionality in the emergent language, when using a REINFORCE-like algorithms with symbolic stimuli, in contrast to [5] finding the opposite when using the (Straight-Through) Gumbel-Softmax approach with visual stimuli: overcomplete communication channel (where the max sentence length is increased) yields higher compositionality than complete ones. This latter is very relevant to the paper given its usage of Gumbel-Softmax approach and visual stimuli (MNIST).\n\nMy point is that this sentence in the paper occludes and misrepresents an important inquiry in EmeCom, and I would urge the authors to reformulate appropriately.\n\nWC3: the term 'unrestricted' is defined in ln148 and ln157, but I fail to understand what it means exactly in this context and when is its opposite considered ?\n\nWC4: In Appendix C.2 Lemma proof, the derivations below ln533 are ambiguous to me, especially the third line. I would like to invite the authors to make all of their proofs easier to understand by explaining what property or equation is used to reach each derivation. My point is that since there is no space restriction in the appendices, the paper would be made easier to understand by putting as many explanations as possible. I would hope for us to break away with this legacy/bias of taking ‘mathematical derivations being difficult to read’ as a badge of quality or importance of the derivations presented (assuming that this is the implicit bias that was at play here).\n\nWC4: Theorem 5.2 requires $\\Theta$ to contain ‘at least one semantically consistent protocol’. This requirement is surprising to me. It would increase the clarity of the paper if some insights or motivation for it could be provided, please?\n\nWC5: Acronym ‘KKT’ used in ln567-568 without definition.\n\nWC6: Equations could be more easily reference with numbering in all cases.\n\n## Significance :\n\nWS1: missing citation for the 'classification discrimination' setting (and a lesser extent the 'supervised discrimination' context) defined in ln169-172 to [1] that defines it as an object-centric (as opposed to stimulus-centric) and descriptive variant, highlighting it is first instantiated by [7]. It is later investigated in [6]. Adding those references would show the reader that those ideas have some precedence and are of value to the field of EmeCom, and provide adequate attribution.\n\nWS2: Experimenting with MNIST is not motivated in the paper, and not really in line with previous works in EmeCom, for instance because (i) it does not have much structure, and (ii) it does not have annotations of different attributes and values that would allow compositionality(/disentanglement) metrics to be computed, which is of concerned here.\n\nThus, I would hope for the paper to propose some experiments with symbolic stimuli for starters, following the work of [paper 7], or with a (Colored) dSprites dataset [13 - similarly to [14]] or 3dshapes dataset [12 - visual dataset with a lot of structure, i.e. up to 6 attributes], or the simplified dataset from [7, reproduced in an open-source version in 6].\n\nMoreover, in order to evaluate systematic generalisation and how it relates to semantic consistency, the experiments should use a train/test split strategy as shown in [paper 7] for symbolic stimuli, or [5 Sec. 3, 6] to be more in touch with dSprites and 3dshapes-like visual datasets.  \n\nWS3:  the paper does not provide a clear-cut metric for semantic consistency despite the clear definition in Definition 3 (ln195). ln302 does mention using message variance and points to Appendix E.2, but I think the paper would be made more impactful and clearer if this had been discussed around Definition 3 too and made very explicit. The same goes for spatial meaningfulness (ln307) despite definition 4 in ln257 not mentioning the related metric in a clear-cut fashion. My point is that I think that the paper would be more significant by providing very clear metrics that the community can adopt. Obviously, open-sourcing the codebase to compute those metrics would go a long way in that direction.\n\nWS4: Following up on WS3, in order to build a bridge between the previous works on compositionality in EmeCom **as a proxy for human-like communication/naturalness** (as acknoeledge in Section 2.2), and the current paper that proposes to evaluate human-like communication using the proposed definition of semantic consistency, I would have expected the paper to report some correlation metrics between relevant compositionality metrics (topsim, and posdis [paper 7], at least, and possibly the refined speaker-centred posdis from [6] at best) and a proposed semantical consistency metric (e.g. explained variance). These correlation measures would enable the community to reflect on the extent to which the compositionality-as-proxy-for-naturalness approach is lacking in terms of semantics and spatiality.\n\nMy point is that the current contribution is interesting and its qualitatively value has been well-discussed, but it is unclear how it relates to the previous works in a quantitative way. For instance, when considering quantitative results, are semantic consistency and compositionality just two sides of the same coin or do they capture really different aspects of human-like communication?\n\nWS5: Following up on WS2 and WS4, another valuable contribution that would allow comparison with previous works in EmeCom could be to evaluate the systematic generalisation of the resulting communication protocol and measure their correlation with a semantic consistency metric in order to clarify the extent to which semantic consistency aligns with systematic generalisation, which has been of great interest in EmeCom.\n\nThis would increase the significance of the work presented here, but it might be slightly out of the scope. In any case, I would like to inquire on the author’s perspective: why not having conssidered it so far? What kind of correlations would you expect? Can the formal setup provided in the paper and the concept of optimality it relies on say anything about systematic generalisation, please?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for this insightful and detailed review. We are glad you have found our theoretical contributions novel and valuable. The papers that you have mentioned, especially regarding compositionality and input representation, are indeed relevant to our core contributions, and we will add references to them in our next revision.\n\nWe will now address your comments individually:\n\nSS2: We agree that comparing our definitions to existing ideas is important. An interesting connection between our definitions and Topographic Similarity (topsim) is that while topsim considers the relationship between every pair of inputs, our definitions only consider pairs that correspond to similar messages. The latter follows an intuitive asymmetry: inputs with similar messages are expected to be similar, but inputs with dissimilar messages do not have to be different. Further discussion will be added in the next revision.\nOn the empirical front, new experiments are described in the response for comment WS2, including correlation with topsim, posdis, bosdis, and speaker-posdis.\n\nWO1, WQ1, WQ2: The missing citations will be added in the next revision. Thank you for the references.\n\nWQ3: Section 2.3 will be expanded in the next revision.\n\nWQ4: This explicit equation for explained variance will be added to the relevant paragraph: $\\text{Var}_{m \\sim S(X)}\\mathbb{E}[X | S(X)=m]$\n\nWC1: This is a good observation, which aligns with a central theme in our paper: distinguishing \"solves the task\" from \"has (consistent) meaning.\" In the next revision, we will clarify that \"meaningful\" refers to human-recognizable properties.\n\nWC2: We apologize for the mischaracterization. Our framework is mainly applicable to setups with a small-capacity messaging channel, where inputs necessarily get mapped to the same message. Nonetheless, research investigating larger message spaces is an important part of EC literature, including alternative types of messages such as quantization or explicit codebooks. We will reformulate the relevant paragraph accordingly.\n\nWC3: The term 'unrestricted' means that the agent can assume any function from its domain to its range. This assumption allows closed-form, interpretable solutions for synchronized receivers. However, it is unrealistic in the context of natural language, as physical and mental limitations play an important role in shaping human communication. Removing this assumption motivated the second part of our paper (dealing with spatial meaningfulness), where agents are not assumed to be unrestricted.\n\nWC4(1): We apologize for any confusion and definitely did not intend for the proofs to be hard to read. The referred-to third line is the law of total expectation, where the events are the assignment to message m. We have made efforts to ensure all proofs are written clearly and rigorously, so their correctness is easy to assert. We will add additional explanations for derivations in our revision.\n\nWC4(2): This assumption simply prevents the degenerate case where every available communication protocol is semantically inconsistent, which would render every optimal solution inconsistent regardless of the objective function.\n\nWC5: The KKT conditions characterize optimal solutions to convex problems. We will add the appropriate citation [2].\n\nWC6: Thank you for the suggestion, we will add equation numbering to our lemmas and to key equations throughout the proofs section for easier reference.\n\nWS1: The missing citations will be added in the next revision. Thank you for the references.\n\nWS2: We fully agree with your comment. The empirical analysis will better support our findings if we experiment with more sophisticated data. With this in mind, we have conducted additional experiments on a special version of the Shapes dataset [1] where each input is an image of an object with several sampled attributes (shape, color, position, rotation, and size). The multiplicity of attributes allowed us to calculate compositionality/disentanglement measures from the literature, including your suggestions. The results are appended to the general author rebuttal (and will be added to the paper in the next revision).\n\nThe results show a significant gap in semantic consistency (lower message variance is preferable), on par with the previous evaluation on MNIST. Additionally, topographic similarity strongly favors the reconstruction setting (0.37, versus 0.08 for discrimination), confirming a positive relationship between semantic consistency (as measured by message variance) and compositionality (as measured by topsim). We also report correlations between every pair of measures, taken over 10 runs (5 for each task). These correlations should be considered with caution as they are affected by the inclusion of both tasks for calculation. We may report separate tables for each game type to avoid this issue.\n\nWS3: We prefer to keep the theoretical and empirical parts separate to avoid splitting attention. However, it is crucial to present the metrics before the results, so we restructured the empirical section to mention the metrics before introducing any results. As for the codebase, it will be open-sourced with the publication, and you can access it now via the zip attached to the original submission.\n\nWS4: See our responses to WS2 and SS2.\n\nWS5: Systematic generalization is not included in our theoretical framework. Given our recent results, we might expect semantically consistent protocols to generalize better to unseen attribute values, as the relationship between topsim and systematic generalization has sometimes been shown to be positive.\n\nWe greatly appreciate your space-saving tips. We will try to implement them in the next revision as well.\n\n[1] Kuhnle, Alexander and Copestake, Ann. \"ShapeWorld - A new test methodology for multimodal language understanding\". arXiv preprint arXiv:1704.04517\n\n[2] Bertsekas, Dimitri P. \"Nonlinear programming\". Journal of the Operational Research Society.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We greatly appreciate the time and effort you have dedicated to evaluating our paper.

We have submitted individual responses to your reviews. To this message we have attached a pdf with new results on the Shapes dataset, along with visual illustrations of the trained agents' performance in each of the tasks.

Meta-review: This paper investigates two goal-agnostic properties/prerequisites for meaningful emergent communication and supports the theoretical findings with results across discriminative & reconstructive tasks on the MNIST dataset, as proof-of-concept. The submission initially received mixed reviews from the four reviewers. All reviewers appreciated the well-written and interesting theoretical results that reveal fundamental properties of emergent communication. Beyond requesting clarifications, the reviewers raised concerns about inadequate experimental validation, missing references, and unclear mathematical formalization. 

In the discussion phase, the authors addressed many of these concerns by adding experiments on a variation of the Shapes dataset, as suggested by reviewer o4TS, and providing detailed responses to questions from the reviewers. After the rebuttals, reviewers XigD and o4TS increased overall ratings, with o4TS indicating a move to a strong accept. Reviewer TFDu & cajw maintained their inclination to accept & reject, respectively.  

The AC recommends acceptance and agrees with the majority of reviewers -- this work presents a novel theoretical understanding of emergent communication. While the committee has some residual concern regarding the insufficiency of empirical results, the paper’s current contributions hold significant value and are reasonably complete in theory, especially within the scope of NeurIPS, which recognizes new approaches. The authors are encouraged to improve the draft for the final version with a better discussion of spatial meaningfulness results (XigD, cajw), scalability and context of large-scale models (XigD, cajw), rigorous mathematical formalization (TFDu), and adding relevant citations & new experiments (o4TS) -- based on the constructive discussion between the authors and the reviewers.
Decision: Accept (poster)