Emergence of shared sensory-motor graphical language from visual input
Authors: Yoann Lemesle
Tristan Karch
Clément Moulin-Frier
Romain Laroche
Pierre-Yves Oudeyer
Venue: Submitted to ICLR 2023 (2023)
Keywords: Emergent Communication
Visual Communication
Sensory-motor communication
Contrastive Learning
Language Game

Abstract: The framework of Language Games studies the emergence of languages in populations of agents. Recent contributions relying on deep learning methods focused on agents communicating via an idealized communication channel, where utterances produced by a speaker are directly perceived by a listener. This comes in contrast with human communication, which instead relies on a sensory-motor channel, where motor commands produced by the speaker (e.g. vocal or gestural articulators) result in sensory effects perceived by the listener (e.g. audio or visual). Here, we investigate if agents can evolve a shared language when they are equipped with a continuous sensory-motor system to produce and perceive signs, e.g. drawings. To this end, we introduce the Graphical Referential Game (GREG) where a speaker must produce a graphical utterance to name a visual referent object consisting of combinations of MNIST digits while a listener has to select the corresponding object among distractor referents, given the produced message. The utterances are drawing images produced using dynamical motor primitives combined with a sketching library. To tackle GREG we present CURVES: a multimodal contrastive deep learning mechanism that represents the energy (alignment) between named referents and utterances generated through gradient ascent on the learned energy landscape. We, then, present a set of experiments and metrics based on a systematic compositional dataset to evaluate the resulting language. We show that our method allows the emergence of a shared, graphical language with compositional properties

Under review as a conference paper at ICLR 2023
EMERGENCE OF SHARED SENSORY-MOTOR GRAPHICAL LANGUAGE FROM VISUAL INPUT
Anonymous authors Paper under double-blind review
ABSTRACT
The framework of Language Games studies the emergence languages in popula- tions of agents. Recent contributions relying on deep learning methods focused on agents communicating via an idealized communication channel, where utterances produced by speaker are directly perceived by a listener. This comes in contrast with human communication, which instead relies on a sensory-motor channel, where motor commands produced by the speaker (e-g. vocal or gestural articula- tors) result in sensory effects perceived by the listener (e.g. audio or visual). Here, we investigate if agents evolve shared language when equipped with con- tinuous sensory-motos system to produce and perceive signs, c.g. drawings. To this end, introduce the Graphical Referential Game (GREG) where speaker must produce a graphical utterance to name a visual referent object consisting combinations of MNIST digits while a listener has select the corresponding object among distractor referents, given the produced message. The utterances are drawing images produced using dynamical motor primitives combined with a sketching library. To tackle GREG we present CURVES: multimodal contrastive deep learning mechanism that represents the energy (alignment) between named referents and utterances generated through gradient ascent on the learned energy landscape. We, then, present set of experiments showing that our method al- lows the emergence of a shared, graphical language that generalizes to feature compositions never seen during training. We also propose a topographic metric investigate the compositionality of emergent graphical symbols. Finally, conduct an ablation study illustrating that sensory-mote motor constraints are required yield interpretable lexicons.
INTRODUCTION
Understanding the emergence and evolution of human languages is significant challenge that has involved many fields, from linguistics to developmental cognitive sciences (Christiansen & Kirby) 2003). Computational experimental semiotics (Galantucci & Garrod 2011) has seen some success in modeling the formation of communication systems in populations artificial agents (Cangelosi & Parisi 2002 Kirby al., 2014). More specifically, Language Game models (Steels & Loetzsch, 2012), have been used to show how a population agents can self-organize a culturally shared lexicon without centralized coordination. Given the recent successes of artificial neural networks in solving complex tasks such as image classification (Krizhevsky et al., 2012 He et al. [0155] 2016 Dosovitskiy al. 202T) and natural language understanding (Devlin
2019 Radford al.
2019 Brown et al. 2020), many works have leveraged them to study the emergence of communica- tion groups of agents (Lazaridou & Baroni 2020), mainly using multi agent deep reinforcement
learning and language games (Nguyen et al. 2020; Mordatch & Abheel] 2018 Lazaridou et al.)
2018 Portelance et al. 2021] Chaabouni et al. (021). These advances have made possible to scale up language game models to environments where linguistic conventions are jointly learned with visual representations of raw image perception, as well as to environments where emergent communication is used a tool to achieve joint cooperative tasks (Barde al. 2022).
So far, most of these methods have considered only idealized symbolic communication channels based on discrete tokens (Lazaridou et al., 2017 Mordatch & Abbeel, 2018 Chaabouni al.) 2021) or fixed-size sequences word tokens (Havrylov & Titov] 2017] Portelance et al. 2021). This predefined means of communication is motivated by language's discrete and compositional nature.

---

Under review as a conference paper at ICLR 2023
But how this specific structure emerge during vocalization drawing, for instance? Although fundamental in the investigation the origin of language (Dessalles) 2000) Cheney & Seyfarth 2005 Oller et al.] 2019), this question seems to be neglected by recent approaches to Language Games (Moulin-Friee Frier & Oudeyer 2020). We, therefore, propose to study how communication could emerge between agents producing and perceiving continuous signals with constrained sensory- motor system.
Speaker Referent FiS
Speaker Contest Rs
SPEAKER
sample
by
prodhos
command
Game
=?
Matener Context Rg
selects
peroeived by
LISTENER
Listener Referent
V.
perocived by
During the game, the speaker's goal produce motor command e that will yield an utterance in order denote referent sampled from context R.s. Following this step, the listener needs to interpret the utterance order guess the referent it denotes among context Rg. The game is success if the listener and the speaker agree on the referent (rz = rs).
Figure 1:
The Graphical Referential Game:
Such continuous constrained systems have been used in the cognitive science literature as mod- els of sign production to study the self-organization of speech in artificial systems (de Boer 2000) Oudeyer 2006 Moulin-Frier et al., 2015). In this paper, focus
drawing sensory-motor
system producing graphical signs. The sensory-moto system made of Dynamical Motor Primi- tives (DMPs) (Schaal 2006) combined with sketching system (Mihai & Hare 2021a) enabling the conversion of motor commands into images. Drawing systems have the advantage of producing 2D trajectories interpretable by humans while preserving the non-linear properties of speech models, which were shown to ease the discretization of the produced signals (Stevens) 1989 Moulin-Frier al., 2015). We introduce the Graphical Referential Game: variation of the original referential game, where a Speaker agent (top of Figure 1 has to produce a graphical urterance given a single target referent while a Listener agent (bottom of Figure I has to select an element among context made of several referents, given the produced utterance (agents alternate their roles). In this set- ting, we first investigate whether population of agents can converge on an efficient communication protocol to solve the graphical language game. Then, evaluate the coherence and compositional properties the emergent language, since is one of the main characteristics of human languages.
Early language game implementations (tteels 1995 2001 achieve communication convergence by using contrastive methods to update association tables between object referents and utterances. While recent works use deep learning methods target high-dimensional signals they do not explore contrastive approaches. Instead, they model interactions as a multi-agent reinforcement learning problem where utterances outcomes of the games as the reward signal (Lazaridou et al. 2017). .. In the meantime, recent models leveraging contrastive multimodal mechanisms such CLIP (Radford impressive results in modeling associations between images and texts. Combined with efficient generative methods (Ramesh et al. (02T). they can compose textual elements that are reflected in image form the composition of their associated visual concepts. Inspired by these techniques, propose CURVES: Contrastive Utterance-Re Referent associatiVE Scoring, an algorithmic solution to the graphical referential game. CURVES relies on two mechanisms: 1) The contrastive learning of an energy landscape representing the alignment between utterances and referents and 2) the generation of utterances that maximize the energy for given target referent. We evaluate CURVES in two instantiations of the graphical referential game: one with symbolic referents encoded by one-hot vectors and another with visual referents derived from the multiple MNIST digits (LeCun et al.
actions, and agents are optimized with policy gradients, using the
al., 202I) have achieved
2

---

Under review as a conference paper at ICLR 2023
1998). We show that CURVES converges to shared graphical language that enables population of agents not only to name complex visual referents but also to name new referent compositions that were never encountered during training.
Scope. The idea of using a sensory-motor system to study the emergence of forms of combi- natoriality in language dates back to methods investigating the origins of digital vocalization sys- tems (de Boer] 2000; Oudeyer] 2005] Zuidema & De Boer, 2009). Such studies were conducted in the context of imitation games at the level of phonemes to observe the formation of speech utter- ances (syllables, words) that were systematically composed from lower-level meaningless elements (phonemes). This corresponded to the first level of compositionality within the notion of duality of patterning (Hockett & Hockett 1960). Yet. these works did not consider referential games and did not study agents' ability to compose meaningful words to denote referents, i.c. they did not address the second level of the duality of patterning.
One of the goals of emergent communication research is to develop machines that can interact with humans. As result, variety of referential game approaches ensure that the emergent language is as close to natural language. This can be achieved by adding a supervised image captioning objective encourage agents use natural language in order solve their communicative tasks (Havrylov) & Titov| 2017;; Lazaridou et al.) 2017). Other methods use constraints such as memory restric- tions (Kottur et (a.7) to act as an information bottleneck to increase interpretability and compo- sitionality. While we purposefully chose a graphical sensory-motor system to ease the visualization of the emerging language, we do not inject prior knowledge or pressures to facilitate the emergence of an iconic language. Our produced utterances are completely arbitrary. This fundamentally dif- ferentiates our work from Mihai & Hare (2021b) that trains agents to communicate via sketches replicating the visual referents they name. Note also that their drawing setup does not include dy- namical motor primitives and utterances are directly optimized in image space. They, moreover, a decentralized approach.
allow gradients back- pook-prop from listener to speaker while Finally, they do not consider contrastive learning. To our knowledge, CURVES is the first contrastive deep-learnigg algorithm successfully applied referential game.
There is a large body of work exploring the factors that promote compositionally in emerging lan- guages (Kottur et al.) 2017 Li & Bowling 2019 Rodriguez Luna al., 2020 Ren al. 2020
Chaabouni et al. 2020 Gupta et al. 2020). In this context, a crucial question is how to actually mea- sure in the first place Mu & Goodman 202T). To this end, (Choi al. 2018) proposes measure communicative performances on unseen compositions of known objects as a way to evaluate compo- sitionality. However, has been shown that good performance in this test may be achieved without leveraging any actual compositionality in language (Andreas [019: Chaabouni et al. (2220). Thus, others instead compute topographic similarities (Brighton & Kirby 2006), measuring the correla- tion between distances in the utterance space (distance between signs) and distances in the referents space (such as the cosine similarity between the embeddings of objects) (Lazaridou et al. 2018). In this paper we propose to do both and study 1) the generalization to unseen combinations of abstract features and 2) topographic measures based on the Hausdorff distances between utterances denoting composition and utterances denoting isolated features.
Contributions. This paper introduces:
The Graphical Referential Game (GREG): variation of the referential game to study the formation of signs from a graphical sensory-motor system.
CURVES: algorithmic solution to GREG, consisting of contrastive multimodal encoder
coupled with a generative model enabling the emergence of a graphical language.
A study of CURVES'S generalization performances compositions of features never seen during training in a simplified control setting and a more perceptually challenging one. A complementary analysis of the compositionality of the emerging graphical language measuring the Hausdorff distance between utterances denoting compositions and utterances denoting their constituents.
An ablation study measuring how the motion primitives of the sensory-mote system shape the emerging symbols.
3

---

Under review as a conference paper at ICLR 2023
2 PROBLEM DEFINITION
Graphical referential game. We consider a group of two agents playing fixed number of refer- ential games, each time alternating their roles (speaker or listener). During game, first present a context I of n objects, called referents to speaker S and a listener L. At the beginning of each game, the target r* R is assigned to the speaker. Given this target referent r", produces an utterance (u) to designate it. Based on the produced utterance u, L selects referent (f) in R. The game outcome success if the selected referent (f) matches the target r*.
Referents. Referents compositions of orthogonal vector features (one-hot vectors). Given a set of m orthogonal features Fm. we define the set of all possible referents as R,, (Eres fls F. The subset of referents made of exactly le features thus: R = (es flS C Fm. = = our experiments, we fix m
From these orthogonal referents, we propose to generate objects made of digit images sampled from the MNIST dataset (LeCun et al. 1998). More precisely, we define the stochastic mapping T RRn - Rum that maps each feature f E Fm to digit class in the MNIST dataset. For each feature a referent, we sample a random instance from the corresponding class and randomly place it on 4 x 4 grid such that no number overlap. Note that the listener and speaker can perceive different realizations O in this case, we say that they see different perspectives of referents. More precisely, the speaker perceives the context as Rs and its target r* as rīs. Similarly, the listener perceives the context R as RG, and selects a referent & among it.
We this formalism instantiate three settings of the Graphical Referential Game (GREG):
one-hor: where referents are one-hot vectors r E Rmn
visual-shared: where referents MNIST digits Rm and agents share the same per- spective: Rş R2-
visual- unshared where referents MNIST digits Rm and agents have different per- spectives of referents in their contexts Rs + RL-
Sensory-motor drawing system. Utterances are produced by a sensory-motor system M R +u Dx mimicking an arm drawing sketches displayed in Figure 2 The arm mo- tion is derived from Dynamical Motor Primitives (DMPs) (Schaal (006). The DMP is parametrized by a command vector c E 1R It converts e into a 2-dimensional drawing trajectory made of 10 coordinates T {r; i=0...... This trajectory is then fed to Differentiable Sketching model (Mi- hai & Hare 2021a) generating an D D image (in our implementation,
52). See Suppl. Section A. for additional implementation details of the Sensory-motor drawing system.
fg
T
R
a
(b)
Figure 2: (a) Sketching mensory-motor system: The sensory-motor system imitates a robotic arm drawing a sketch on 2D plan. DMPs first convert continuous command into a sequence of coordinates T. This trajectory then rendered 52 x 52 graphical utterance thanks to differentiable sketching library. (b) Referent transformation: An example of a one-hot context It being transformed into two contexts Rg and R2 by the stochastic transformation 4. The contexts different perspectives of the objects.
Objectives. This paper investigates how group of two agents agree shared compositional language to denote referents, given a continuous sensorimotor system to produce utterances. Beyond the game's success, evaluate the emerging language along two dimensions.
Coherence. First, we measure the coherence of the emerging lexicon. As utterances are 2- dimensional paths, compute similarities using the Hausdorff distance it is known capture
4

---

Under review as a conference paper at ICLR 2023
geometric features of trajectories, in particular, their shape (Besse al. 2013). The Hausdorff dis- tance dæ is the maximum distance from any coordinate in a trajectory to the closest coordinate in the other: du T2) max(supeer, T2), d(Ti, In particular, we compute the following metrics.
Agents Coherence (A-coherence): For a given referent with the same perspective for all agents, measure the mean pairwise similarity between each agent's ' utterance.
Perspective Coherence (P-coherence): For given agent and given referent r, measure the mean pairwise similarity between utterances produced from different perspectives Referents Coherence (R-coherence): For given agent, measure the mean pairwise sim- ilarity between utterances produced for different referents.
Compositionaliiy. The second dimension of our evaluation explores the compositional properties of the emerging language. To this end, we first evaluate the generalization performances of our group to compositional referents never seen during training. More specifically, we train agents on Ruaain = R2 (referents made of one feature) and test them Restt R2 (referents made of two features). For visuals about compositional referents, see Suppl. Section A.2 We use the success rate SR to monitor the performances. However, satisfactory success rate on this testing set does not necessarily imply that the emerging language is in fact compositional (Chaabouni et al. 2020).
To complement this analysis, we estimate what extent utterances denoting compositional referents are actually made of utterances denoting their constituents. To this end, we introduce topographic score based on the Hausdorff distance p that quantifies how utterance denoting compositional referent made of feature and (u(ri;)) actually closer to utterances denoting isolated features urre) u(r;) than the utterance naming other compositional referents (u(ray), i, y + j). For a detailed derivation of metric p. see Suppl. Section A.3
CURVES CONTRASTIVE UTTERANCE-REFEREE REFERENT ASSOCIATIVE SCORING
CURVES is an mergy-basedd approach that relies on two mechanisms:
1. The contrastive learning of energy landscape E(r; u) is defined as the cosine similarity between utterance and referent embeddings.
2. The generation of utterance that maximizes the energy for a given target referent rš--
Agents modules and interactions. Each agent A E {A1, A2] perceives utterances and ref- erents using two distinct CNN encoders fA (for referents) and gA (for utterances | fA and 9A map referents and utterances in a shared d-dimensional latent space: fa(-, ()a) Rm + Rd and
ga(-, (a())0ga)= U + Idd such that A = fA(r) and ZnA = ga (u),,, as displayed in Figure 3 a). The agent then computes the energy landscape as: (A u) =
A given referential game unfolds as follows. Agents have randomly attributed roles, for instance, A is the speaker A + S and A2 is the listener A₂ < L. The speaker is given context Rs and target referent perceived as r to produce an utterance á intending approach the utterance ut that maximizes (rs(r', u). The listener observes á and selects referent in context Rg that maximizes Ey (r, ū):
argmax Es(rš. u) (utterance generation from speaker)
uell
argmax Fr, (r, û) (referent selection from listener)
reR,
The outcome of the game is then o = 1 [ where is a baseline parameter representing the mean success across previous games.
Contrastive representation learning in referential games. For a given context R, agents are ran- domly assigned their roles and play speaker agent successively selects each referent of the context Rg as the target rš. During interac- tions, the speaker collects data [(r], wf, o'))i=i,...., while the listeners observes ((u), (r,))r,,,..........
|R| games. During these n games, roles are fixed and the
'when referents are onc-hot vectors fA is a fully-connected network. Parameters for both encoders are given in Suppl. table
5

---

Under review as a conference paper at ICLR 2023
Ea(T, m)
Encadex SA
cos
3
Redocent
ΣA
Encoder
Knooder
Lat (5a)
(a)
(b)
Figure 3: (a) Agents's 's dual encoder architecture. Referents and utterances are mapped to a share latent space. The energy between referent and an utterance is computed as the cosine similarity between their respective embeddings. (b) Cosine similarity matrix update from collected samples. Agents compute the energy for referents and utterances collected to form squared matrix E.. During contrastive updates agents maximize blue circles and minimize white ones.
From the collected data each agent can compute the squared cosine similarity matrices Σ whose el- ements are (Ea)..j = (A(rA: w²) as shown in Figure 3)). Contrastive updates are then performed using the objective JA that applies Cross Entropy (CE) on the i-th row and i-th column of EA. ,) = [CE((Ea)),Im: e,) +
Im,;;
(2)
& being a one-hot vector of size n with value at index i. Depending on the role of the agent, JA instantiated either as Js (speaker) Jr (listener). Thus, the speaker updates its representation using the outcomes og of the games (reinforcing the successful associations while decreasing the unsuccessful ones):
minimize Oig Aag
oo:Js(Es.i)
On the other hand, the listener needs to make sure that the selection matches the speaker's ' referent Steels 2015) and hence always increases associations (no matter the games' outcomes):
minimize 8ig BoL
SrCr) i)
(4)
i=1
Note that in Eq. ri is the target referent perceived by the listener. This means that the speaker indicates the referent (as perceived by the listener) that they were naming the end of the game. This retroactive pointing mechanism was employed in both early language game implementations (Steels
& Kaplan 1999) and more recent ones (Chaabouni et al. 2020 Portelance al. 2021..
Speaker's utterance optimization. We distinguish two utterance generation strategies:
The descriptive generation: in which the speaker agent only considers the target referent rss to produce an utterance that maximizes the cosine similarity between the embeddings of r's and an utterance produced by our sensory system u Mc(c from motor command c. Since M is fully differentiable, we inject the sensory-motor constraint in equation] [ and seek for the optimal motor command c* using gradient ascent:
c² = argmax F(r',, M(e))
(5)
cEEr
The discriminative generation: in which the speaker also perceives the context Rg dur- ing production. This is achieved by finding the motor command that minimizes the cross entropy given target referent r's and its context Rg:
c* argmin CE(as, (rg)
(6)
CERP
where σ is the vector with coordinates ds; vector of size [Rsl with value at the position of r's in Rs. This discriminative generation process is only used at test time when investigating CURVES's generalization capabilities.
[E(r', ((c))) and erg is the one-hot

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS AND RESULTS
This section focuses first on CURVES's training dynamics agents interact in GREG before show- casing its ability to generalize compositional referents that were never seen during training. We, then, evaluate the compositional structure of the emerging graphical language by providing visuals of utterances and computing topographic scores defined Section Each of these studies carried out with one-hot, shared-vise visual, and unshared-visut referents explained in Section 2 We finally investigate the impact of motion primitives on the emerging language. Training and testing metrics correspond to the mean and standard deviation computed from training pairs of agents 10 seeds.
Do agents converge to a shared graphical language? T the training performances of group of two agents interacting in GREG. For each referent type, the group reaches perfect success rate of SR 1. Moreover, a group starts to converge when inter-agent and inter antrrperspeccive coherence distances decrease. This correlation is proof of emergent communication that agents start agreeing on signs to denote referents. Finally, the constant (for one-hot referent) and increasing (for visual referents) values the R-coherence suggest that agents use distinct signs to name referents.
it indicates
14
1.0
6..
Headdoff
A-Coherence
8.
2.0
2.6
C-Chhrrncce
13
13
200
439
cdo
1000
2003
4005
6coo
8000
10002
2003
4085
ecee
aoco
a2002
Iterations (5 gemes)
Iterations (3 games)
iterations qames)
(a)
(b)
(c)
Figure 4: Training success rate (SR) and Coherence distances (a) one-hot referents (b) risual-shared refer- ents visual-unshared referents.
An example of an emerging lexicon describing visual referents produced by agents trained on un- shared perspectives can be visualized in F Other visualizations for one-hot and shared visual referents are available in Suppl. Section B.2 We also provide illustrations of P-coherence in Suppl.
Section B.3
RED1
RE13
RI2I
R131
RE4]
L
AD
A
Figure Instance of an emerging lexicon. Utterances are produced by a pair agents trained with unshared perspectives (1 seed). The perspective for each referent is chosen randomly.
Are agents able generalize to compositional referents? Table [ exposes the generalization performances of group of agents evaluated on referents r E Rg. During an evaluation, the context exhaustive and contains all the combinations rates to a random baseline where the listener always selects the referent é.. randomly no matter the utterance (SRyandom = 0.1). We also introduce a I-feature baseline where the speaker produces an utterance u that only denotes one of the two features contained in r and the listener randomly selects one of the four combinations containing the communicated feature (SR 0.25). The success rates for all referent types are significantly higher than the baseline values suggesting that agents are indeed able to communicate about compositional referents. Generalization performances nearly perfect with one-hot referents but they decrease in visual settings. This performance gap can be explained by the extra difficulty of adding inter enterrperspective variability to the multi-agent interaction dynamic during the contrastive learning of referent representations. The better success
features: | 10. We compare the success

---

Under review as a conference paper at ICLR 2023
rates obtained in auto-learning (where single agent plays both the speaker and the listener roles) provided in Suppl. Section B.1 seem to corroborate this hypothesis. Surprisingly, we observe that success rates for descriptive (Eq. 5 and discriminative (Eq()) generation very similar. This sug- gests that optimizing utterances so as to minimize their energy between non-targeted compositional referents (r E R + r*) does not improve generalization performances.
Referents
Descriptive SR
Discriminative
One-hot
0.99 = 0.01
0.99 = 0.01
Visual-shared
0.57 = 0.04
0.56 - 0.03
Visual-unshared
0.39 0.02
0.40 = 0.02
Table 1: Generalization performances. referents R5 for both generative (Eq.
Success rates evaluated on exhaustive context |R] 10 with and discriminative (Eq utterance generation.
Is the emerging language compositional? terances we propose the topographic maps displayed in Figure 6 Each point in a topo- graphic map is an utterance naming compositional referent E R3 and has coordinate (du (u(r:). du (u(r)), If utterances naming the composition of two features are indeed the compositions of the utterances used to denote each of the isolated features, them to land in the bottom left of the topographic maps. Figure 6 shows that some ut- terances compositional referents are indeed close in Haussdorf distance to the utterances denoting the isolated constituent features (Figure ((b)) but others not (Figure ((a)).
To investigate the compositionaliyy of
expect
Hausdorif Distance (R[2))
1.2
MI-11
Hausdorit Distance (R(31)
1.0
1.0
0.8
0.4
0.2
0.2
022
1.0 Hausdorff Distance (R(2))
Hausdorff Distance (R[1])
(a)
(b)
Figure 6: Topegraphic map examples for single seed unshared-vistee referents setting (a) Correspond-
ing to the worst topographic score = -00 -0.113 (b) Corresponding to the best topographic score = 0.203. Each utterance names compositional referent and is colored in blue if it contains feature i, orange if it contains feature j. green if contains both, and black if it contains none.
RIZ1
Unfortunately, the feature maps do not allow to draw strong conclusions about the composition properties the emerging language. It is hard to tell if agents are indeed composing utterances or the Haussdorf dis- tance simply does not capture compositionality. This seems to be verified by additional topographic maps pro- vided in Suppl. Section B.4 In particular, the topo- graphic maps for one-hot referents (Figure 19 indicate that strong generalization performances can be achieved by producing utterances that are not necessarily close to the isolated feature utterances. This difficulty in evaluat- ing compositionality can be experienced visually thanks to Figure 7 which displays a matrix of composition for unshared-vie visuull referents. More instances of composi- tion matrices are available in Suppl. Section B.5
R291
RE3
R22
R33
R(4]
Figure 7: Matrix of compositions. Blue frames represent utterances generated for perspective in R!. other utterance denote corresponding compositions R2
Are representations compositional? Finally, compositionality is visually hard to analyze in graphical space, it seems be much more apparent in the utterance and referent embedding spaces.
8

---

Under review as a conference paper at ICLR 2023
Figure shows that the embeddings for compositional referents well the embedding of the ut- terances naming them are indeed close to the embeddings of their constituents. This is not surprising
since this is the space in which our energy landscape is learned.
Referents Embeddings
Utteronces Embeddings
R221
MAI
T-sne of utterance and referent embeddings. Embeddings are computed for 100 perspectives. Training conditions unshared visual referents. Additional t-snes are provided in Suppl. Section B.6
Figure 8:
What is the impact of sensory-motor constraints on the emergent lexicon? We conduct an ab- lation study where the speaker directly optimizes the utterance in image space (without = M in Eq. S & 6 As reported by the training and testing performances in table 2 the pair of agents succeed during training and their generalization performances ances with DMPs. Crucially, the emerging lexicons, displayed in figure g look like noisy pixel maps and therefore have geometrical structure. This prevents any coherence or compositionality analy- sis and reinforces the interest in incorporating motor constraints into the utterance generation system.
par with pairs producing utter-
[(
SRain One-hot 0.99 + 0.01 Visual
SR 0.96 + 0.02
A
0.99 + 0.01
0.41 + 0.02
Table 2: Training and generalization suc- cess without DMPs. Utterances are gener- ated in descriptive mode, and visual refer- ents from different perspectives.
Figure 9: Emerging lexicon without motion primitives. Utterances naming referents with unshared perspectives.
5 DISCUSSION AND FUTURE WORK
This work formalizes GREG: a new referential game where two agents must communicate via continuous sensory-motor system imitating robotic arm drawing sketches. To tackle GREG, propose CURVES: a contrastive representation learning algorithm inspired by early language game contrastive implementation that scales to high dimensional signals. CURVES allows group of two agents two converge on a shared graphical language in contexts where referents are one-hot vectors images of MNIST digits. The representations that agents learn enable them communicate about compositional referents never encountered during training. Our ablation study shows that motor constraints the utterance generation system are required to facilitate the emergence of structured lexicons. Despite the visualizable nature of graphical signs, compositions of utterances are hard draw
identify. Our proposed analysis based on the Hausdorff distance could not allow systematic conclusions. On the other hand, compositions are salient in the space of embeddings.
Future work may look into finding other metrics or evaluation strategies to investigate the composi- tion of utterances in more depth. An analysis of the impact of the sensory-motor constraints on the topology of graphical signs could also provide valuable insight into the factors facilitating the emer- gence of a compositional graphical language. CURVES is agnostic to the modality used to represent utterances. As such, it could tackle other sensory-motor systems. The central element of CURVES lies in the contrastive learning of utterance-referent associations. In our implementation. utterances by maximizing this energy via gradient ascent. Much like CLIP opened many avenues for multi-modal generation, we could plug in more complex generative strategies such diffusion models (Rombach et al.) [02TT: (Sahani et al., (022). Finally, more realistic visual referents and the impact of training larger groups of agents generalization could be investigated in GREG.
optmize

---

Under review as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
We ensure the reproduciblity of the experiments presented this work by providing our code. Addi- tional information regarding the methods and hyper-parameters can be found in Suppl. Section [A.4 Information about our derived metrics can also be found in Suppl. Section A.3] We ensure to display the variance of our experimental results by using 10 random seeds, reporting the standard deviation.
REFERENCES
Jacob Andreas. Measuring compositionality in representation learning. ArXiv, abs/1902.0718..
2019.
Paul Barde, Tristan Karch, Derek Nowrouzezahrai, Clément Moulin-Frier, Christopher Pal, and Pierre-Yvel Oudeyer. Learning to guide and to be guided in the architect-builder problem. In International Conference Learning Representations, 2022.
net/
Philippe Besse, Brendan Guillouet, Jean-Michel Loubes, and Royer François. Review and perspec- tive for distance based trajectory clustering, 2015. /tLp 1508.
04904
Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence of topographic mappings. Artificial Life, 12:229-242, 2006.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, J. Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. ArXiv, abs/2005. 141665, 2020.
Angelo Cangelosi and Domenico Parisi. Simulating the evolution of language. In Springer London, 2002.
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni.
Compositionality and generalization in emergent languages. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 4427-4442, Online, July 2020. Association for Computational Linguistics. doi: 10. 11118/61/22220.acllll main.407. URL Shallattholooy.oor/222222 acl-maaa...........mmmmmmmm..mam 407
https:
Rahma Chaabouni, Fugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences of the United States of America, 118, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In Hal Daumé III and Aarti Singh (eds.), Pro- ceedings of the 37th International Conference on Machine Learning, volume 119 of Proceed- ings of Machine Learning Research, pp. 1597-1607. PMLR, 13-18 Jul 2020. URL https: Vproccedings.. mlr www.w..w..pp....mmmmm..m.m. wwwppprsssssp.. wwtps/vvl/////// chen20j. /tt. html
Dorothy L Cheney and Robert M Seyfarth. Constraints and preadaptations in the earliest stages of language evolution. 2005.
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Multi-agent Multi-agentt compositional communication learning from raw visual input. In International Conference on Learning Representutions, 2018.
URL
net/ forum? 2d=rknt2Be0-
Morten H. Christiansen and Simon Kirby. Language evolution: consensus and controversies. Trends in Cognitive Sciences. 777)3300-37, 2003. ISSN 1364-6613 doi: https///oi.orr//10 00066113644666130000001666666666
10

---

Under review as a conference paper at ICLR 2023
Bart G. de Boer. Self-organization in vowel systems. J. Phonetics, 28:441-4440 1611,46654 2000.
Jean-Louis Dessalles. Aux origines du langage. Hermès-science, 2000.
Jacob Devlin, Ming-Wer Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep
bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Language Technologies, Volume (Long and Short Papers), pp. 417 1-71-, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.1863//1/N19-111 1423. URL https:
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recogni- tion at scale. In International Conference on Learning Representations, 2021. URL https: //openreview. www.ee/fooommieemm CiiyiicbaNN cNTTY
Bruno Galantucci and Simon Garrod. Experimental semiotics: A review. Frontiers in Human Neuroscience, 5, 2011. ISSN 1662-5161. doi: 10.3389/fnhum.. 201 1.00011. URL https:
7/ 7/www. www.:ffootieesiiccommm wtt.t//erreiioorrtttttlleeeeeeee 3389 fnhum. 2011.00011
Abhinav Gupta, Cinjon Resnick, Jakob Foerster, Andrew Dai, and Kyunghyun Cho. Composi- tionality and capacity emergent languages. Proceedings of the 5th Workshop on Repre- sentation Learning for NLP, pp. 34-38, Online, July 2020. Association for Computational Lin- guistics. doi: 10.18653/1120022rmp/4m/pp/
rep14nip-1.5
Serhii Havrylov and Ivan Titov. Emergence of language with multi-agent games: Learning com- municate with sequences of symbols. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fer- gus, S. Vishwanathan, and Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:/ttts/sppcseetteeeeee ngs neurips. cC/ pddf
paper//211//5le//0022222 (ttp://22220440008080800000000 httpa//0000.....ccc.aa2333333333
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. 2015 IEEE International Conference on Computer Vision (ICCV), pp. 1026-1034, 2015.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 770-778, 2016.
Charles F Hockett and Charles Hockett. The origin of speech. Scientific American, 203(3):88-97 1960.
Simon Kirby, Tom Griffiths, and Kenny Smith. Iterated learning and the evolution of language. Current Opinion in Neurobiology, 28: 28:108-1144 114, 2014.
Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally" in multi-agent dialog. In EMNLP, 2017.
Alex Krizhevsky. Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convo- lutional neural networks. Communications of the ACM, 60:84 90, 2012.
Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. ArXiv, abs/2006.02411 2020.
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. In International Conference on Learning Representations, 2017.
URL /pLLitttps//ppeeeevvee
Eorum? witrrumiiddliNNNSSS...
Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Confer- ence on Learning Representations, 2018. URL httpp::/t///oo.o /tttt::://ppeeeevie. net wfrummrddd
HJGG11.-A
11

---

Under review as a conference paper at ICLR 2023
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86:2278-22 2324, 1998.
Fushan Li and Michael Bowling. Ease-of-teaching and language structure from emergent commu- nication. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Buc, E. Fox, and R. Gar- nett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL httpps://prroceedingss rewpiiiinnnneeruugggee neurips htt.://eeeeaaaaaae.rrrrroor/rprr
.ddf
Daniela Mihai and Jonathon S. Hare. Differentiable drawing and sketching. ArXiv, abs/2103.16194, 2021a.
Daniela Mihai and Jonathon S. Hare. Learning to draw: Emergent communication through sketch- ing. NeurIPS, 2021b.
Igor Mordatch and P. Abbeel. Emergence of grounded compositional language multi-agent pop- ulations. In AAAI, 2018
Clément Moulin-Frier and Pierre-Yvee Oudeyer. Multi-agent reinforcement learning as a compu- tational tool for language evolution research: Historical context and future challenges. ArXiv, abs/2002.08878. 2020.
Clément Moulin- Frier, Julien Diard, Jean-Luc Schwartz, and Pierre Bessière. Cosmo ("communicat- ing about objects using sensory- operations"): A bayesian modeling framework for study- ing speech communication and the emergence of phonological systems. Journal of Phonetics, 53: 5-41, 2015. ISSN 0095-4470. doi: http:///ooooor///ooooooooooooooa
URL https:
// //www. sc www.ssciencccccc rect com/ wwwpec.cooscccieeeeeee the cognitive nature of speech sound systems.
On
Jesse Mu and Noah Goodman. Emergent communication of generalizations. M. Ran- zato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances
in Neural Information Processing Systems, volume 34, pp. 17994-1880 Curran Asso-
ciates, Inc., 2021. URL
neurips. cc cc/ppaper/2221/iieee pdf
Thanh Thi Nguyen, Ngoc Duy Nguyen, and Saeid Nahavandi. Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications. IEEE Transactions on Cybernetics, 50:3826-3839, 2020.
D Kimbrough Oller, Ulrike Griebel, Suneeti Nathani Iyer, Yuna Jhang, Anne S Warlaumont, Rick Dale, and Josep Call. Language origins viewed in spontaneous and interactive vocal rates of human and bonobo infants. Frontiers in psychology. 10:729, 2019.
Pierre Yves Oudeyer. The self-organization of speech sounds. Journal of theoretical biology, 233 3:435 49, 2005.
Pierre-Yve Oudeyer. Self-organization in the evolution of speech. In Oxford Studies in the Evolu- tion of Language, 2006.
Eva Portelance. Michael C. Frank, Dan Jurafsky, Alessandro Sordoni, and Romain Laroche. The emergence of the shape bias results from communicative efficiency. In CONLL, 2021.
Alec Radford, Jeff Wu. Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation, 2021. URL https: //trr//a.vv...//////////.a....iv org/ abs/ /2ss/210..0 12092
12

---

Under review as a conference paper at ICLR 2023
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional lan- guages emerge in a neural iterated learning model. International Conference on Learning Representations, 2020. URL https: http://pppweeee..eeeeeeeoeeeeeee net. htt.:///www...ttoooooooooooaaaae
Diana Rodríguez Luna, Edoardo Maria Ponti, Dieuwke Hupkes, and Elia Bruni. Internal and exter- nal pressures on language emergence: least effort, object constancy and frequency. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4428 4437, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. (0.1863/vv12020 findings- findings-emnlp..3 emnlp.397. URL BtKLLhttts:://clanttii /ttp///caaaaataa woattholooy...ooo org. 2000 nd ngs emn .997
Robin Rombach, Blattmann, Domínik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. ArXiv, abs/21 12. 2021.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, Seyedeh Sara Mahdavi, Raphael Gontijo Lopes, Tim Salimans, Jonathan Ho, David Fleet, and Mohammad Norouzi. Photorealistic text- to-image diffusion models with deep language understanding. ArXiv, abs/2205.
11487, 11887220551117..... 2022.
Stefan Schaal. Dynamic movement primitives -a framework for motor control in humans and hu- manoid roboties. 2006.
Luc Steels and Frederic Kaplan. Situated grounded word semantics. Proceedings of the 16th International Joint Conference Artificial Intelligence Volume 2, IJCAI'99, pp. 862-867, San Francisco, CA, USA, 1999. Morgan Kaufmann Publishers Inc.
Luc L. Steels. A self-organizing spatial vocabulary. Artificial Life, 2:319-332, 1995.
Luc L. Steels. Language games for autonomous robots. IEEE Intelligent Systems, 16:16-28 12.16,22,00 2001.
Luc L. Steels. The Talking Heads experiment. Number in Computational Models of Lan- guage Evolution. Language Science Press, Berlin, 2015. doi: 10.17169/FUDOCS ddccumee.... 000000022455.
Luc L. Steels and Martin Loetzsch. The grounded naming game. 2012.
Kenneth N. Stevens. On the quantal nature of speech. Journal of Phonetics, 17(1):3-45, 1989. ISSN 0095-4470. doi: http://0bba////10000055555411110 htthttss::/ www.
sccincedi rect. com/ sci htt.://c.c.............ooooooooa
Willem Zuidema and Bart De Boer. The evolution of combinatorial phonology. Journal of Phonetics, 37(2):125-1444 2009.
13

---

Under review as a conference paper at ICLR 2023
SUPPLEMENTARY MATERIAL
This Supplementary Material provides additional derivations, implementation details and results. More specifically:
Section provides supplementary implementation details in the form of:
Images of testing set of visual referents;
Topographic score derivation;
Training procedures and hyperparameters: Pseudo-code.
Section provides supplementary results:
cottreeommeehnnionn generalization performances;
Additional Lexicons;
Utterances examples across perspectives illustrating coherence;
Topographic maps & scores;
Composition matrix examples;
T-SNEs of embeddings;
A SUPPLEMENTARY METHODS
A.1 SENSORY-MOTOR SYSTEM
Dynamic Motion Primitives. This subsection provides additional details about the implementation of Dynamical Movement Primitives use to produce 2-dimensional trajectories. Our drawing system consists of 2-dimensional system that mimics the motion of a pen in plan. Each of the x and positions the pen is controlled by DMP starting at the center of the image and parameterized by 10 weights. These weights are the parameters of the motion of one-dimensional oscillator that generates a smooth trajectory 10 points. The parameters of the two DMPs are given table
Parameter
Value
Number weights
10
Delta time
0.1
Number of points
10
Weights range
[-500, 500]
Position Init.
Table 3: DMP parameters for each of the two coordinate motions
Sketching Library. Trajectories obtained with the DMPs are then mapped to a 52x52 grid which is converted to an image with the raster and softor functions of the sketching library Mihai & Hare aa02 The drawing thickness parameter is fixed to le - 2.
14

---

Under review as a conference paper at ICLR 2023
A.2
TESTING SET
Figure 10 displays examples of compositional referents made of features.
3
/
2
D
1
2
O
4
3
Figure 10: Perspective instances of the testing set R2
A.3 TOPOGRAPHIC SCORE
To evaluate the compositionality of the emerging language define the topographic score:
Pij |[(O, h1))ll2 (1),,,)))l with k argmin (f.j) |/, hayll2)
is obtained by computing the Hausdorff distance between the utterances denoting compositional referents with respect to both the utterance denoting the single feature i (M(rr)) the one denoting the single feature (d (dz(u(r)) (u(r)): To derive our metric, we define groups of utterances denoting compositional referents.
u(ray) the utterances for referent made of feature and j.
u(rzj, x o i) the utterances denoting referent made by composing feature with any other feature different than
u(rig. y * j)) the utterances denoting referent made by composing feature with any other feature different than
u(rzy) the utterance denoting all other compositional referents in R.
and compute their Hausdorff distances to u(rx) and u(r)). As displayed in Figure 11 utterances u(riy) are compositional we expect them be the same time close u(r;) and close to u(r,) and hence to land in the bottom left corner of the distance graph. Moreover, they should be closer to the origin than u(rzz) and u(rsy)- To quantify to what extent it is the case we compute the barycenter of each group h4, h2, hy and hey and compute "how closer to the origin" is the compositional barycenter hu compared its closest barycenter using equation]
= de(u(ts).))
u(ry)) u
= dg(((rs).)
Figure Idealized mapping of utterances denoting compositional referents the plan representing distances to utterances naming isolated features and j.
15

---

Under review as a conference paper at ICLR 2023
A.4 TRAINING PROCEDURE AND HYPERPARAMETERS
Agents have twO separate encoders based on the same model architecture described in Table Each agent performs association updates with single step of gradient descent, using its own Adam optimizer with a learning rate of le To allow faster convergence, agents perform an association update between an abstract referent r and an utterance by using a batch of 64 perspectives ( From a cognitive science perspective, this is comparable to an agent "walking around" an object to better understand how different perceptions relate to the same object. From computer science perspective, this is similar to the self-supervised framework of SimCLR (Chen ct al., 2020), where agents learn representation by contrastively aligning the embeddings of an input with these of the same transformed input.
Layer
Activation
Conv2D(iilter===,, stride=2, padding=1)
ReLU
Conv2D((ilerrs166, 2tri padding=1)
ReLU
Conv2D(tilter==32,, 2ttpe/iileesss22..rdddddpaaaaae padding==))
ReLU
Linear(128)
ReLU
Linear(32)
None
Table 4:
Model architecture used for both the referent and utterance Encoders. (when referents are one-hot vectors, the 3 Conv2D layers are replaced by a Linear layer with ReLu activation)
While the drawing pipeline is fully differentiable, it is highly sensitive local minima. Thus, solve equation 5 in the descriptive case or equation 6 in the discriminative scenario by simultane- ously performing gradient descent on a batch of 64 randomly initialized command vectors over 100 iterations, using newly initialized Adam optimizer each time with learning rate of le" -
1e-
16

---

Under review as a conference paper at ICLR 2023
A.5
PSEUDO-CODE
Algorithm Speaker's Utterances
Require: perceived referents Rs, speaker's referent encoder Ss, speaker's utterance encoder gs.
sensory-motor system M
Z, 4 fs(Rs)
Uniform()
for / range Npreduction) do
Ug +- (c)
Zu - gs
S sim 7 Zu)
+ mean(diag(S)) * (-1)
GD step minimize
end for Return (c)
Algorithm Listener's Selections & Binary Outcomes
Require: perceived referents ft, produced utterances Us. listener's referent encoder St, listener's
utterance encoder gr.
7, - (R,)
Z, < (Ug))
S + simoos Zy,, Z.)
argmax (S, axis=l)
- 0
for range (Nieterems) do
f- (t:=i)
end for Return
Algorithm Agents's Association Losses
Require: perceived referents R., produced utterances UA. outcomes agent's referent encoder fA. agent's utterance encoder 9A
Z. + (A(RA)
Z, < 9A(UA)
S + sim [Zy, Zu)
Lo + CE(S, Flaee)
C1 + CE(S reduction=False)
L (Ly L2)/2
if A = "S" then
C (C. /)/Nrefeeentss
else
L (f. (C)11)/i... 1)/NNeeerensss
end if
Return
17

---

Under review as a conference paper at ICLR 2023
B
SUPPLEMENTARY RESULTS
B.1
AUTO-COMPREHENNIOO GENERALIZATION PERFORMANCES
Ref.
Auto
Social
One-hot
0.997 = 0.005
0.991 = 0.015
Visual-shared
0.862 0.034
0.559 = 0.027
Visual-unshared
0.425 + 0.016
0.388 + 0.02
Table 5: Descriptive Success Rate
Ref.
Auto
Social
One-hot
0.997 0.005
0.992 + 0.009
Visual-shared
0.812 + 0.019
0.567 + 0.034
Visual-unshared
0.466 + 0.019
0.404 = 0.019
Table 6: Descriminative Success Rate
We define the Auto performance metric as the communicative success rate, on test set, for language games involving single agent playing both the speaker and listener. We compare Auto and Social performances (the latter involving pairs of different agents, as done until now) in Tables 6 &
B.2 ADDITIONAL LEXICONS
R[O]
R[1]
R[2]
R[3]
R(4)
A0
S
1
S
4
Figure 12: Instance of emerging lexicon. (Visual-shared),
R[O]
R[1]
R[2]
R[3]
R[4]
y
A1
Figure 13: Instance of an emerging lexicon. (One-hot).
18

---

Under review as a conference paper at ICLR 2023
B.3
UTTERANCES EXAMPLES ACROSS PERSPECTIVES ILLUSTRATING COHERENCE.
The following figures illustrate the P-coherence and A-coherence of an emerging lexicon (Visual- unshared) displaying, for each referent in Ri, the descriptive utterance produced for 10 random perspectives.
AO
1
A1
Figure 14: Utterances examples for referent O.
Figure 15: Utterances examples for referent 1.
2
3
2
}
AO
1
2
P
1
9
Q
A1
Figure 16: Utterances examples for referent
AO
w
R
A1
5
Figure 17: Utterances examples for referent
AO
A1
Figure 18: Utterances examples for referent
19

---

Under review as a conference paper at ICLR 2023
B.4 TOPOGRAPHIC MAPS & SCORES
B.4.1 ONE-HOT
Headdoff Distance (RE3))
Hausdarff Distance (R(44)
Hourdorff Dictance (RIB))
2.0
1.0
$0.0
6.8
0.8
0.6
04
Hausdorff Distance (RE21)
Hausdorff Distance 08121)
Hausderff Distance (RE1))
p
0.401
0.111
0.085
Heusdorf DrusdoorDDistteeee Distencee (ME27)
Hausdorff Distaace (A22
Haurdertt Distonne (RID
1.0
0.7
0.7
0.8
0.5
82
52
0.4
0-
5.8
1.2
0.2
2.4
0.8
1.0
0.4
0.5
1.0
Hausdorff Distance (R(01)
Hausdorrf Distance (RI11)
Hausdorff Distance (R[O))
0.041
0.09
p 0.094
Heusdorff Distnnce (R(4T)
RLA
Hausdorff Distance (9131)
Hausdertt Dictaince (R(44))
RIAA
R.A
4000
0.8
0.5
0.6
0.4
9.4
0.1
02
022
0.4
0.4
5.0 Hausdortf Distance (RL3))
Houssorti (R[1])
Hausdorff
(R[O])
= 0.1
p= 0.114
= 0.133
Heustorff Distance (A(4))
0.8
Hausdorff Distance (R(O7)
P = 0.147
Figure 19: Topographic maps and their associated topographic scores for each combination of features with one-hot hot referents
20

---

Under review as a conference paper at ICLR 2023
B.4.2
VISUAL SHARED PERSPECTIVES
Hausdom Distoice (R(23)
Hauseort Distance (R13))
12
0.8
8.8
Hauzdofft
0.4
0.4 0.6 08
1. 12
022
0.4
9.6
na
3.0
1.2
1.4
0.6
08
32
Distance (R(2])
(RLO])
Hausdorft Distance (R[2))
0.047
P p= -
0.005
PSuuOomt Distncc (RL3))
Hourdomf (R(4))
Hourdomf Distance
2.0
0.6
Hausdorff Distance (R10))
Hausdorff Distance (R12)
Heusdorff Distance (R(O))
P = 0.073
p= 0.081
p = 0.137
Housdom Distince IRE30
Hourdortt Distaree (R)2)
1.0
1.0
Hauzdortt Distance
3.0
0.8
0.0
0.0
0.6
0.6
0.2
0.4
0,5
08
1.0
12
1.4
0.2
0.4
0.6
0.8
1.0
1.2
1.4
0.2
0.4
2.6
088
1.0
1.2
Hausdorff Distance (R]1
Hausdonf Distancce (R[O])
Hausdorff Distance (R13])
p 0.16
P 0.215
p 0.242
Housdort Distance (R143
1.2
1001
0.8
0.2
0.6
68 1. 12
Hausdorff Distance (R(3])
= 0.501
Figure 20: Topographic maps and their associated topographic scores for each combination of features with shared-visual referents
21

---

Under review as a conference paper at ICLR 2023
B.4.3
VISUAL UNSHARED PERSPECTIVES
5.2
Hausdooffy Distance (RC2))
Molicooft Distance (RE4T)
Heundorff Distance
2.0
0.8
0.3
0.5
0.6
06
12
92
04
06
68
1.
3.2
0.4
0.6
08
1.0
12
Hausdorft Distance (R(1))
Hausdorff Distance (R[0])
Hausdorth Distance (R(AJ)
p = -0.113
p= -0016
MuaddorfDDissaeee (AE31)
Distance (0643)
Pousdomt (RE3))
Hauddorf Duudoor
1.0
8.8
0.8
0.6
0.6
0.4
Hausdorff Distance (R(O))
Hausdorff Distance (R(3))
Hausdorff Distance (R[1))
p 0.019
p 0.026
p p= 0.13
Heuddrff Distance (R143)
2.4
Hausdorff Distance (R143
Soaardom Dictanne MA22
3.9
1.0
0.5
0.5
0.2
0.2
0.4
066
08
2. 1.2
02
0.4
0.6
9.5
1.2
12
0.2
2.4
0.6
8.8
1.0 3.2
Hausdorfff Distance (R(OT)
Housdorff Distance (R[2])
Hauaderff Distance (ACO))
p 0.137
p 0.156
p 0.18
Heusdorff Distence (M)30
3.0
0.6
0.4
02
0.6
6.8 3.0
Housdorff Distance (R(2))
0.203
Figure 21: Topographic maps and their associated topographic scores for each combination of features with unshared-visue referents
22

---

Under review as a conference paper at ICLR 2023
B.5
COMPOSITION MATRIX EXAMPLES (VISUAL- UNSHARED PERSPECTIVES)
RLDI
RIXI
R121
RI3I
R141
RED]
RIXI
R121
R13I
R14]
R221
R221
1
1
REO]
P[A]
[[3
R14]
REO]
R[4]
[L9]
REA
1
X
RL3
&
RU3
4
A
2
7
R13I
RL3I
>
103
PC3
[[4]
RC4]
Figure Instances descriptive utterances referents from Rí (blue frames) and Ra.
23

---

Under review as a conference paper at ICLR 2023
B.6 T-SNES OF EMBEDDINGS (VISUAL- UNSHARED PERSPECTIVES)
B.6.1 Ro REFERENTS & DESCRIPTIVE UTTERANCES
Referents Embeddings
Utterances Embeddings
Referents Embeddings
Utterances Emheddingg
Rjol
R12
112
Referents Embeddings
Utterances Embeddings
Referents Embeddinng
Utterances Embeddings
A111
R[4]
K[4]
NO.3
N.o.
Referents Embeddings
Utterances Embediiing
Referents Embeddings
Utterances Embeddings
ALll
A13
R121
4221
PE31
A[1]
RE1
A141
Referents ReferontSEmbeddieg Embeddings
Utterances Embeddings
Referents Embbddinggs
Utterances Embeddings
RE31
Referents Embeddings
Utterances Embeddings
Referents Embeddings
Utterances Embeddings
REol
R1ll
R13
R121
2221
RL31
5
M2
Figure 23: T-sne of referent and deseriptive utterance embeddings. Embeddings computed for 100
perspectives referents from Ra. Training conditions are unshared visual referents.
24

---

Under review as a conference paper at ICLR 2023
B.6.2 R2 REFERENTS & DISCRIMINATIVE UTTERANCES
Embeddings
Embeddings
NCl
Rill
M1
RE2
R221
R131
AL3]
[[-]
K14)
[[D.1]
M5.2 21
Referents Embeddings
Utterances Embeddings
Referents Embeddings
Utterances Embeddings
Rjol
RL2
Referents Embeddings
Embeddings
Embeddings
Embeddings
R1:1
R131
K[4]
R11 7]
RE3, 31
Referents Embeddings
Utter Embedidings
Referents Embediinng
Utterances Embeddiing
Ajol
AC
Alll
A131
R121
AE21
R22.
Referents Embeddings
aeerances Embeddings
Referents Embedidings
Utterances Embeddings
A13.4)
Figure 24: T-sne of referent and discriminative utterance embeddings. Embeddings are computed for 100 perspectives of referents from Rg. Training conditions are unshared visual referents.
25


Reviews:
[{"review_id": "zd6Z1AFEoT", "reviewer": "Reviewer_XbbP", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates whether agents evolve a shared language when equipped with a sensorimotor system that can produce and perceive drawings. The paper introduces a variant of the referential game, (called GREG) in which two agents must communicate via drawings. Agent A (the speaker) produces a drawing (a continuous graphical utterance) to indicate the ‘name’ of an object in a shared visual scene and communicates this to agent B. Agent B (the listener) then has to select the object indicated by agent A.   The agents switch roles between rounds. The drawings are produced by the agents involved using basic actions over a sketching environment, and so produce apparently continuous signals instead of discrete symbols (as would be the case in written language). Note that the drawings are not constrained to visually reflect the referents. \n\nThe paper then presents a way to train agents to succeed in GREG. The method (Contrastive utterance-reference associative scoring) uses contrastive learning to represent the alignment between utterances (drawings) and referents (objects in the environment), and the generation of utterances maximises energy shared with the target referent to make  the utterance as identifiable (to the speaker, not to the listener) as possible.\n\nThe paper examines performance in this game, analyses the resulting learned language using coherence and compositionality metrics and claims that the learned language demonstrates componsitional properties.\n", "summary_of_the_review": "This paper proposes a variant on the referential game, which creates a new medium for communication: drawings. This results in a continuous space of output to be perceived by the listener, which bring the game setting closer to verbal communication between agents. While the game itself is interesting, the method to solve it does not seem particularly general (which is not a big drawback, just a small one) and the section on compositionality is a weak point.", "strength_and_weaknesses": "\n*Strengths:*\n- The experiments are well designed (one-hot vs visual-shared vs visual-unshared). This difference in perspective between the speaker and listener is a nice addition to this game.\n\n- In the main text there is a balanced set of conclusions around compositionality which match the evidence: that the presence/absence of compositionality in the learned language is difficult to assess and conclude anything about. However (see weaknesses) the abstract makes a different claim.\n\nWeaknesses:\n\n- Despite the lack of clear evidence for compositionality in the main text, the abstract claims in the final sentence “we show that our method allows the emergence of a shared, graphical language with compositional properties”. This claim is unsubstantiated and should be removed.\n\n- It wasn’t clear to me how the drawn image is actually perceived by the listener. Is there a CNN in here somewhere to instantiate this perception, that I am missing? If the listener’s perception happens instead in the action space of the speaker then is it really a continuous signal? The perception of the image by the listener needs to be clarified in the text.\n\n- Compositionality is a hypothesis which is generally difficult to falsify empirically. It is not clear what data would have yielded a clear compositional vs not-compositional result either way. It is therefore not clear what this section of the paper on measuring compositionality contributes to the literature.\n\n- Figure 4 is a bit hard to parse as it has multiple axes going on at the same time. These results would be clearer if there was another way to present them.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "v-cdBHoDNx", "reviewer": "Reviewer_ZH5s", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces a sensory-motor shared language task between two agents, metrics for evaluating the coherence and compositionality of the language that the agents develop, and an initial algorithm for solving this task using a contrastive multi-modal encoder coupled with a generative model. Experiments conducted on this solution show that the languages that develop are coherent, but not very compositional. ", "summary_of_the_review": "This paper poses an interesting and important problem and appears to provide reasonable means of evaluating solutions, although it is difficult to say for sure without an actual full solution. It does provide one solution, but it seems to fall short in terms of compositionality. The paper could provide slightly more detail on the actual sensory-motor aspect of the utterance production process, which is a crucial aspect of its novelty.", "strength_and_weaknesses": "Strengths:\n* The proposed approach addresses an important and timely problem of agents learning to generate a shared language under realistic sensory-motor constraints. As discussed in the paper, several approaches have shown promising results without such constraints, but these are not realistic models of human language development because they lack them.\n* The proposed problem formulation and framework appear to be well suited to evaluating this problem. They are simple enough to be tractable, but sophisticated enough to capture the important aspects of the problem.\n* The proposed metrics seem reasonable in measuring performance\n* The proposed baseline shows promising results for coherence, but not compositionality. It shows that solutions to the task are possible and that the metrics can capture the performance of solutions.\n* The paper is clearly written and explained.\n\nWeaknesses:\n* The abstract states, \"We show that our method allows for the emergence of a shared, graphical language with compositional properties.\" I'm not necessarily convinced that this is true. The proposed solution does not seem to include much in the way of compositionality. The proposed metrics seem to allow for compositionality, but it is difficult to judge the extent to which they would successfully identify a compositional language if one developed. This statement also misled me into thinking that the proposed solution was compositional, although on re-reading, it does not actually state this.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "m9JH13ebqAo", "reviewer": "Reviewer_3FEJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors study emergent machine language. They propose a game and then a system for solving the game. In the game, two agents, a speaker and a listener, must communicate the identity of a referent symbol. In the simplest version of the game, the speaker receives a target referent symbol from a set of context symbols. The speaker produces an utterance, a 2D curve in the plane, which the listener must interpret to predict the target. The authors propose a system for playing this game in which both speaker and listener use a contrastive objective to learn associations between utterances and referents. The proposed system is able to play the game better than chance. Finally, the authors propose a topographic similarity metric to study the compositionality of the emergent symbols. They find that the proposed metric makes suggestive, but not conclusive, findings about the language of their proposed system.", "summary_of_the_review": "The proposed system solves a game in a very limited setting (5 symbols) and the proposed metric for studying compositionality does not conclusively show compositionality for the given emergent language. A revised version of the work that shows why this metric should be trusted, perhaps by giving validation on real-world symbols, would be a much stronger submission.", "strength_and_weaknesses": "# Strengths\n- Exploring compositionality using an automatic topographic metric is an interesting idea. It would be nice to see what this metric gives for real world ideograms/pictograms to get a sense of its trustworthiness and interpretability.\n- The continuous setting is a very nice departure from the typical setup\n\n# Weaknesses\n- The setting is limited. In the single-target case, the alphabet has 5 symbols. In the joint-target case, there are 5 choose 2 = 10 symbols.\n- Studying compositionality in this setting is thus limited to exploring how conjunctive symbols may be encoded, as opposed to studying negation, disjunction, adjectives, adverbs, hierarchy, etc.\n- In this work, compositionality is studied by looking at the form of produced symbols. For instance, to study the compound symbol $u(r_{ij})$, the authors use a topographic score to compare the compound symbol to the corresponding individual constituent symbols. However, there are many ways for a writing system to distinguish between semantics without dramatically distinguishing between forms and vice versa. It would make more sense to do this type of analysis in terms of the symbols decodability. Look at what sort of mistakes a decoding model makes. If the symbol $u(r_{ij})$ only encodes the joint symbol, and nothing about the constituent pieces, then we would expect no correlation among the incorrectly decoded symbols. But if compositionality is employed, we would expect to see predictions of the form $u(r_{ik})$ and $u(r_{ki})$ at higher frequency.\n- The emergent language is not conclusively shown to be compositional (section 4) using the proposed topographic measure\n- I don't think the setting can be rightly called a \"language game\" in which the speaker and listener are jointly learning to communicate. From section 3, pgs. 5 and 6, as best I can tell, the listener's loss is directly optimized to learn the association between the ground truth referent and the utterance. Please correct me if I'm wrong (see clarification question in section below). I believe the correct thing to do would be for the listener to learn an association between the predicted referent and the utterance. As it stands, this is information that is typically not available to the listener during the language game\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Kvh0tRdMV4S", "reviewer": "Reviewer_Jswi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduce a graphical referential game to study the emergence of visual conventions between communicating agents. The framework is based on recent computational approaches exploring the emergence of language through language games. In particular, the authors exploit a “Listener / Speaker” architecture, where the goal of the speaker is to produce an utterance that can be perceived by the listener to identify a target referent. Utterances are generated by feeding command vectors to a differentiable sketching model, with a contrastive learning objective promoting the alignment between referent and utterance embeddings. The authors also explore the compositional properties of the emerging lexicon.", "summary_of_the_review": "I enjoyed reading this paper and I found it stimulating. At the same time, I think that the overall clarity of the manuscript should be improved, and the results should be extended to clarify several aspects of the proposed architecture. The feeling is that part of the results is more “preliminary” than ready to be published in a top-tier venue like ICLR.", "strength_and_weaknesses": "A strength of the paper is that it tackles a non-trivial problem by combining recent ideas from different disciplines (e.g., language games, speaker/listener communication architectures, contrastive representation learning, differentiable computer graphics…). The proposed framework is well-motivated, and the work stimulates further thinking.\n\nAt the same time, however, I see several weaknesses that might downplay the significance of the work:\n- Besides an increase in ecological / cognitive realism, it is unclear what could be the advantage of using motor primitives to create utterances in the graphical space, rather than directly optimizing the continuous graphical representation (as done in previous work). Introducing a “differentiable motor system” might be relevant to better understand how sensory-motor constraints shape our ability to create written symbols, but this aspect has not been investigated in the present work.\n- The authors also argue that a possible advantage of exploiting a drawing framework is to increase the interpretability of the emerging lexicon; however, also this aspect has not been emphasized in the simulations (and, in fact, the emergent signs appear quite arbitrary).\n- It is not clear what is the relationship between the complexity of the concept space (e.g., similarity between objects and/or number of different objects to represent) and the emerging graphical conventions; further analyses would be needed to clarify this important aspect.\n- What is the role of the dimensionality of the graphical space in the emerging lexicon? I guess this parameter might have a major impact on the type of signs that could emerge, but it seems that the dimensionality was arbitrarily chosen.\n- From my understanding, the contrastive learning mechanism should (indirectly) promote an increase in similarity between the referent images and the generated utterances. However, this does not seem to happen in practice. Am I missing something?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "J-tx8Npdy1g", "reviewer": "Reviewer_5nVr", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a new variation of the signalling game for emergent language with 2-dimensional sketch-based messages produced via dynamical motor primitives.\nTo address this new environment, the authors also introduce contrastive learning-based method which aims to align the message embeddings with that of the observation in a joint space.\nFinally, the paper provides an analysis of the performance, message characteristics, and compositionality of the emergent language.\n", "summary_of_the_review": "I am recommending marginal reject because while I find the environment to be well-formulated, I do not see the novelties as being significant from a research perspective.\nI think that given where the field of emergent language is at, the introduction of new environments needs to be focused on more tangible advancements in the field; research which is primarily exploratory is not sufficient for this.\n\n\n### Misc.\n- \"emergent language is as close to natural language\" -> add \"as possible\"\n- \"no number overlap\" -> \"no number overlaps\" \n- \"that quantifies how an utterances denoting ...\" -- check for typos\n- Equation 1: What is $u^\\star$?\n", "strength_and_weaknesses": "### Strengths\n- (major) The paper presents a new, well-formulated emergent language environment and learning method.\n- (minor) The experiments give an adequate characterization of the empirical behavior of the system.\n### Weaknesses\n- (major) The contributions of the paper is introducing a new environment, yet this environment is not strongly motivated.\n  Simply differing from previous work is not motivation enough.\n- (major) The field of emergent language does not stand to benefit that much from new environments if they do not address a relevant issue in the field.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject