Meta-Referential Games to Learn Compositional Learning Behaviours
Authors: Kevin Yandoka Denamganai
Sondess Missaoui
James Alfred Walker
Venue: ICLR 2022 Submitted (2022)
Keywords: language emergence
language grounding
compositionality
systematicity
few-shot learning

Abstract: Referring to compositional learning behaviours as the ability to learn to generalise compositionally from a limited set of stimuli, that are combinations of supportive stimulus components, to a larger set of novel stimuli, i.e. novel combinations of those same stimulus components, we acknowledge compositional learning behaviours as a valuable feat of intelligence that human beings often rely on, and assume their collaborative partners to use similarly. In order to build artificial agents able to collaborate with human beings, we propose a novel benchmark to investigate state-of-the-art artificial agents abilities to exhibit compositional learning behaviours. We provide baseline results on the single-agent tasks of learning compositional learning behaviours, using state-of-the-art RL agents, and show that our proposed benchmark is a compelling challenge that we hope will spur the research community towards developing more capable artificial agents.

Under review as a conference paper at ICLR 2022
META- GAMES TO LEARN COMPOSI- TIONAL LEARNING BEHAVIOURS
Anonymous authors Paper under double-blind review
ABSTRACT
Referring compositional learning behaviours as the ability learn to generalise compositionally from limited set of stimuli, that are combinations of supportive stimulus components, to larger set of novel stimuli, i.e. novel combinations of those same stimulus components, we acknowledge compositional learning be- haviours as a valuable feat of intelligence that human beings often rely on, and assume their collaborative partners to use similarly. In order to build artificial agents able to collaborate with human beings, we propose novel benchmark to investigate tofte-of-theaarr artificial agents abilities to exhibit compositional learn- ing behaviours. We provide baseline results on the single-agent tasks of learning compositional learning behaviours, using state-of-the-art RL agents, and show that our proposed benchmark is a compelling challenge that we hope will spur the research community towards developing more capable artificial agents.
INTRODUCTION
Framing the ability to learn to generalise compositionally from a limited set of combinations of supportive stimulus components stimulus components as compositional learning behaviours, we acknowledge compositional learn- ing behaviours as a valuable feat of intelligence that human beings often rely on, and assume their collaborative partners to use similarly.
larger set of novel combinations of those same supportive
In order build artificial agents able to collaborate with human beings, important endow the former with similar abilities. Thus, we propose to investigate state-of-the-arr artificial agents abilities to exhibit compositional learning behaviours.
Compositional Language Emergence as Proxy for Compositional Behaviours. The field of
language emergence raises the question of how to make artificial languages emerge with similar properties to natural languages, with compositionality at the forefront of those properties(Lazarido)
et al. 2018: Baroni 2019 Guo et al. 2019 Li Bowling 2019 Ren et al. 2020). Referential game Lewis 1969) variants are the primary tools used to make language emerge between pair
population of artificial agents (Denamganai & Walker (220a)). In this context, it has been shown that emerging languages are far from being 'natural-like' protolanguages (Kottur et al. 2017 Chaabouni jet al. 2019a b). but sufficient conditions can be found to further the emergence of compositional
languages and generalising learned representations (e.g. (Koturr et al. Lazarioo al. 2018
Choi et al. 2018; Bogin et al. 22018: Guo et al. 2019 Korbak et al.] | Chaabouni et al. 2020
Denamganai & Walker) (020b)). Nevertheless, the ability of deep-learning agents to generalise compositionally in systematic fashion has been called into question, especially when it comes to language grounding in general (Hill al. (0196), on relational reasoning tasks (Bahdanau et al.)
2019), or on the SCAN benchmark (Lake & Baroni 2018 Loula et al. 2018] Liška et al. (0188) and more recently the gSCAN benehmark (Ruis al., 2020). Neural networks induction biases have been investigated towards finding necessary conditions that favour the emergence of compositional
generalisation/sssttmaaiio (Hill et al. 2019b; Slowik et al. 2020: Korrel et al. 2019 Lake 2019
Russin et al. 2019).
Chaabouni et al. (200)) showed that, when a specific kind of compositionality is found in the emerging languages (the kind that scores high on the positional disentanglement (posdis) metric for compositionallit that they proposed), then it is a sufficient condition for systematicity to emerge. And, more importantly, they showed that emergence of compositional language (in the sense

---

Under review as a conference paper at ICLR 2022
of any of the existing compositionallty-eesinn metric, e.g. topographic similarity (Brighton & Kirby| 2006)) is not a necessary condition for systematic generalisation (as evaluated by a zero-shot compositional learning test) non-compositional, emerging languages have been shown to support systematicity of the pair of agents wielding them in a (generative) referential game.
In this work, we focus directly on systematic generalistionnlystemaiitt as a behaviour, as success in our proposed task is directly related to the agents ability to exhibit compositional learning be- haviours, thus investigating whether they can generalise compositionally in systematic and online fashion (see Section 4
Moreover, contrary to our framework, when studying compositionality in the context of any ref- erential game variant instantiated with a given dataset of stimuli (of any nature, c.g. one-hot- encoded (Kottur et al. 2017:
Chaabouni et al. 2020 Ren et al. 2020 Resnick et al. 2019), vi-
sual (Lazaridou et al., 2018 Choi et al. 2018). or mudti-modal (Evtimova et al. 2017) p). then the trained pair of agents only exhibits compositional behaviours for said dataset within the constraint of the training, thus failing short of learning compositional behaviours that generalises to novel siuatiins/stimuliaatasss
In other words, agents are failing to learn compositional learning be- haviours. Our framework aims to remedy this gap via, firstly, a meta-learning framing of the task and, secondly, via the introduction of a novel stimulus representation that compromises between the mocoodd representation of most symbolic stimuli benchmarks and the continuous rep- resentations akin to latent embeddings found in the unsupervised learning approaches, without sacrificing the symbolic aspects (see Section 3.
Meta- Reta-Referental Games for Compositional Learning Behaviours. Learning compositional learn- ing behaviours pertains to meta-learning. In order to frame referential games in a meta-learning con- text where successful agents are able generalise their abilities to exhibit compositional behaviours onto novel sttuationn/datasetts. it is necessary to provide the agent at training- -tiiing-tmmmee with a distribu- tion of tusks/stimllidaataeess that encompasses the kind of tasks/stimul/dataseet that the agent will experience after being deployed.
For instance, using a set of datasets with different generative factors/attributes as our distribution of tasks would force us to focus on only one modality shared among all datasets, for instance the visual modality. Doing so would add to our work the assumption of working with visual stim- uli only, which is quite constraining. Instead, we choose to assume that stimuli are disentangled, independently of their nature. Thus, using one-hot-encoded (OHE) vectors, that are disentangled abstract stimuli, would fit the bill with regards to assumption and our work would retain some external validity. It would be applicable whenever stimuli can be discretely categorised over a set of generative factors/attributes with discrete values.
Further more, once a semantic structure is chosen, i.e. a number of generative factors/attributes, Naim: and number of possible values for each (d(i)) it is impossible to extend our meta-distribution to differently semantically structured stimuli without changing the shape of our OHE vectors, datim Exe(t:Na..)) d(i). Most state-of-the-art deep learning architectures rely on the assumption that the dimensions of their input spaces are constant. Thus, in order to provide a meta distriiitti over differently semantically structured stimuli spaces, we propose different representation, entitled symbolic continuous stimulus representation (SCS), it is detailed in Section 3 Using this SCS representation, we can meta-train agents in a meta-referential game settings where the semantic structure observed can be randomised over without changing the shape of the stimulus space and without the overly constraining assumption that the representation must be discrete.
Our contributions are threefold:
We propose the Symbolic Continuous Stimulus (SCS) representation as a versatile encod- ing of symbolic spaces, which, on the contrary to one-hot-encoded representation, allows for infinitely many semantic structure to instantiated without changing the shape of the representation, and rely continuous values that are more akin to the kind of stimuli found in the real world, as opposed to discrete valued mne-hot-encodings.
We cast the problem of learning compositional behaviours as meta-rein meta-reenforcement learning problem, using (discriminative) referential games, containing a meta-reiiforcement learn-
2

---

Under review as a conference paper at ICLR 2022
ing formulation of the binding problem, thus allowing it to be studied under the lens of state-of-the-ane RL algorithms.
We provide baseline results on the single-agent task of learning compositional learning behaviours, using state-of-the-art RL agents, and show that our proposed benchmark is a compelling challenge that we hope will spur the research community towards developing more capable artificial agents.
2 BACKGROUND & RELATED WORKS
Referential games are at interface between the language processing subfields of language emer- gence, language grounding, and, when not using on-hot-encoded stimuli, the computer vision sub- field of unsupervised representation learning. While language emergence raises the question of how to make artificial languages emerge with similar properties to natural languages, or at least 'natural- like' protolanguages, with compositionality
the forefront of those "oopertiesBarrnn 2019 Guo et al. 2019; Li & Bowling 2019; Ren et al. (020), language grounding is concerned with the ability to ground the meaning (natural) language utterances into some sensory processes, with the visual modality being the main focus of research. On one hand, emerging artificial languages' compo- sitionality has been shown to further the learnability of said languages (Kirby) 2002, (Smith et al.) 2003 Brighton 2002; Li & Bowling 2019) and, on the other hand, natural languages' composi- tionality promises to increase the generalisation ability of the artificial agent that would be able to rely on them as a grounding signal, as it has been found to produce learned representations that generalise, when measured in terms of the Sata-efficiency of subsequent transfer and/or curriculum learning (Higgins et al. 2017; Mordatch & Abbeell Moritz Hermann et al.) Jiang al., 2019). More in touch with the current context of this study, Chaabouni et al. (2020) showed that, when a specific kind of compositionality is found in the emerging languages (the kind that scores high the posi- tional disentanglement (posdis) metric for compositionality that they proposed), then it is a sufficient condition for systematicity to emerge.
Language Compositionality & Compositional Systematic CeneralisatonnSSstemtttiii As a concept, compositionality has been the focus of many definition attempts. For instance, it can be defined as "the algebraic capacity to understand and produce novel combinations from known com- ponents" "onouls"(Loula
al.] (2018) referring to Montague (1970)) or as the property according to which "the meaning of a complex expression is function of the meaning of its immediate syntactic parts and the way in which they are combined" (Krifka 200T). Although difficult to define, the commmu- nity seem agree on the fact that it would enable learning agents to exhibit systematic generalisation abilities (also referred to as combinatorial generalisation (Battaglia et a Some of the ambiguities that come with those loose definitions start to be better understood and explained, as in the work of Hupkes et al. (201)). While often studied in relation to languages, it is usually defined with a focus on behaviours. In this paper, we refer to compositional behaviours as "the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents" ((oonntts"FFoooo (Fodor) let al. 1988), and thus use it interchangeably with systematicity, following the classification made by Hupkes et al. (2019).
Compositionality, as a property of languages, will be referred as linguistic compositionality. in this paper. It can be difficult to measure. [Brighton& & Kirby](2006) 's topographic similarity (topsim) which is acknowledged by the research community as the main quantitative metric for composition-
ality (Lazaridou et al.) 2018 Guo et al., 2019; Stowik et al. 2020; Chaabouni et al. 2020 Ren al. 2020p. Recently, taking inspiration from disentanglemes
metrics, [Chaabouni et al. (2020) proposed two new metrics entitled posdis (positional disentanglement metric) and bosdis (bag-or- symbols disentanglement metric), that have been shown to be differently 'opinionated" the sense that they each seem to capture different ways in which a language can be shown to be (linguistically) compositional.
Binding Problem & Meta- Following Greff et al. (2020), we refer to the binding problem as the "inability to dynamically and flexibly bind information that is distributed throughout the network" of deep learning architectures. We note that relational responding ("adjusting the (task- specific) response to an object based on its relation to other objects" ; for instance, one way to facilitate it in neural network "is to organise its internal information flow (i.e. computations) in
3

---

Under review as a conference paper at ICLR 2022
Day
ObjectiCertric Sarppl Dansity Bistribution Butioon ever [d:+1]
Example Stimull
(fy 2,
[3, 1, 21
10001
(0010
Dan
Representation
Sample
0100
00101
1000
0100)
Leten Din
Sample Reprosentation
SCS
F0.82
0.55
-0.62
$0.25
0.74
10.04
(a)
(b)
Figure I: (a): representation of the gaussian kernels corresponding each value sections on each latent factor/attribute dimension, in the case of Ndim 3. Black vertical bars indicate the Рт(a) value samples from each gaussian kernels corresponding to the (()) values instantiated on each latent dimension construct the SCS representations of the example stimuli in (b). (b): OHE and SCS representations of example stimuli for a semantic structure/symbolic space with Neim 3, d(0) 4, d(1) 2, d(2) = 3.
a way that reflects the graph structure of relations and objects") is central to solving the binding problem instantiated in our proposed benchmark.
Relational Frame Theory (RTF) distinguishes "two types of entailment that humans primarily use to derive (unobserved) relations: mutual entailment [(derive additional relations between two ob- jects based on a given relation between them)) and combinatorial entailment [(derive new relations between objects, based their relations with shared third object)]"" Within those terms, the ability to perform (sequential) combinatorial entailment is at the center the benchmark challenges.
3 SYMBOLIC CONTINUOUS STIMULUS REPRESENTATION
We introduce symbolic continuous (as opposed to discrete) stimulus (SCS) representation which has the particularity of enabling the representation of stimuli sampled from differently semantically structured symbolic spaces while maintaining the same representation shape, as opposed to the one- hot encoded (OHE) representation. We will refer to this the representation shape invariance property of the SCS representation.
Namely, defining the semantic structure of an Nain -dmensiione symbolic space by the tuple (d(i))e(LLNNa..) where Naim is the number of factor dimensions, d(i) is the number of values for each factor dimension i, then the representation shape of any stimulus sampled from any such Naim- dimensioned symbolic space is vector over [-1, + Note that this shape does not depend on the d(i)'s 's values, as opposed to the OHE representation which samples vectors from the discrete Namm d()).
space (0, dons where dOnE = -==1
From given semantic structure, (d(t))ie(I,Nan)) the representation sampling space is built fol- lows: for each factor dimension i, the 1, +1| range is partitioned in d(i) value sections, each corresponding to one of the d(i) symbolic values available on the i-th factor dimension. Sampling stimulus from this symbolic space boils down instantiating latent values I(i) on each factor dimension i, such that 1(v) E (1))) Differently from the OHF representation, the SCS represen- tation of this stimulus is continuous vector whose i-th dimension is populated with sample from a corresponding gaussian distribution over the ((i)-th partition, ((()) ~ (((()) T(G)). where μ(()) the mean of the gaussian distribution, uniformely sampled to fall within the range of the 1(i)-th partition, and (i(e) the standard deviation of the gaussian distribution, uniformely sampled the range [12)()) 6d(z)- (*() and di(s) are sampled in order to guarantee (i) that the scale of the gaussian distribution is large enough, but (ii) not larger than the size of the partition section it should fit in. Figure la shows example of such instantiation of the different gaussian distributions each factor dimensions' [- +1] range, and Figure Iblighligtht how the two representations differ when representating the same example stimuli.
4

---

stimulus Under review as a conference paper at ICLR 2022
Snaays 2ehois Accuracy
80
20
Updates
(a)
(b)
Figure 2: (a): 5-ways 2-shots accuracies on the Recall task with different stimulus representation( OHE:blue: SCS; orange). (b): Illustration of a discriminative 2-players L-signal / N-round variant of referential game.
3.1 OF THE BINDING PROBLEM INSTANTIATED BY THE SCS REPRESENTATION
The SCS representation differs from the OHE one primarily in terms of the binding problem (Greff et al. 2020) that the former instantiates while the latter does not.
More specifically, the OHE representation inherently discloses the semantic structure (i.e. the d(i)'s) of the NNdim-diiennt -dimensioned symbolic space via the representation shape of the stimuli, i.c. discrete vectors sampled from {0, 1jdons where dOhE is dependant the d(i)'s. ss. On the other hand,
the SCS representation, i.e. continuous vectors sampled from +1] keeps the semantic structure hidden.
Thus, the semantic structure can only be inferred after observing multiple SCS- SCS-represented stimuli. Indeed, we hypothesised that it is via the dynamic binding of information extracted from each ob- servations that an estimation of a density distribution over each dimension i's [-1,+ +1] range can be performed. And, estimating such density distribution is tantamount to estimating the number of likely gaussian distributions that partitions each dimension i's associated [-1, +1] range.
Towards highlighting that there is a binding problem taking place, show results of baseline RL agents evaluated on simple recall task. The Recall task structure borrows from few-shot learning tasks as it presents over shots an entire symbolic space.
Each shot consists of a series of games, one for each stimulus that can be sampled from an Naim of the symbolic space
3-dimensioned symbolic space. The semantic structure randomly sampled at the beginning of each episode, i.e. for each i, d(i) ~ ((2 where (2;;5) is the uniform discrete distribution over the integers in the range [2; 5].
Each game consists of two turns: in the first turn, stimulus is presented to the RL agent, and only a no-operation (NO-OP) action is made available to the RL agent, while, on the second turn, the agent is asked to recall the discrete ((i) latent value that the previously-presentee had instantiated, on given i-th dimension, where the current game's 's i is uniformly sampled from (1; Naim) at the beginning of each game. On the second turn, the agent's 'ss available action space now consists of discrete actions over the range 1; maz,d() ddj)], where mar,d() is hyperparameter of the task representing the maximum number of latent values for any factor dimension.
In our experiments, maa 5. While the agent is rewarded at each game for recalling correctly, we only focus on the performance over the games of the second shot, that is to say on the games where the agent has acquired theoretically enough information to infer the density distribution over each dimension i's [-1, +1] range, because observing the whole symbolic space once (on the first shot) is sufficient (but not necessary, especially as seen in the case of the OHE representations).
Our results in Figure 2a present a large gap performance in terms of accuracy over all the games of the second shot, depending whether the recall task is evaluated using OHE or SCS repre- sentations. We attribute the poor performance in the SCS context to instantiation of the binding problem.
5

---

Under review as a conference paper at ICLR 2022
4 META-REINFORGEMENT LEARNING WITH REFERENTIAL GAMES
4.1 REFERENTIAL GAMES
The first instance of an environment that demonstrated primary focus on the objective of com- municating efficiently is the signaling game or referential game by Lewis) (1969), where a speaker agent is asked to send message to the listener agent, based on the state/stimulus of the world that it observed. The listener agent then acts upon the observation of the message by choosing one of the actions available to it. Both players goals are aligned (it features pure coordination/commnn interests), with the aim of performing the 'best' action given the observed state, where the notion of 'best' action defined by the goal/interests common to both players.
Under the nomenclature presented in Denamganai & Walker (2000)), our benchmark instantiates a discriminative fully-observable / 2-players / L = 1-signal distributed-K 3-distractors / object-centric variant. Figure 2b illustrates this setup in the general
Ndim-round -oorro / uniformly-
case.
Full vs. Partial Observability. This feature characterises whether the stimuli that are presented to the speaker agent consist of all the stimuli experienced by the listener agent or solely of the target stimulus. For simplicity, and in order for both agents to have the same state space, we employ full observability in this benchmark. It also simplify the problem as it allows the speaker agent to reason pragmatically. Specifically, in Figure [b]] the orange arrow highlights the additional information available when the speaker agent has full observability.
Variable-length Communication This feature characterises the ability from the speaker agent to send/utter more than one symbol/signal to the listener agent, up to maximal possible length, I., for the sequence of symbols. The basic referential game is I-signalled. Variable-length communication channels were first introduced by Havrylovv & Titov (2017) and has quickly been adopted by the research community as standard, independently of what approach used to support the communi- cation channel (Lazaridou et al. 2018 Choi et al.) 2018). In this work, in order for the action space to remain manageable, we use 1. and allow for N Nam communication rounds.
Multi- Round Communication. This feature characterises (i) whether the listener agent can send messages back to the speaker agent and (ii) how many communication rounds can be expected before the listener agent is finally tasked to discriminate between the stimuli it observes and have to act by pointing at the one it estimates as the target stimulus. In our proposed benchmark, this feature is parameterizable, but, in this work, in order for both agents to have the same state and action spaces, we allow the listener to send messages similarly to the speaker, but the environment zeros out those messages coming from the listener.
Stimulus vs. Object Centricism. The basic (discriminative) referential game is ttmuulus-cnntrig which assumes that both agents would be somehow embodied in the same body, and they tasked to discriminate between given stimuli. On the other hand, the object-centric variant incorporates the issues that stem from the difference of embodiment. The agents are tasked with discriminat- ing between objects (or scenes) independently of the viewpoint from which they may experience them. In this variant, the game more about bridging the gap between each other's cognition rather than (just) finding common language. It was introduced in the work of Choi et al. (2018). in its deseriptive-one
form. Needless to say that the objuse- variant adds difficulty to the task. It has been highlighted that embodiment may hold some key to the systematic generalisation abil- ities of deep learning agents (Hill et al. 2019a), and therefore it is highlighted as a very important research direction pursue.
In an even more abstract approach, the object-focused setting could be acknowledged as an emphasis on the concept or semantic meaning behind the observed stimulus, and the listener agent would thus be tasked with learning the semantic, while being prompted with different instances of it. In the paramcterisation,
current work, as we are presenting baseline results, we employ a stimulus-cente but the benchmark we propose incorporates object-centristt and it will be investigated in subsequent works.

---

Under review as a conference paper at ICLR 2022
Gow
Cape
Gare
4's
Reaen RC xg
MMNi
MAtk Sugjet
Figure 3: Left: Instantiation of a dataset of SCS-represented stimuli sampled from the current episode's 's symbolic space, whose semantic structure is sampled out of the meta-distribution of avail- able semantic structure over N Nai -dimensioned symbolic spaces. Right: illustration the resulting (meta- trenffrreiin learning episode consisting of a series of referential games.
4.2 META-REFERENTIE GAMES
Thanks to the representation shape invariance property of the SCS representation, after fixing a number of latent/factor dimension Ndsm: we can define many differently semantically structured Nlam-dimensionee symbol spaces. In other words that are more akin to the meta-learning field, we can defin distribution over many kind of tasks, where each task instantiates a different se- mantic structure, that we want our agent to be able to adapt to. And, by defining the tasks as zero-shot compositional learning test that are parameterized by differently semantically structured Naim- -iimeniiniii symbolic spaces, we aim for the agent to learn to exhibit compositional learning behaviours over -dddmm-niiiooed symbolic spaces.
Figure 3 hiighlights the structure of an episode, and its reliance on differently semantically structured Namm dimensioned symbolic spaces. Similarly to the Recall task, this meta-referential game-base based benchmark presents over 2 shots all the space, and then over only one shot all the testing-purpont purposed stimuli, similarly to how a zero-shot compositional test would be performed with referential games. Each shot consists of series of referential games, one for each relevant stimulus.
stimuli of the current episode's s symbolic
4.2.1 VOCABULARY PERMUTATION ON THE COMMUNICATION CHANNEL
We bring the readers attention on the fact that simply changing the semantic structure of the Naim- dimensioned symbolic space, is not sufficient to draw out MARL agents adaptation. Indeed, they can learn to cheat by relying on an epissde/task-invariant (and therefore semantic structure invariant) emerging language which would encode the continuous values of the SCS representation like analog-to-diginn converter would. This cheating language would consist of mapping fine-enough partition of the [-1, +1] range onto the vocabulary in a bijective fashion.
For instance, for a vocabulary size ||V|| 10, each symbol can be unequivocally mapped onto 2 -th increments over [-1, and, by communicating Naim symbols (assuming Nain S L), the TO
speaker agents can communicate to the listener the (digitized) continuous value on each dimension i of the SCS-represe SCS-represented stimulus. If maz,d(j) < ||V]] then the cheating language is expressive- enough for the speaker agent to digitize all possible stimulus without solving the binding problem, i.e without inferring the semantic structure. Similarly, it is expressive-enough for the listener agent to convert the spoken utterances to continuous/analog-ll thus enabling the listener agent to skirt the binding problem when trying to discriminate the target stimulus from the different stimuli it observes.
values over the [-1, +1] range,
Therefore, in order to guard the MARL agents from making a cheating language emerge, we employ vocabulary permutation scheme (Cope & Schoots 2021) that samples at the beginning of each episode/task a random permutation of the vocabulary symbols. This approach is similar to the Other-Play algorithm from Hu et al. (2020).
7

---

Under review as a conference paper at ICLR 2022
Shat Componitonal Accusny (N) DM0
Stimltx Reconetructio Accunscy (S)
LOTM
LOTD-Ric
CMO jus toroy)
LSVV you bolle slave)
80
40
94
22
# Updckes
Updakis
20
(b)
Figure 4: (a): 4-ways (3 distractors) zero-shot compositional test accuracies of different architec- tures. seeds for architectures with DNC and LSTM, and seeds for runs with DNC+Rec and LSTM+Rec, where the auxiliary reconstruction loss is used. (b): Stimulus reconstruction accura- cies for the archiecture augmented with the auxiliary reconstruction task. Accuracies are computed on binary values corresponding each stimulus' latent dimension's 's reconstructed value being close enough to the ground truth value, with a threshold of 0.05.
EXPERIMENTS & ANALYSIS
5.1 SINGLE-AGER REINFORCEMENT LEARNING SETTING
While referential game usually involves a speaker and listener agent, in the present paper propose to focus solely on the listener agent by replacing the speaker agent with a rule-based speaker agent whose language is (linguistically) compositional in the sense of the posdis compositionality metric (Chaabouni et al. 2020).
Indeed, the listener agent of referential game is solely focused with the problem of language acquisition, which is assumed easier than the problem of language emergence that the speaker agent has to solve by searching for expressive enough artificial language to describe SCS-represented stimuli.
By focusing solely on the language acquisition problem, in the context of a (linguistically) compo- sitional language, we focus in effect on the listener agent's ability to learn compositional learning behaviours, and nothing more. We will explore the other problems of this framework in subsequent works.
5.2 AGENT ARCHITECTURE
The bascline RL agents that we consider are made of standard architecture for reinforcement learning in multi-modal (stimulus language) environments. The stimulus is processed at every timestep by a fully-connected 3-layers network. The language input, represented as one-hot- encoding, is concatenated with the stimulus embeddings and, finally, core memory processes the information over time. A fully-connected layer followed by softmax activation maps the output of this core memory module to a distribution over 29 actions, which corresponds to an action space combining both the language output and the decision output. Optimization is performed via an R2D2 algorithm(Kaptunn (Kapurowski et al. 2018).
We investigate both an LSTM (Hochreiter & Schmidhuber 1997 and Differentiable Neural Com- puter (DNC) (Graves et al. 2016) as core memory module. More details can found, for re- producibility purposes, in our open-source implementation MUIDDEN-FOR-EVIES REVIEW-PURPOSE Hyperparameters have been selected via fine-tuning using Weights&Biases Hyperparemeter Sweep feature.
5.3 AUXILIARY RECONSTRUCTION TASK
Following the work of Hill et al. (2020), augment the agent with auxiliary reconstruction task aiming to help the agent learning to use its core memory module. The reconstruction loss consists of mean squared-error between the stimuli observed by the agent given time step and the
8

---

Under review as a conference paper at ICLR 2022
output of prediction network which takes input the current state of the core memory module after processing the current timestep stimuli. In the case the LSTM, the hidden states are used, while in the case of the DNC, the memory is used as input the prediction network.
5.4 RESULTS
Figure 4a shows the 4-ways (3 distractors) zero-shot compositional (ZSC) test accuracies of the different agents throughout learning. The zero-shot compositional test accuracy is the accuracy over testing-purpose stimuli only, after the agent has observed for two consecutive times the supportive iraining-purpose stimuli for the current episode/task parameterised by the current semantic structure. The DNC-based architecture has difficulty learning how to use its memory, even with the use of the auxiliary reconstruction loss, and therefore it utterly fails reach better-than-chance accuracies. On the otherhand, the LSTM-based architecture is fairly successful on the auxiliary reconstruction task, but is not sufficient for training really take-off. This result hints at the fact that indeed new inductive biases must be investigated to be able to solve the problem posed by the benchmark that we propose.
test
6 CONCLUSION
In order to build artificial agents able to collaborate with human beings, have proposed novel benchmark to investigate artificial agents abilities at learning compositional learning behaviours. Our proposed benchmark casts the problem of learning compositional learning behaviours (meta-)reinff trenfoareinooreementt learning problem, using iterated (discriminative) referential games, containing its core an instantiation of the binding problem. This instantiation of the binding problem and the meta-lcarning formulation of referential games is made possible by a novel representation sheeme, entitled the Symbolic Continuous Stimulus (SCS) representation, which acts as a versatile repre- sentation for symbolic spaces as it allows for infinitely many semantic structure to be instantiated without changing the shape of the representation, and relies on continuous values that are more akin to real world stimuli, as opposed to discrete valued me--hot-nncoiings.
Finally, we have provided baseline results on the single-agent tasks of learning compositional learn- ing behaviours, using state-of-the-art RL agents built around core memory modules, and our results show that our proposed benchmark is currently out of reach for. Thus, we hope will spur the research community towards developing more capable artificial agents.
REFERENCES
Dzmitry Bahdanau, Shikhar Murty, Michael Noukhovitch. Thien Huu Nguyen, Harm de Vries, and Aaron Courville. Systematic Generalization: What Is Required and Can Be Learned? International Conference on Learning Representations, nov 2019. URL
abs / 11111111111 12889
Marco Baroni. Linguistic generalization and compositionality in modern artificial neural networks. mar 2019. UR..htp:://axxw.oo//////// 00157711407777 00157
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Monnhez-GGnzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song. Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. URL http:p:://axx...oogg//........ /etoooss/////tteees///a//rooooo/ o12p:/22pa./1000.0121.......
Ben Bogin, Mor Geva, and Jonathan Berant, Emergence of Communication in an Interactive World with Consistent Speakers. sep 2018. URL http::/tttpii...t/t// http:/xwww..rr////////////// 00549
Henry Brighton. ficial, 2002. URL http::::/wwwww.mappeesss/s/sss/s fournals. ozg/////abs/110.oooooo
Compositional syntax from cultural transmission. MIT Press. Arti-
7064544022133644755

---

Under review as a conference paper at ICLR 2022
Henry Brighton and Simon Kirby. Understanding Linguistic Evolution by Visualizing the Emer-
gence of Topographic Mappings. Artificial Life, 12(2):229-243) jan 2006. ISSN 1064-5462.
doi: 10.1 10.11622ar1.2006.222.2229 10. 206. 22.................2........
URL http://we /wwp///www..... www.peisssoouuaaaaee wwt.
Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient en- coding emergent communication. NeurIPS, may 2019a. URL /tK/:tttpp//rxxi.orr///bss 1905. 12561
Rahma Chaabouni, Bugene Kharitonov, Alessandro Lazaric, Emmanuel Dupoux, and Marco Baroni. Word-order biases in deep-agent emergent communication. may 2019b. URL htLhttp: htttp::///axx org/ //5//ab/// 1905 12330
Rahma Chaabouni, Eugene Kharitonov, Diane Bouchacourt, Emmanuel Dupoux, and Marco Baroni. Compositionality and Generalization in Emergent Languages. apr 2020. URL/ttt:://axxx
org/abs/2004.0 09124
Edward Choi, Angeliki Lazaridou, and Nando de Freitas. Compositional Obverter Communication Learning From Raw Visual Input. apr 2018. URL
02344
Dylan Cope and Nandi Schoots. Learning to communicate with strangers via channel randomisation methods. arXiv preprint
100::2004409957 2021.
Kevin Denamganai and James Alfred Walker. Referentialgym: A framework for language emer- gence & grounding in (visual) referential games. 2020a.
Kevin Denamganai and James Alfred Walker. On (emergent) systematic generalisation and compo- sitionality in visual referential games with straight-throug gumbel-softmax estimator. 2020b.
Katrina Evtimova, Andrew Drozdov, Douwe Kiela, and Kyunghyun Cho. Emergent Communication in MMulii-Modal Multi-Step Referential Game. may 2017. URL http://arxiv..ottt
765. 10369
Jerry A Fodor, Zenon W Pylyshyn, et al. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-2) 1988.
Alex Graves, Greg Wayne, Malcolm Reynolds, Tim Harley, Tvo Danihelka, Agnieszka Grabska- Barwińska, Sergio Gómez Colmenarejo, Edward Grefenstette, Tiago Ramalho, John Agapiou, et al. Hybrid computing using neural network with dynamic external memory. Nature, 538 (7626):471-476, 2016
Klaus Greff, Sjoerd van Steenkiste, and Jürgen Schmidhuber. On the binding problem in artificial neural networks. arXiv preprint arXiv:2012.05208,, 2020.
Shangmin Guo, Yi Ren, Serhii Havrylov, Stella Frank, Ivan Titov, and Kenny Smith. The emergence of compositional languages for numeric concepts through iterated learning neural agents. arXiv preprint arXiv: 19t0v::990005291 2019.
Serhii Havrylov and Ivan Titov. Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols. may 2017. URL
Irina Higgins, Nicolas Sonnerat, Loic Matthey, Arka Pal, Christopher Burgess, Matthew Botvinick, Demis Hassabis, and Alexander Lerchner. SCAN: Learning Abstract Hierarchical Compositional Visual Concepts. jul 2017. URL/ arwiv..ivvv. orggabs/1707.. 03389
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. McClelland, and Adam Santoro. Emergent Systematic Generalization Situated Agent. 2019a. RtLhttt:/// arxiv. www.oogg//a.... attp//bss1999 00171115111111
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L Me- Clelland, and Adam Santoro. Environmental drivers of systematicity and generalization a situated agent. October 2019b.
10

---

Under review as a conference paper at ICLR 2022
Felix Hill, Olivier Tieleman, Tamara Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark DeepMind. Grounded language learning fast and slow. Technical report, 2020.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1997.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. ""thher-pay"" for zero-shot coordination. In International Conference on Machine Learning, pp. pp. 4399944410. PMLR, 2020.
Dieuwke Hupkes, Verna Dankers, Mathijs Mul, and Elia Bruni. Compositionallty decoomoose how do neural networks generalise? aug 2019. URL /ttppp/p//aarrrs..rrr////////..0
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language an Abstraction for Hier- archical Deep Reinforcement Learning. jun 2019. URL
07343
Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent ex- perience replay in distributed reinforcement learning. In International conference on learning representations, 2018.
Simon Kirby. Learning, bottlenecks and the evolution of recursive syntax. 2002.
Tomasz Korbak, Julian Zubek, Lukasz Kuciñski, Piotr Milos, and Joanna Rgczaszek-Leonardi De- velopmentally motivated emergence of compositional communication via template transfer. oct 2019. http:tttt///iiiiictt
1910. C6079
Kris Korrel, Dieuwke Hupkes, Verna Dankers, and Elia Bruni. Transcoding compositionally: using attention to find more generalizable solutions. jun 2019. URL
1906.01239
Satwik Kottur, José M. F. Moura, Stefan Lee, and Dhruv Batra. Natural Language Does Not Emerge 'Naturally' in Multi-Agent Dialog. jun 2017. URL http:://arxiv.orr///3/.
7776.08502.
Manfred Krifka. Compositionality. The MIT encyclopedia of the cognitive sciences, pp. 152-153, 2001.
Brenden M. Lake. Compositional generalization through meta sequence-to-ssequence learning. jun 2019.
Brenden M. Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sent -ouuuueeeeeee recurrent networks. 35th International Conference on Machine
Learning. ICML 2018, 7:4487-4499, oct 2018. URL http:// http//rxiii..///// http::/aaxiv..rrrt wttpxii..ro//b////121111 00350
Angeliki Lazaridou. Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of Linguistic Communication from Referential Games with Symbolic and Pixel Input. apr 2018. URL |
//arxiv.org/aabe /ttp///ss844400.00
David Lewis. Convention: A philosophical study. 1969.
Fushan Li and Michael Bowling. Fase-of-Teaching and Language Structure from Emergent Com-
munication. jun 2019.
02403
Adam Liška, Germán Kruszewski, and Marco Baroni. Memorize generalize? Searching for compositional RNN in a haystack. feb 2018.
João Loula, Marco Baroni, and Brenden M. Lake. Rearranging the Familiar: Compostioon Generalization in Recurrent Networks. jul 2018. URL http::::/aaaxiv.og//a///////
07545
Richard Montague. Universal grammar. Theoria, 36(3):373-398, 1970.
Igor Mordatch and Pieter Abbeel. Emergence Groundee Compositional Language in Multi-Agent Populations. URL http://p////aaakkkoo/oo//////00. .pff

---

Under review as a conference paper at ICLR 2022
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang. Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojciech Marian Czamecki, Max Jaderberg. Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, Phil Blunsom, and Deepmind London. Grounded Language Learn- ing in a Simulated 3D World, U.R.httts://arxiiv org/ ptf///df//f// /1766 0.00066665557 06511. .ddf
Yi Ren, Shangmin Guo, Matthieu Labeau, Shay B. Cohen, and Simon Kirby. Compositional Lan- guages Emerge in Neural Iterated Learning Model. feb 2020. URL http://arxiv.orr/
abs/2002. 01365
Cinjon Resnick, Abhinav Gupta, Jakob Foerster, Andrew M. Dai, and Kyunghyun Cho. Capacity, Bandwidth, and Compositionality in Emergent Language Learning. oct 2019. URL http:
//axxiv.crg/aab///13 11424
Laura Ruis, Jacob Andreas, Marco Baroni, Diane Bouchacourt, and Brenden M Lake. A benchmark for systematic generalization in grounded language understanding. March 2020.
Jake Russin, Jason Jo, Randall C. O'Reilly, and Yoshua Bengio. Compositional generalization in deep seq2seq model by separating syntax and semantics. apr 2019. URL http://arxiv.
org/abs 1104.990.0000 09708
Agnieszka Slowik, Abhinav Gupta, William L. Hamilton, Mateja Jamnik, Scan B. Holden, and Christopher Pal. Exploring Structural Inductive Biases in Emergent Communication. feb 2020.
abs 2002. 01335
K Smith, S Kirby, H Brighton Artificial Life, and Undefined 2003. Iterated learning: A framework for the emergence of language. Artificial Life. 9(4):371-389 2003. URL https: / /www.
mitpressjournals. http///sssaaaaooooo.o.//////.112
12


Reviews:
[{"review_id": "BJz8rBDXnp", "reviewer": "Reviewer_RCbW", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors introduce a symbolic continuous stimulus representation that has the particularity of enabling the representation of stimuli sampled from differently semantically structured symbolic spaces while maintaining the same representation shape. \n\nThe paper is a bit hard to read, I am not sure I fully understand this paper. Could the authors provide more details of why the proposed representation is helpful for compositional learning?\n\nIs the proposed method adaptable to solve other tasks that do not belong to preferential games? \n\nThe experiments are not sufficient to support the claimed contributions. The authors did not compare the proposed SCS representation with other representations, for example, which baselines use one-hot-encodings?\n\nThe text in Figure 1, Figure 2, and Figure 3 are too small.", "summary_of_the_paper": "In this paper, the authors propose a new benchmark to investigate state-of-the-art artificial agents' abilities to exhibit compositional learning behaviors. The authors propose a Symbolic Continuous Stimulus (SCS) representation and cast the problem of learning compositional behaviors as a meta-reinforcement learning problem. The authors compare state-of-the-art RL methods on solving the single-agent task of learning compositional learning behaviors.\n\n", "summary_of_the_review": "I think the idea of compositional learning is interesting. However, I didn't fully understand why the proposed representation is helpful for compositional learning? I hope to see more explanations from the authors in their feedback. \n\nThe experiments are not sufficient as well. The authors simply compared several baselines without extra ablation study of their proposed method. It is hard to know why LSTM+Rec is better than other baselines. The authors just use 2 seeds for runs with DNC+Rec and LSTM+Rec, they should report the results of using more seeds in Figure 4. \n\nI think this paper needs to be improved and it has the potential to become a promising paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6yYdglgLmal", "reviewer": "Reviewer_3vEd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**SCS**\n\nThe authors propose a new method to represent symbolic stimuli as continuous representations as an alternative to one-hot encodings. The SCS representations seem like a flexible way to define differently structured semantic spaces in synthetic settings (one might need to change the dimensions of the input if they were to use one-hot encodings). The paper then does a good job of showing how using one-hot encodings compared to SCS change the problem definition leading to a difference in performance on the same task.\n\nDespite this, I fail to see how the SCS representations might be useful outside the scope of the experiments shown in the paper. I also encourage the authors to simplify the experiment described in section 3.1 to make it more clear. The applicability and the novelty of the SCS representation seem limited. I would like the authors to have a more extended discussion of how SCS can be used outside of their work.\n\n**BENCHMARK & BASELINES**\n\nThe paper does an excellent job of introducing the meta-referential games and the different setup settings they use, and the corresponding reason behind them. \n\nWhile the authors say that they use vocabulary permutation, it is not clear what the size of the vocabulary is. Similarly, I could not find the value of the dimension of the symbolic space $N_{dim}$ that was used to generate the stimuli. In a similar vein, I would have appreciated more details about the model architecture and the training regimes used (possibly in the appendix).\n\nI found the result that the LSTM-based model successfully reconstructs the stimuli but fails on the main game to be interesting. But without additional details about the failure or examples of failure cases and a discussion about what these results imply outside the tasks they were tested for, I was not convinced by their utility. I would also urge the authors to have a speculative discussion on what successful inductive biases might look like.\n\n**Other Comments:**\n\nPage 2: Further more → Furthermore \n\nPage 3: On-hot-encoded → One-hot-encoded\n\nPage 4: Relational Frame Theory (RTF) →Relational Frame Theory (RFT)\n\nPage 6: Simplify → Simplifies\n\nPage 6: observes and have → observes and has\n\nPage 9: zsc → ZSC\n\nSome lines in the introduction and the related work sections are incredibly similar. Eg: \n\nThe following line is exactly the same in the Introduction and Related Works section.\n\nIntroduction and in Related Woks: \"Chaabouni et al. (2020) showed that, when a specific kind of compositionality is found in the emerging languages (the kind that scores high on the positional disentanglement (posdis) metric for compositionality that they proposed), then it is a sufficient condition for systematicity to emerge.\"\nI would advise the authors to make the introduction and related works sections more succinct.", "summary_of_the_paper": "The paper presents a new benchmark based on a referential communication game (where a speaker has to communicate the stimuli they observe to the listener and the listener has to find the right stimuli among distractors) to test for compositional learning behaviors. The stimuli vary over differently structured semantic spaces (every dimension can have different numbers of semantic types). Models are trained on a series of meta-referential games, where each game has a different semantic distribution structure (different number of semantic types) and are evaluated on zero-shot performance on a referential game with a new semantic structure. To represent the stimuli, the authors propose a continuous vector representation for different referents instead of using a one-hot embedding. Finally, the authors show how commonly used deep learning architectures fail on their benchmark, posing a challenge to the community.", "summary_of_the_review": "I wasn't convinced of the utility of the three main contributions of the paper outside the limited setting described in the paper: SCS, the meta-referential game and the failure of models on the same. Further, the paper is unclear in several places and does not provide enough details. Thus, in its current form, I recommend that the paper be rejected.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "6iHow_EeoC", "reviewer": "Reviewer_o5wV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the posed problem, using meta-RL for continual referential games with different semantic structures, is an interesting problem. However, the paper is difficult to follow, and details are missing from models and experiments.\n\n1- The paper is difficult to follow; especially the introduction where the authors give a lot of related work without a clear introduction of the problem and why it is important. I think it would help if the introduction is kept simple and references are moved to related work. \n\n- Upon first reading the abstract, I thought one of the main contributions of the work is a new benchmark for compositional learning. But, it is only mentioned on the last paragraph of the introduction without much explanation on what it entails. I think there needs to be more emphasis on what the benchmark is; how it is different from existing compositional benchmark, what it is measuring, how it is structured etc.\n\n- There are several concepts that the paper uses but are never defined such as \"positional disentanglement metric\" (where the speaker uses a language with this characteristic) or \"systematic generalization\" (that the paper is targeting).\n\n- I think more concrete examples are needed to make the paper more readable such as a real stimulus example (for Figure 1), sample conversations between two agents to understand compositional learning, sample tasks and meta-tasks, etc.\n\n- Sentences are in general long and repetitive in some cases such as the last sentence of abstract or \"differently semantically structured symbolic spaces\".\n\n2- I think more discussion on why OHE shouldn't be used and SCS should be preferred is needed. Stimuli are already discrete (Figure 1) and should come from a bounded vocabulary, nonetheless a very large one, why would using OHE be prohibitive for meta-learning? Considering that humans communicate with a finite vocabulary and can generate unlimited compositional structures, why can't we use a similar approach to represent stimuli? I think a more concrete example that shows a realistic continual task learning problem with discrete stimuli where the boundary of the stimuli can't be fixed and needs to be significantly increased over time is needed. \n\n- Your initial experimental results in Figure-2 also suggest that OHE clearly gives higher performance and SCS might be introducing unnecessary noise to the process.\n- It is also not clear to me whether the performance of SCS in Figure-2 can be improved or not. For example, can we use millions of updates to achieve results on par with OHE? Instead of sampling d(i) randomly, if you equally divide the interval [-1,1] into two sections and take N_{dim}=2, what would be the performance?\n\n3- For the experiments in Section 3.1., there is no setup. I think a more detailed setup that includes what is the baseline RL agent, how it is trained in this 2-player game, what is its architecture, how many conversations took place, etc.\n\n4- The setup for meta-referential games using SCS is not clear to me. If at any point in the future, we introduce more stimuli which increases the boundary, causing problems with OHE, this means that we are dividing the interval [-1, +1] into more and more sections. Wouldn't this cause intersections with previous SCS representations and lead to ambiguity? Are you suggesting that, over time the agents would forget some of the earlier tasks that they learned to accommodate for new tasks?\n- I think some ablations are needed here, including increasing the boundary of the stimuli gradually for new tasks to see learning progress (and forgetting), increasing number of latent dimensions N_{dim}.\n\n5- In Figure 4 (a), there is no improvement other than LSTM+Rec. This is strange to me as using only ~1K updates gives the same performance as using 8K updates. Could you explain the reason why?\n\n6- Could you explain what you mean by \"supportive training-purpose stimuli\" in Section 5.4? Do you mean these are supportive for the test task adaptation?\n\n7- Some grammatical mistakes:\n- Apostrophes are missing in some places, \"Neural networks' induction biases\", \"Both players' goals\".\n- \"Further more\" --> \"Furthermore\"\n- \"On the otherhand\" --> \"On the other hand\"\n-\"shceme\" --> \"scheme\"\n- \"we hope will spur\" --> \"we hope it will spur\"", "summary_of_the_paper": "This paper studies the problem of learning compositional behaviors through referential games. The authors first introduce the problem of fixed shape representations that are not suited for continual learning with infinitely many tasks. They propose a continuous representation of discrete stimulus by discretizing a small interval and randomly creating gaussian distributions at each interval. They then define the compositional learning problem using meta-reinforcement learning where a set of new tasks are introduced over time and two agents communicate to develop a compositional language. The authors present some empirical results where DNC and LSTM based agents are not well tailored to solve the problem and exhibits very low accuracy while the LSTM based agent is able to reconstruct an input stimulus successfully. ", "summary_of_the_review": "I think the overall problem of compositional learning through meta-referential games is interesting but the paper is hard to follow and its main contributions should be emphasized more; especially the benchmark should be detailed with more results, ablations and the new representation, SCS, should be clarified. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-McbUfD4UvY", "reviewer": "Reviewer_VGZD", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is an ambitious paper that could offer some promise. Learning to generalize compositionally is an intriguing idea. However, I think the work needs substantially more development along a number of axes: \n\nSCS vs. purely continuous representations\n---------------\n\nThis may partly be an issue of communication (see below), but it's not entirely clear to me why the authors don't consider fully continuous representations in addition to OHE or SCS ones. The argunments of prior work referenced below (e.g. Chalmers, 1990) show that continuous representations can be effective for compositional generalization, as do many of the empirical studies the authors exhibit in RL, as well as the successes of large language models, etc. All modern approaches rely on continuous representations, and often achieve some degree of effective compositional generalization. It certainly seems important to run a continuous-representation comparison condition to justify the value of the SCS encoding.\n\nIndeed, the authors (accurately) criticize OHE for not being able to extend without changing the shape of the vector representations. However, their SCS representation is subject to limitations as well, so far as I can see: it cannot adapt to more latent dimensions than are pre-allocated, and I suspect that it also would fail to extrapolate to dimensions with a larger $d(i)$ than the architecture saw in training (although this is speculation, and perhaps something that could/should be evaluated experimentally in a revised version of this work). \n\nFurthermore, constructing an SCS seems to require a) that the semantics of the task be simple enough to be stated as discrete values along a small number of latent dimensions, and b) that we be able to construct these representations from the inputs (which might generally be e.g. images). Indeed, continuous representations are used in basically all modern applications for just this reason---we don't know how to construct an appropriate discrete representation for the space, and it's not clear if one even necessarily exists. Thus, in order for this method to be applicable beyond toy settings, it would be useful for the paper to explain how this might work. Ideally, the authors would *demonstrate* such success on existing benchmarks for compositional generalization (see below). \n\nClarity\n------------\n\nNeither the paradigm nor the results are stated in sufficient detail or clarity to understand how to build upon the work. For example: \n\n* The rule-based speaker the authors use to evaluate uses compositional language, but what exact form does this take? This should be clearly stated, since it is central to understanding the main experimental evaluations. \n* The only description of how the agent's take communication actions is that the output is \"a distribution over 29 actions, which corresponds to an action space combining both the language output and the decision output.\" How are these combined? \n\nThese kind of details can make a substantial difference in RL, and should certainly be reported. Furthermore, there are many other clarity issues, e.g.:\n\n* The paper spends a great deal of time discussing issues like \"object-centric\" representations vs. \"stimulus-centric\" ones, but I do not see how such notions could apply to the SCS representation as suggested by the paper. Or are the stimuli represented differently as inputs to the agents? Even this detail of the experimental evaluation is not clearly stated.\n* What the paper calls a \"One-Hot Encoding\" OHE seems to really be a multi-hot encoding (that is, it is one-hot along each dimension, but a single stimulus has multiple  1s within it). However, this is not clearly explained, and it's only really communicated if the reader carefully examines figure 1. A point like this should be clarified in the text. \n\n\nEvaluation\n------------\n\nThe results of this approach are barely-above chance performance (29% at most, where chance is 25%). While this may be interesting in some sense, it is not a particularly compelling endorsement of the SCS representation.\n\nIn addition, even this experimental evaluation is limited. First, the results suggest poor tuning of the  hyperparameters: the DNC performs worse than an LSTM memory, but a DNC *contains* an LSTM in the controller, and so should almost certainly perform better (even if just by learning to ignore the memory part). Of course, these are challenging architectures to optimize, but this means that the claims of the paper are suspect: perhaps the performance is low for reasons of poor training, rather than inherent difficult of the task. Furthermore, the authors perform effectively no ablation studies, or further evaluation of which aspects of the approach matter. All of these would be necessary to make a valuable contribution to the literature.  \nFinally, it would be much more interesting if the authors could demonstrate the value of their approach on existing compositional generalization challenges such as SCAN or gSCAN (which they cite), where baselines do exist. Could SCS offer benefits in these more complex settings? This would force the authors to grapple with some of the challenges outlined above about using SCS in environments with slightly more complexity (but still much simpler and more amenable to structured representations than the stimuli used for reference games in cognitive psychology). \n\nLiterature\n-----------\n\nAs the authors note, compositional generalization has been a major topic of research for some time. The work needs to be better situated within this broader literature, to clarify its contributions. For example, the author's points about discrete vs. continuous representations overlap with long-standing replies to Fodor & Pylyshyn's claims about compositionality. For example, see Chalmers (1990) or Smolensky (1987), as well as much of Smolensky's subsequent research (e.g. Smolensky, 1990; McCoy et al., 2018). This paper would convey its contributions much more clearly if it were situated within this broader literature. \n\nI don't think it's as essential to engage with with, but the authors may also be interested in Santoro et al. (2021), an opinion piece which discusses a number of issues related to several aspects of this paper, including discussions of discrete vs. continuous symbol representations, and arguments that communication provides a unique path to instilling symbols. \n\n\n\n\n\n\n\nReferences\n-------------\n\nChalmers, D. (1990, July). Why Fodor and Pylyshyn were wrong: The simplest refutation. In Proceedings of the Twelfth Annual Conference of the Cognitive Science Society, Cambridge, Mass (pp. 340-347).\n\nSantoro, A., et al. (2021). Symbolic behaviour in artificial intelligence. arXiv preprint arXiv:2102.03406.\n\nSmolensky, P. (1987). The constituent structure of connectionist mental states: A reply to Fodor and Pylyshyn. Southern Journal of Philosophy, 26, 137-161.\n\nSmolensky, P. (1990). Tensor product variable binding and the representation of symbolic structures in connectionist systems. Artificial intelligence, 46(1-2), 159-216.\n\nMcCoy, R. T., Linzen, T., Dunbar, E., & Smolensky, P. (2018). RNNs implicitly implement tensor product representations. arXiv preprint arXiv:1812.08718.", "summary_of_the_paper": "This paper proposes to investigate meta-learning to generalize compositionally. This is an important challenge. The paper proposes a meta-learning setting consisting of reference games, which are a classic paradigm for investigating communication. The paper proposes a hybrid symbolic-continuous stimulus representation, and explores how different agents use this representation scheme within their paradigm. ", "summary_of_the_review": "This is an ambitious paper that could offer some promise. Some of the ideas are intriguing. However, to achieve its potential, this paper needs substantially more development conceptually, in terms of evaluation, in clarity of communication, and in engagement with the existing literature. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers have agreed that the topic of evaluating compositional skills of agents is an important one and cast it as compositional learning as meta-reinforcement learning is an interesting approach. At the same time, reviewers have raised concerns with respect to the benchmark itself, the exposition and clarify of the ideas as well as the experimental evidence used to support some of the claims. The authors have not provided an author response but have acknowledged the reviewers feedback. 

As this paper stands I cannot recommend acceptance for the current manuscript.
Decision: Reject