Meta-Referential Games to Learn Compositional Learning Behaviours

1 INTRODUCTION . Framing the ability to learn to generalise compositionally from a limited set of combinations of supportive stimulus components to a larger set of novel combinations of those same supportive stimulus components as compositional learning behaviours , we acknowledge compositional learning behaviours as a valuable feat of intelligence that human beings often rely on , and assume their collaborative partners to use similarly . In order to build artificial agents able to collaborate with human beings , it is important to endow the former with similar abilities . Thus , we propose to investigate state-of-the-art artificial agents abilities to exhibit compositional learning behaviours . Compositional Language Emergence as a Proxy for Compositional Behaviours . The field of language emergence raises the question of how to make artificial languages emerge with similar properties to natural languages , with compositionality at the forefront of those properties ( Lazaridou et al. , 2018 ; Baroni , 2019 ; Guo et al. , 2019 ; Li & Bowling , 2019 ; Ren et al. , 2020 ) . Referential game ( Lewis , 1969 ) variants are the primary tools used to make language emerge between a pair or population of artificial agents ( Denamganaı̈ & Walker , 2020a ) . In this context , it has been shown that emerging languages are far from being ‘ natural-like ’ protolanguages ( Kottur et al. , 2017 ; Chaabouni et al. , 2019a ; b ) , but sufficient conditions can be found to further the emergence of compositional languages and generalising learned representations ( e.g . ( Kottur et al. , 2017 ; Lazaridou et al. , 2018 ; Choi et al. , 2018 ; Bogin et al. , 2018 ; Guo et al. , 2019 ; Korbak et al. , 2019 ; Chaabouni et al. , 2020 ; Denamganaı̈ & Walker , 2020b ) ) . Nevertheless , the ability of deep-learning agents to generalise compositionally in a systematic fashion has been called into question , especially when it comes to language grounding in general ( Hill et al. , 2019b ) , on relational reasoning tasks ( Bahdanau et al. , 2019 ) , or on the SCAN benchmark ( Lake & Baroni , 2018 ; Loula et al. , 2018 ; Liška et al. , 2018 ) , and more recently the gSCAN benchmark ( Ruis et al. , 2020 ) . Neural networks induction biases have been investigated towards finding necessary conditions that favour the emergence of compositional generalisation/systematicity ( Hill et al. , 2019b ; Słowik et al. , 2020 ; Korrel et al. , 2019 ; Lake , 2019 ; Russin et al. , 2019 ) . Chaabouni et al . ( 2020 ) showed that , when a specific kind of compositionality is found in the emerging languages ( the kind that scores high on the positional disentanglement ( posdis ) metric for compositionality that they proposed ) , then it is a sufficient condition for systematicity to emerge . And , more importantly , they showed that emergence of a compositional language ( in the sense of any of the existing compositionality-testing metric , e.g . topographic similarity ( Brighton & Kirby , 2006 ) ) is not a necessary condition for systematic generalisation ( as evaluated by a zero-shot compositional learning test ) as non-compositional , emerging languages have been shown to support systematicity of the pair of agents wielding them in a ( generative ) referential game . In this work , we focus directly on systematic generalisation/systematicity as a behaviour , as success in our proposed task is directly related to the agents ability to exhibit compositional learning behaviours , thus investigating whether they can generalise compositionally in a systematic and online fashion ( see Section 4 ) . Moreover , contrary to our framework , when studying compositionality in the context of any referential game variant instantiated with a given dataset of stimuli ( of any nature , e.g . one-hotencoded ( Kottur et al. , 2017 ; Chaabouni et al. , 2020 ; Ren et al. , 2020 ; Resnick et al. , 2019 ) , visual ( Lazaridou et al. , 2018 ; Choi et al. , 2018 ) , or multi-modal ( Evtimova et al. , 2017 ) ) , then the trained pair of agents only exhibits compositional behaviours for said dataset within the constraint of the training , thus failing short of learning compositional behaviours that generalises to novel situations/stimuli/datasets . In other words , agents are failing to learn compositional learning behaviours . Our framework aims to remedy this gap via , firstly , a meta-learning framing of the task and , secondly , via the introduction of a novel stimulus representation that compromises between the one-hot-encoded representation of most symbolic stimuli benchmarks and the continuous representations akin to latent embeddings as found in the unsupervised learning approaches , without sacrificing the symbolic aspects ( see Section 3 ) . Meta-Referential Games for Compositional Learning Behaviours . Learning compositional learning behaviours pertains to meta-learning . In order to frame referential games in a meta-learning context where successful agents are able to generalise their abilities to exhibit compositional behaviours onto novel situations/datasets , it is necessary to provide the agent at training-time with a distribution of tasks/stimuli/datasets that encompasses the kind of tasks/stimuli/datasets that the agent will experience after being deployed . For instance , using a set of datasets with different generative factors/attributes as our distribution of tasks would force us to focus on only one modality shared among all datasets , for instance the visual modality . Doing so would add to our work the assumption of working with visual stimuli only , which is quite constraining . Instead , we choose to assume that stimuli are disentangled , independently of their nature . Thus , using one-hot-encoded ( OHE ) vectors , that are disentangled abstract stimuli , would fit the bill with regards to our assumption and our work would retain some external validity . It would be applicable whenever stimuli can be discretely categorised over a set of generative factors/attributes with discrete values . Further more , once a semantic structure is chosen , i.e . a number of generative factors/attributes , Ndim , and a number of possible values for each ( d ( i ) ) i∈ [ 1 ; Ndim ] , it is impossible to extend our meta-distribution to differently semantically structured stimuli without changing the shape of our OHE vectors , dstim = ∑ i∈ [ 1 ; Ndim ] d ( i ) . Most state-of-the-art deep learning architectures rely on the assumption that the dimensions of their input spaces are constant . Thus , in order to provide a meta-distribution over differently semantically structured stimuli spaces , we propose a different representation , entitled symbolic continuous stimulus representation ( SCS ) , it is detailed in Section 3 . Using this SCS representation , we can meta-train agents in a meta-referential game settings where the semantic structure observed can be randomised over without changing the shape of the stimulus space and without the overly constraining assumption that the representation must be discrete . Our contributions are threefold : • We propose the Symbolic Continuous Stimulus ( SCS ) representation as a versatile encoding of symbolic spaces , which , on the contrary to one-hot-encoded representation , allows for infinitely many semantic structure to be instantiated without changing the shape of the representation , and rely on continuous values that are more akin to the kind of stimuli found in the real world , as opposed to discrete valued one-hot-encodings . • We cast the problem of learning compositional behaviours as a meta-reinforcement learning problem , using ( discriminative ) referential games , containing a meta-reinforcement learn- ing formulation of the binding problem , thus allowing it to be studied under the lens of state-of-the-art RL algorithms . • We provide baseline results on the single-agent task of learning compositional learning behaviours , using state-of-the-art RL agents , and show that our proposed benchmark is a compelling challenge that we hope will spur the research community towards developing more capable artificial agents . 2 BACKGROUND & RELATED WORKS . Referential games are at an interface between the language processing subfields of language emergence , language grounding , and , when not using on-hot-encoded stimuli , the computer vision subfield of unsupervised representation learning . While language emergence raises the question of how to make artificial languages emerge with similar properties to natural languages , or at least ‘ naturallike ’ protolanguages , with compositionality at the forefront of those properties ( Baroni , 2019 ; Guo et al. , 2019 ; Li & Bowling , 2019 ; Ren et al. , 2020 ) , language grounding is concerned with the ability to ground the meaning of ( natural ) language utterances into some sensory processes , with the visual modality being the main focus of research . On one hand , emerging artificial languages ’ compositionality has been shown to further the learnability of said languages ( Kirby , 2002 ; Smith et al. , 2003 ; Brighton , 2002 ; Li & Bowling , 2019 ) and , on the other hand , natural languages ’ compositionality promises to increase the generalisation ability of the artificial agent that would be able to rely on them as a grounding signal , as it has been found to produce learned representations that generalise , when measured in terms of the data-efficiency of subsequent transfer and/or curriculum learning ( Higgins et al. , 2017 ; Mordatch & Abbeel ; Moritz Hermann et al . ; Jiang et al. , 2019 ) . More in touch with the current context of this study , Chaabouni et al . ( 2020 ) showed that , when a specific kind of compositionality is found in the emerging languages ( the kind that scores high on the positional disentanglement ( posdis ) metric for compositionality that they proposed ) , then it is a sufficient condition for systematicity to emerge . Language Compositionality & Compositional Systematic Generalisation/Systematicity . As a concept , compositionality has been the focus of many definition attempts . For instance , it can be defined as “ the algebraic capacity to understand and produce novel combinations from known components ” ( Loula et al . ( 2018 ) referring to Montague ( 1970 ) ) or as the property according to which “ the meaning of a complex expression is a function of the meaning of its immediate syntactic parts and the way in which they are combined ” ( Krifka , 2001 ) . Although difficult to define , the commmunity seem to agree on the fact that it would enable learning agents to exhibit systematic generalisation abilities ( also referred to as combinatorial generalisation ( Battaglia et al. ) ) . Some of the ambiguities that come with those loose definitions start to be better understood and explained , as in the work of Hupkes et al . ( 2019 ) . While often studied in relation to languages , it is usually defined with a focus on behaviours . In this paper , we refer to compositional behaviours as “ the ability to entertain a given thought implies the ability to entertain thoughts with semantically related contents ” ( Fodor et al. , 1988 ) , and thus use it interchangeably with systematicity , following the classification made by Hupkes et al . ( 2019 ) . Compositionality , as a property of languages , will be referred to as linguistic compositionality , in this paper . It can be difficult to measure . Brighton & Kirby ( 2006 ) ’ s topographic similarity ( topsim ) which is acknowledged by the research community as the main quantitative metric for compositionality ( Lazaridou et al. , 2018 ; Guo et al. , 2019 ; Słowik et al. , 2020 ; Chaabouni et al. , 2020 ; Ren et al. , 2020 ) . Recently , taking inspiration from disentanglement metrics , Chaabouni et al . ( 2020 ) proposed two new metrics entitled posdis ( positional disentanglement metric ) and bosdis ( bag-ofsymbols disentanglement metric ) , that have been shown to be differently ‘ opinionated ’ in the sense that they each seem to capture different ways in which a language can be shown to be ( linguistically ) compositional . Binding Problem & Meta-Learning . Following Greff et al . ( 2020 ) , we refer to the binding problem as the “ inability to dynamically and flexibly bind information that is distributed throughout the network ” of deep learning architectures . We note that relational responding ( “ adjusting the ( taskspecific ) response to an object based on its relation to other objects ” ; for instance , one way to facilitate it in a neural network “ is to organise its internal information flow ( i.e . computations ) in a way that reflects the graph structure of relations and objects ” ) is central to solving the binding problem instantiated in our proposed benchmark . Relational Frame Theory ( RTF ) distinguishes “ two types of entailment that humans primarily use to derive ( unobserved ) relations : mutual entailment [ ( derive additional relations between two objects based on a given relation between them ) ] and combinatorial entailment [ ( derive new relations between two objects , based on their relations with a shared third object ) ] ” . Within those terms , the ability to perform ( sequential ) combinatorial entailment is at the center of the benchmark challenges .

Summary: In this paper, the authors propose a new benchmark to investigate state-of-the-art artificial agents' abilities to exhibit compositional learning behaviors. The authors propose a Symbolic Continuous Stimulus (SCS) representation and cast the problem of learning compositional behaviors as a meta-reinforcement learning problem. The authors compare state-of-the-art RL methods on solving the single-agent task of learning compositional learning behaviors.