Meta-Learning of Structured Task Distributions in Humans and Machines

1 INTRODUCTION . While machine learning has supported tremendous progress in artificial intelligence , a major weakness – especially in comparison to humans – has been its relative inability to learn structured representations , such as compositional grammar rules , causal graphs , discrete symbolic objects , etc . ( Lake et al. , 2017 ) . One way that humans acquire these structured forms of reasoning is via “ learning-to-learn ” , in which we improve our learning strategies over time to give rise to better reasoning strategies ( Thrun & Pratt , 1998 ; Griffiths et al. , 2019 ; Botvinick et al. , 2019 ) . Inspired by this , researchers have renewed investigations into meta-learning . Under this approach , a model is trained on a family of learning tasks based on structured representations such that they achieve better performance across the task distribution . This approach has demonstrated the acquisition of sophisticated abilities including model-based learning ( Wang et al. , 2016 ) , causal reasoning ( Dasgupta et al. , 2019 ) , compositional generalization ( Lake , 2019 ) , linguistic structure ( McCoy et al. , 2020 ) , and theory of mind ( Rabinowitz et al. , 2018 ) , all in relatively simple neural network models . The meta-learning approach , along with interaction with designed environments , has also been suggested as a general way to automatically generate artificial intelligence ( Clune , 2019 ) . These approaches have made great strides , and have great promise , toward closing the gap between human and machine learning . However , in this paper , we argue that significant challenges remain in how we evaluate whether structured forms of reasoning have indeed been acquired . There are often multiple strategies that can result in good meta-test performance , and there is no guarantee a priori that meta-learners will learn the strategies we intend when generating the training distribution . Previous work on metalearning structured representations do partially acknowledge this . In this paper , we highlight these challenges more generally . At the end of the day , meta-learning is simply another learning problem . And similar to any vanilla learning algorithm , meta-learners themselves have inductive biases ( which we term meta-inductive bias ) . Note that meta-learning is a way to learn inductive biases for vanilla learning algorithms Grant et al . ( 2018 ) . Here , we consider the fact the meta-learners themselves have inductive biases that impact the kinds of strategies ( and inductive biases ) they prefer to learn . In this work , the kind of structure we study is that imposed by compositionality , where simple rules can be recursively combined to generate complexity ( Fodor et al. , 1988 ) . Previous work demonstrates that some aspects of compositionality can be meta-learned ( Lake , 2019 ) . Here , we introduce a broader class of compositionally generated task environments using an explicit generative grammar , in an interactive reinforcement learning setting . A key contribution of our work is to also develop control task environments that are not generated using the same simple recursively applied rules , but are comparable in statistical complexity . We provide a rigorous comparison between human and meta-learning agent behavior in tasks performed in distributions of environments of each type . We show through three different analyses that human behavior is consistent with having learned the structure that results from our compositional rules in the structured environments . In contrast , despite training on distributions that contain this structure , standard meta-learning agents instead prefer ( i.e . have a meta-inductive bias toward ) more global statistical patterns that are a downstream consequence of these low-dimensional rules . Our results show that simply doing well at meta-test on a tasks in a distribution of structured environments does not necessarily indicate meta-learning of that structure . We therefore argue that architectural inductive biases still play a crucial role in the kinds of structure acquired by meta-learners , and simply embedding the requisite structure in a training task distribution may not be adequate . 2 EMBEDDING STRUCTURE IN A TASK DISTRIBUTION . In this work , we define a broad family of task distributions in which tasks take place in environments generated from abstract compositional structures , by recursively composing those environments using simple , low-dimensional rules . Previous work on such datasets ( Lake & Baroni , 2018 ; Johnson et al. , 2017 ) focuses primarily on language . Here we instead directly consider the domain of structure learning . This is a fundamental tenet of human cognition and has been linked to how humans learn quickly in novel environments ( Tenenbaum et al. , 2011 ; Mark et al. , 2020 ) . Structure learning is required in a vast range of domains : from planning ( understanding an interrelated sequence of steps for cooking ) , category learning ( the hierarchical organization of biological species ) , to social inference ( understanding a chain of command at the workplace , or social cliques in a high school ) . A task distribution based on structure learning can therefore be embedded into several domains relevant for machine learning . Kemp & Tenenbaum ( 2008 ) provide a model for how people infer such structure . They present a probabilistic context-free graph grammar that produces a space of possible structures , over which humans do inference . A grammar consists of a start symbol S , terminal and non-terminal symbols Σ and V , as well as a set of production rules R. Different structural forms arise from recursively applying these production rules . This framework allows us to specify abstract structures ( via the grammar ) and to produce various instantiations of this abstract structure ( via the noisy generation process ) , naturally producing different families of task environments , henceforth referred to as task distributions . We consider three structures : chains , trees , and loops . These exist in the real world across multiple domains . Chains describe objects on a one-dimensional spectrum , like people on the left-right political spectrum . Trees describe objects organized in hierarchies , like evolutionary trees . Loops describe cycles , like the four seasons . Here we embed these structures into a grid-based task . Exploration on a grid is an extensively studied problem in machine learning , particularly in reinforcement learning . Further , it is also a task that is easy for humans to perform on online crowdsourcing platforms – but not trivially so . This allows us to directly compare human and machine performance on the same task . Fig . 1 displays the symbols of the grammar we use and the production rules that give rise to grids of different structural forms . 2.1 A TASK TO TEST STRUCTURE LEARNING . Here we describe the specific task built atop this embedding of structural forms . We use a tile revealing task on the grid . Humans as well as agents are shown a 7 × 7 grid of tiles , which are initially white except for one red tile . The first red tile revealed at the beginning of the episode is the same tile as the initial start tile of the grid ’ s generative process ( see Fig . 1 ) . Clicking white tiles reveal them to be either red or blue . The episode finishes when the agent reveals all the red tiles . There is a reward for each red tile revealed , and a penalty for every blue tile revealed . The goal therefore is to reveal all the red tiles while revealing as few blue tiles as possible . The particular configuration of the red tiles defines a single task . The distribution of tasks for meta-learning is defined by the grammar from which these structures are sampled . Here , we randomly sampled from a uniform mixture of chains , trees , and loops as defined in Fig . 1 . 2.2 A STATISTICALLY EQUIVALENT NULL TASK DISTRIBUTION . Previous approaches to evaluating whether machine-learning systems can extract compositional structure ( Lake & Baroni , 2018 ; Dasgupta et al. , 2018 ) have relied on examining average performance on held-out examples from compositionally structured task distributions . However , we argue that this often confounds whether a system has truly internalized this underlying structure or whether it is relying on statistical patterns that come about as a consequence of compositional rules . To directly examine whether structured reasoning is a factor in how humans and meta-learning agents perform this task , we need a control task distribution that is similar in statistical complexity , by generating one based on those statistics rather than the direct use of the compositional grammar . To this end , we trained a fully connected neural network ( 3 layers , 49 units each ) to learn the conditional distribution of each tile given all the other tiles on the compositional boards . Note that these conditional distributions contain all the relevant statistical information about the boards . We do this by training on an objective inspired by masked language models like BERT ( Devlin et al. , 2018 ) . The network was given a compositional board with a random tile masked out and trained to reproduce the entire board including the randomly masked tile . The loss was binary cross entropy between the predicted and actual masked tiles . The network was trained on all possible compositional boards for 104 epochs , and achieved a training accuracy of ∼99 % . We then sampled boards from these conditionals with Gibbs sampling . We started with a grid in which each tile is randomly set to red or blue with probability 0.5 . We then masked out a tile and ran the grid through the network to get the conditional probability of the tile being red given the other tiles , turning the tile red with that probability . We repeated this by masking each tile in the 7 × 7 grid ( in a random order ) to complete a single Gibbs sweep , and repeated this whole Gibbs sweep 20 times to generate a single sample . We refer to the distribution of boards generated this way as the null task distribution . Fig . 2 shows example compositional and null distribution grids . While the statistical structure looks similar , the non-compositional null boards shown could not have been generated by the grammar in Fig . 1 . The conditional distributions for the two distributions are similar by design ; we further quantify statistical similarity using Ising statistics ( Zhang , 2007 ) . We compared the 0th order , 1st order , and 2nd order effects defined as follows . The 0th order statistic corresponds to the number of red minus number of blue tiles . The 1st order statistic counts the number of agreeing neighbours ( vertically or horizontally adjacent ) minus the disagreeing ones , where agreeing means being of the same color . The 2nd order statistic is the number of triples ( tile + its neighbor + its neighbor ’ s neighbor ) that agree , minus those that don ’ t . Fig . 2b shows that the two distributions are not significantly different in terms of the Ising statistics measured ( p > 0.05 for all three orders ) . The principal difference between these two task distributions is the way in which they were generated . The compositional task distribution was generated through the recursive application of simple , low-dimensional rules that generates a mixture of three discrete structures , whereas the null task distribution was generated through a more complex Gibbs sampling procedure that is not explicitly compositional and does not utilize explicit simple , low-dimensional rules . Although it is true that some boards within the null task distribution may be consistent with a simple compositional grammar , the distribution as a whole was not generated through a compositional grammar .

Summary: This work is an exploration of model behaviour upon meta-learning tasks with compositional structure. The authors discover that, unlike humans, machine learning models do not readily pick up on the underlying compositional generative structure of a set of tasks, and hence cannot match the performance of humans. Conversely, when the task is structured to leverage other statistical patterns, models do well.