Meta-Learning of Structured Task Distributions in Humans and Machines
Authors: Sreejan Kumar
Ishita Dasgupta
Jonathan Cohen
Nathaniel Daw
Thomas Griffiths
Venue: ICLR 2021 Poster (2021)
Keywords: meta-learning
human cognition
reinforcement learning
compositionality

Abstract: In recent years, meta-learning, in which a model is trained on a family of tasks (i.e. a task distribution), has emerged as an approach to training neural networks to perform tasks that were previously assumed to require structured representations, making strides toward closing the gap between humans and machines. However, we argue that evaluating meta-learning remains a challenge, and can miss whether meta-learning actually uses the structure embedded within the tasks. These meta-learners might therefore still be significantly different from humans learners. To demonstrate this difference, we first define a new meta-reinforcement learning task in which a structured task distribution is generated using a compositional grammar. We then introduce a novel approach to constructing a "null task distribution" with the same statistical complexity as this structured task distribution but without the explicit rule-based structure used to generate the structured task. We train a standard meta-learning agent, a recurrent network trained with model-free reinforcement learning, and compare it with human performance across the two task distributions. We find a double dissociation in which humans do better in the structured task distribution whereas agents do better in the null task distribution -- despite comparable statistical complexity. This work highlights that multiple strategies can achieve reasonable meta-test performance, and that careful construction of control task distributions is a valuable way to understand which strategies meta-learners acquire, and how they might differ from humans. 

Published as a conference paper at ICLR 2021
META- LEARNIGG OF STRUCTURED TASK DISTRIBU- TIONS IN HUMANS AND MACHINES
Sreejan Kumart, Ishita Dasgupta", Jonathan D. Co Nathaniel D. Dawl.,, and Thomas L. Griffiths23
Princeton Neuroscience Institute
Depaatment of Computer Science, Princcton University Department of Psychology, Princeton University
ABSTRACT
In recent years, meta-learning, in which model is trained on family of tasks (i.e. a task distribution). has emerged as an approach to training neural networks to perform tasks that were previously assumed to require structured representations, making strides toward closing the gap between humans and machines. However, we argue that evaluating meta-learning remains a challenge, and can miss whether meta-learning actually uses the structure embedded within the tasks. These meta- learners might therefore still be significantly different from humans learners. To demonstrate this difference, we first define task in which a structured task distribution generated using a compositional grammar. We then introduce a novel approach constructing "null task dis- tribution" with the same statistical complexity as this structured task distribution but without the explicit rule-based structure used to generate the structured task. We train standard meta-learning agent, a recurrent network trained with model- free reinforcement learning, and compare it with human performance across the two task distributions. We find a double dissociation in which humans do better in the structured task distribution whereas agents do better in the null task dis- tribution despite comparable statistical complexity. This work highlights that multiple strategies can achieve reasonable meta-test performance, and that careful construction of control task distributions is valuable way understand which strategies meta-learners acquire, and how they might differ from humans.
meta-reinforremmet learning
INTRODUCTION
While machine learning has supported tremendous progress in artificial intelligence, major weak- ness especially in comparison to humans - has been its relative inability to learn structured representations, such compositional grammar rules, causal graphs, discrete symbolic objects, etc. (Lake et al. 2017). One way that humans acquire these structured forms of reasoning is via "learning-to-leamm" " in which we improve our learning strategies over time give rise better reasoning strategies (Thrun & Pratt 1998; Griffiths et al. 2019 Botvinick et al. (201)). Inspired by this, researchers have renewed investigations into meta-learning. Under this approach, model is trained on family of learning tasks based on structured representations such that they achieve better performance across the task distribution. This approach has demonstrated the acquisition of sophisticated abilities including model-based learning (Wang et al.) 2016). causal reasoning (Das-) gupta et al. 2019), compositional generalization (Lake) (201)). linguistic structure (MeCoy et al. 2020), and theory of mind (Rabinowitz et al.) (018). all in relatively simple neural network mod- els. The meta-learning approach, along with interaction with designed environments, has also been suggested general way to automatically generate artificial intelligence (Clune) 2019). These approaches have made great strides, and have great promise, toward closing the gap between human and machine learning.
However, in this paper. we argue that significant challenges remain in how we evaluate whether structured forms of reasoning have indeed been acquired. There are often multiple strategies that

---

Published as a conference paper at ICLR 2021
result in good meta-test performance, and there is no guarantee priori that meta-learners will learn the strategies we intend when generating the training distribution. Previous work on meta- learning structured representations do partially acknowledge this. In this paper, highlight these challenges more generally. At the end of the day, meta-learning is simply another learning problem. And similar any vanilla learning algorithm, meta-learners themselves have inductive biases (which we term meta-inda induiiiduetivve bias). Note that meta-learning is a way to learn inductive biases for vanilla learning algorithms Grant al. (2018). Here, we consider the fact the meta-learners themselves have inductive biases that impact the kinds of strategies (and inductive biases) they prefer to learn.
In this work, the kind of structure we study is that imposed by compositionality. where simple rules can be recursively combined to generate complexity (Fodor et al. 1988). Previous work demon- strates that some aspects of compositionality can be meta-learned (019). Here, we introduce a broader class of compositionally generated task environments using an explicit generative grammar, in an interactive reinforcement learning setting. A key contribution of our work is to also develop control task environments that are not generated using the same simple recursively applied rules, but are comparable in statistical complexity. We provide a rigorous comparison between human and meta-learning agent behavior in tasks performed in distributions of environments of each type. We show through three different analyses that human behavior is consistent with having learned the structure that results from our compositional rules in the structured environments. In contrast, despite training on distributions that contain this structure, standard meta-learning agents instead prefer (i.e. have meta-inductive bias toward) more global statistical patterns that are a downstream consequence of these low-dimensional rules. Our results show that simply doing well at meta-test on tasks in a distribution of structured environments does not necessarily indicate meta-learning of that structure. We therefore argue that architectural inductive biases still play a crucial role in the kinds of structure acquired by leannes, and simply embedding the requisite structure in a training task distribution may not be adequate.
2 EMBEDDING STRUCTURE IN A TASK DISTRIBUTION
In this work, we define a broad family of task distributions in which tasks take place in environments generated from abstract compositional structures, by recursively composing those environments us- ing simple. low-dimensional rules. Previous work on such datasets (Lake & Baroni 2018: Johnson al. (017)) focuses primarily on language. Here we instead directly consider the domain of struc- ture learning. This is fundamental tenet of human cognition and has been linked to how humans learn quickly in novel environments (Tenenbaum et al., 2011: Mark et al., 2020). Structure learning is required in vast range of domains: from planning (understanding steps for cooking), category learning (the hicrarchical organization of biological species), to social inference (understanding chain of command at the workplace, social cliques in a high school). A task distribution based on structure learning can therefore be embedded into several domains relevant for machine learning.
interrelated sequence of
Kemp & Tenenbaum 2008) provide a model for how people infer such structure. They present a probabilistic context-free graph grammar that produces space of possible structures, over which humans do inference. A grammar consists of start symbol S, terminal and non-terminal symbols Σ and V. as well as set of production rules R. Different structural forms arise from recursively applying these production rules. This framework allows us to specify abstract structures (via the grammar) and to produce various instantiations of this abstract structure (via the noisy generation process), naturally producing different families of task environments, henceforth referred to as task distributions.
We consider three structures: chains, trees, and loops. These exist in the real world across multi- ple domains. Chains describe objects on a one-dimensional spectrum, like people on the left-right political spectrum. Trees describe objects organized in hierarchies, like evolutionary trees. Loops describe cycles, like the four seasons. Here we embed these structures into a grid-based task.
Exploration grid is extensively studied problem in machine learning, particularly in rein- forcement learning. Further, it is also a task that is easy for humans to perform on online crowd- sourcing platforms but not trivially so. This allows us to directly compare human and machine performance on the same task. Fig. displays the symbols of the grammar we use and the produc- tion rules that give rise to grids of different structural forms.
2

---

Published as a conference paper at ICLR 2021
A
B
Grammar Symbols
Production Rules
Terminal symbols (V)
Chain
Nonterminal symbols
Tree
(S)
Start (S):
Loop
Figure I: Generative Grammar (A) Grammar symbols and (B) production rules. board formed by beginning with the start symbol and recursively applying production rules until only terminal symbols (red and blue tiles) are left. Each production rule either adds non-terminal symbol (from first column to second) terminal symbol (from second column to third) with 0.5 probability.
2.1 A TASK TO TEST STRUCTURE LEARNING
Here we describe the specific task built atop this embedding of structural forms. We use tile revealing task the grid. Humans well agents shown 7 x 7 grid of tiles, which are initially white except for one red tile. The first red tile revealed at the beginning of the episode is the same tile as the initial start tile of the grid's generative process (see Fig. D Clicking white tiles reveal them to be either red or blue. The episode finishes when the agent reveals all the red tiles. There is reward for each red tile revealed, and penalty for every blue tile revealed. The goal therefore is to reveal all the red tiles while revealing as few blue tiles as possible. The particular configuration of the red tiles defines single task. The distribution of tasks for meta-learning is defined by the grammar from which these structures are sampled. Here, we randomly sampled from uniform mixture of chains, trees, and loops as defined in Fig.
2.2 STATISTICALLY EQUIVALENT NULL TASK DISTRIBUTION
Previous approaches to evaluating whether machine-learning systems can extract compositional structure (Lake & Baroni 2018, Dasgupta et al. 2018) have relied on examining average perfor- mance on held-out examples from compositionally structured task distributions. However, we argue that this often confounds whether system has truly internalized this underlying structure or whether is relying on statistical patterns that come about as consequence of compositional rules.
To directly examine whether structured reasoning is factor in how humans and meta-learning agents perform this task, we need a control task distribution that is similar in statistical complexity, by generating based those statistics rather than the direct To this end, we trained a fully connected neural network (3 layers, 49 units each) to learn the condi- tional distribution of each tile given all the other tiles on the compositional boards. Note that these conditional distributions contain all the relevant statistical information about the boards. We do this by training an objective inspired by masked language models like BERT (Devlin et al. 2018). The network was given compositional board with a random tile masked out and trained to reproduce the entire board including the randomly masked tile. The loss was binary cross entropy between the predicted and actual masked tiles. The network was trained on all possible compositional boards for 10a epochs, and achieved a training accuracy of ~99%.
of the compositional grammar.
We then sampled boards from these conditionals with Gibbs sampling. We started with grid in which each tile randomly set to red or blue with probability 0.5. We then masked out a tile and ran the grid through the network to get the conditional probability of the tile being red given the other tiles, turning the tile red with that probability. We repeated this by masking each tile in the 7 grid (in random order) complete single Gibbs sweep, and repeated this whole Gibbs sweep 20 times to generate a single sample. We refer to the distribution of boards generated this way as the null task distribution. Fig. shows example compositional and null distribution grids.

---

Published as a conference paper at ICLR 2021
A
Compositional
Non-Compauitional
B
Compositional
Board Statistics (Maitch-Nonn (och-oonmmach))
60-
40
20
0th Urder
Order
2nd Order
Figure 2: Comparing compositional and null task distributions (A) Example compositional and null distribution boards. Compositional boards are distinctly either chain, trees, or loops while null boards have similar statistical properties but don't necessarily obey the recursive rules used gen- erate compositional boards. (B) Ising statistics across the two task distributions, Error bars are 95% non-parametric bootstrap confidence intervals across different boards of the respective distribution.
While the statistical structure looks similar, the non-compositional null boards shown could not have been generated by the grammar in Fig. [ The conditional distributions for the two distributions are similar by design; we further quantify statistical similarity using Ising statistics (Zhang/ 2007). We compared the Oth order, 1st order, and 2nd order effects defined as follows. The Oth order statistic corresponds to the number of red minus number blue tiles. The 1st order statistic counts the number of agreeing neighbours (vertically or horizontally adjacent) minus the disagreeing ones, where agreeing means being of the same color. The 2nd order statistic is the number of triples (tile its neighbor its neighbor's neighbor) that agree, minus those that don't. Fig. P shows that the two distributions are not significantly different in terms of the Ising statistics measured (p > 0.05 for all three orders).
The principal difference between these two task distributions is the way in which they were gener- ated. The compositional task distribution was generated through the recursive application of simple, low-dimensional rules that generates mixture of three discrete structures, whereas the null task distribution was generated through a more complex Gibbs sampling procedure that is not explic- itly compositional and does not utilize explicit simple, low-dimensional rules. Although it is true that some boards within the null task distribution may be consistent with simple compositional grammar, the distribution as a whole was not generated through compositional grammar.
3 EXPERIMENTS
We analyze and compare the performance of standard meta-leamers and human learning on our tile- revealing task. We test them on boards that are sampled from the generative grammar and contain explicit compositional structure, as well as boards that are matched for statistical complexity. but are sampled from a null distribution that was constructed without using explicit compositional structure. Comparing performance across these two task distributions allows us to pinpoint the role of simple forms of structure as distinct from statistical patterns that arise as a downstream consequence of compositional rules based on such structure.
3.1 METHODS
Meta- Meta-Reiiforcemmnt Learning Agent Following previous work in meta-reinforcement learning Wang et al. 2016 Duan et al., 2016) we use LSTM meta-learner that takes the full board as input, passes it through 2 fully connected layers (49 units each) and feeds that, along with the previous action and reward, to 20 L.TM units. It trained with linear learning rate schedule and 0.9 discount. The reward function was: +1 for revealing red tiles, -1 for blue tiles, +10 for the last red tile, and -2 for choosing an already revealed tile. The agent was trained using Advantage Actor Critic (A2C) (Stable baselines package Hill al. 2018). The agent was trained for 10 episodes. We performed a hyperparamater sweep (value function loss coefficient, entropy loss coefficient, learning rate) using held-out validation set for evaluation (see Appendix). The selected model's performance was evaluated on held-out test grids. We trained different agents in the same way on the compositional and null task distributions, with separate hyperparameter sweeps for each.
4

---

Published as a conference paper at ICLR 2021
A
B
C
Blue Tils Revealed
Performance vs Time Conelation
Tiee Reevaled (Nuil Boardg)
Compositional
Null
Compositional
Null
Compesttiona- passing
Not Composttiona passing
Figure 3: Human performance. (A) Humans perform better (i.e. less blue tiles) in the composi- tional vs null task distribution (p=c0.0001). (B) Human performance improves over the course of the experiment (indicated by negative correlation between trial number and number of blue tiles revealed), with significantly greater improvement for compositional distribution (p=0.0006). (C) Some null distribution boards can pass compositional humans perform significantly better on these than on other boards in the null distribution (p<c0.0001). Agents (not shown in plot) do not do significantly differently on these boards (p>~0.05).
Human Experiment We crowdsourced human performance on our task using Prolific (www.proling www.prolific.co) for a compensation of $1.50. Participants were shown the 7 x 7 grid on their web browser and used mouse-clicks to reveal tiles. Each participant was randomly assigned to compositional or null task distribution, 25 participants in each. Each participant was evaluated on the same test set of grids used to evaluate the models (24 grids from their assigned task distribution in randomized order). Note that key difference between the human participants and model agents was that the humans did not receive training on either task distribution. While we are interested in examining whether agents can meta-learn abstract structures (by training compositional task distributions), we assume that humans already have this ability from pre-experimental experience. Since participants had to reveal all red tiles move on the next grid, they were implicitly incen- tivized to be efficient (clicking as few blue tiles as possible) in order to finish the task quickly. We found that this was adequate to get good performance. reward structure similar to that given to agents was displayed as the number of points accrued, but did not translate to monetary reward.
Evaluation Unless specified otherwise, performance evaluated as the number blue tiles re- vealed before all red tiles revealed (lower is better). All error bars 95% non-parametric bootstrap confidence intervals calculated across agents participants. Non-overlapping confidence intervals will have significant difference, but also include non parametrraee bootstrapped p-values for differences across different samples (e.g. human vs agent).
3.2 RESULTS
In this section, we first describe human behavior on this novel task. We see that humans perform better on the compositional distribution, without extensive training and even while directly control- ling for statistical complexity. We then compare human performance with that of a meta-learning agent-which has had extensive training this task, and therefore has had the chance to leam the structure relevant to this task distribution. We find significant qualitative and quantitative differences in behavior, and examine the role of meta-inductive bias i.e. what kinds of cross-task structure do meta-learners prefer to represent? In particular, we consider compositional and spatial structure. Finally, demonstrate the effect of an architectural change (adding convolutions) the meta- learner that makes it easier for it to discover spatial structure. We demonstrate that, while this helps agent performance overall, it further highlights the divergence between human and agent behavior in learning the compositional rule- based structure in our task distributions.
Human performance: We found that participants perform better on the compositional task dis- tribution than the null task distribution (see Fig. 31). Despite not having been trained on this task beforehand, human participants do fairly well on this task from the outset, suggesting that humans might have some of the relevant inductive biases from pre-experimental experience. To test if there learning within the experiment itself, we correlated trial number with the number of blue tiles revealed (Fig. 33), and found improvement across both conditions but significantly greater im-
5

---

Published as a conference paper at ICLR 2021
Step
2
3
Original Board
Human Moves
Agent
Propartion of perticppantslaeents
Moves
Figure 4: Human and agent policies the task. Red/blue indicate already revealed tiles while grayscale indicate what proportion of humans or agents (conditioned on the moves so far) revealed that tile in the next step. In this chain example, once humans figure out that the board a chain struc- tural form (step 5), they exhibit perfect performance by choosing tiles along the chain's 's production direction, while agents still choose other blue tiles.
provement for the compositional distribution. Finally, investigate performance the null task distribution more closely. There is some overlap between the null and compositional distributions, because some of the generated null boards could have been generated by the exact production rules of the generative grammar for the compositional task distribution. We split the null test set by whether or the board is compositionallppsee ppasinng and compare human performance across these. To do this, we generated the set of all possible compositional boards on 7 x 7 grid and labeled any happened be part of this set. We find
null task distribution board compositional ppsssing that humans do significantly better on boards that could have have been generated by the composi- tional production rules (Fig. 3D. This further suggests recognition and use of low-dimensional rules that align more closely with the compositional distribution than the null distribution.
Comparing human and agent performance: First, note that the meta-leamers perform rel- atively well on this task (Fig. 5). indicating that they have learned some generalizable information from the distribution of tasks. Since the test has held-out boards of compositional grammar, this might be taken as evidence that the agents discovered the compositional structure used to generate the boards. Here, we attempt decouple this possibily that is, that agents learn infer and use the simple, low-dimensional compositional rules as humans appear to do from the possibity that agents learn statistical patterns that are a consequence of the use of compositional rule.
We start with an example, involving the chain structure, that highlights the difference between hu- man and agent policies on this task (Fig. 4.. this example, once humans figure out that the board is a chain structural form, they never deviate from the chain's production direction while agents do. This suggests that humans are learning the simple generative rule of the chain form and using this rule determine their actions, while the agent is using statistical patterns associated with the chain rule rather than the rule itself.
We now consider various ways quantify this difference. First, we see that humans do better overall on both the compositional and null distributions (Fig. 5 p<00.001 for both task distributions). This despite, unlike the agents, having direct experience with this task. This suggests that humans have useful inductive biases from pre-experimental experience that are valuable in this task (Dubey et al. (2088); for example, the tendency recognize and utilize low-dimensional, composable rules, and the tendency to look for spatial patterns. We discuss the role of these inductive biases in the following sections. The meta-learner has had extensive experience with each task distribution, and had the chance to discover the structure / inductive biases relevant for this task. The differences in performance indicate that standard meta-learner differ from humans in the kinds of structure / inductive biases they learn (i.e. in their meta-ine mnea-inductivve biases).
Bias toward simple discrete rules First, we note that humans perform better the compositional versus the null distribution (Fig. 5b), whereas the agent does better on the null task distribution than on the compositional tasks. This reflects notable difference between their performance. We hy- pothesized that humans perform well on the compositional task distribution by first inferring what kind of structure is relevant for the current task, and then following the production rules for that

---

Published as a conference paper at ICLR 2021
A
B
C
Compesitional Task Distribution
Nuil Distrib istribbtiion
Blue Tiles Revealed
Human Agent
Red Tile Rate
Red Tile Hit Rate
Human
0.
Agent
Compositional
Null
First Half
Second Half
First Half
Second Half
Figure 5: Comparing human and agent performance (A) Humans do better the compositional task than the null (p<0..001), while agents do better at null (p=c0.0001). (B) Humans have a higher success rate revealing red tiles in the second half of a trial for the compositional task (p<0..001), agents do not. Transparent line represents individual human/agent average over trials, thick lines represent average over humans/agents. (C) Humans do not improve their success rates during a trial in the null task while agents do (p=0.0014).
structure. Since such structure was not used to create the null distribution, the act of learning, infer- ring, and using this structure is not as helpful in the null task distribution. Further, we hypothesized that the agents learn statistical patterns instead.
Fig. supports this intuition but here we look to quantify it further. If system represents a set of discrete classes of structures corresponding our compositional rules, we would expect success rate (rate of choosing red tiles) to be low in the beginning of a trial while figuring out what the structure underlying this trial is. Conversely, we would expect a higher success rate towards the end while following inferred production rules to reveal red tiles. To test this hypothesis, we split human and agent behavior in each trial into the first and last half, and examine success rate in each Fig. ( and c). For the compositional distribution, we find that humans have a higher success rate in the second half, providing support for our hypothesis. In contrast, we find that agent success rate does not increase over a trial, and in fact decreases. We also find that humans do not show increasing success rate in the null task distribution while agents do, providing further evidence for our hypothesis.
Bias toward spatial proximity. We note that humans outperform the agent even in the null task distribution, despite extensive training for the agent. One possibility is that good human performance in the null task is explained by their performance task distribution (Fig- (3).. However, another possibility is that humans come to the task with strong inductive biases about spatial proximity[] [ While the starting tile for the grammar can be randomly adjacencies. A system that has a bias toward local spatial structure might therefore perform better the task.
the compositional-pptie passing examples in the null
chosen, the production rules operate over
We test this possibility by comparing performance to heuristic that uses only local spatial informa- tion. This heuristic selects uniformly from the (unrevealed) nearest neighbors of randomly selected red tile. We evaluated this heuristic 1,000 times on each test board and formed a z-score statistic by subtracting the mean heuristic performance from the human' ssageen's 's performance for each board divided by the standard deviation of the heuristic's performance. We find that humans do better than the neighbor heuristic (Fig(b)) while the agent does not. This indicates that humans' inductive bias for spatial proximity may partially explain the differences in performance across humans and agents.
We can give a neural network a bias toward spatial proximity using convolutions (LeCun et al.) 1989). To test if this helps the agent, we replaced the agent's first fully connected layer with a con- volutional layer. We find that this agent outperforms humans on the null task distribution 6p). We also find that it outperforms the spatial heuristic described above GG) Note that this strictly reduces the expressivity (i.c. the number of parameters) of the model, and any improvements are due the right learner). However, humans still perform better than the agent in the compositional task distribution. Crucially, this means that which distribution easier is different for the human and the agent hu- mans find the compositional tasks easier than null, while the agent finds the null tasks easier than the
inductive bias (i.e. the right architectural inductive bias given the meta-
Spatial structure is shared by both distributions (Fig. 2 and can't 't explain why humans are better at compo- sitional tasks while agents are better at null. However, here investigate whether it can explain why humans perform better overall.
7

---

Published as a conference paper at ICLR 2021
A
B
C
Heuristic 2-scor
2-score
0.5
Humman
Blue Tiles Revealed
Convoltton Agent
Heurstic2scome
-0.55
0.0
0.70
0.3
1.0
-.5
-1.40
Corvolution
Human
Compositional
Null
Compositional
Null
Compositional
Null
Figure 6: Role of spatial structure in task performance. (A) Humans outperform the neighbor heuristic (negative z-score), the agent performs worse. (B) Humans outperform the convolution agent the compositional distribution (p<c0.0001) and the convolution agent outperforms humans in the null distribution (p<00.0001). (C) The convolution agent outperforms neighbor heuristic.
compositional. This result exhibits double dissociation between learning simple, abstract structure and statistical learning. It shows that the gap between humans and agents on the compositional task is not due to artificial meta-learners being overall worse learners than humans the convolutional meta-learner actually outperforms humans on the null distribution of equal statistical complexity. This provides further evidence that the inductive bias toward and representing of simple abstract structure is what may give humans a competitive advantage over agent on the compositional task distributions, and that meta-learnen do not learn it despite access compositional training distributions.
4 DISCUSSION
The ability to recognize structure in environments, well learn and utilize this structure, is a central tenet of human intelligence (Lake et al.) (017). One example of this kind of structure is compositional grammars. These of simple, low- dimeesninional rules that can be recursively applied to produce arbitrary complexity and can be widely generalized outside the training distribution. An inductive bias toward structured representations could be of great value machine learning systems. Recent developments in meta- meeaa-eerming hold promise as approach to endowing systems with such useful inductive biases. this work, we make several methodological and scientific contributions to provide rigorous way to test for structured forms of reasoning using compositional grammars case study. We show that human behavior consistent with learning and utilizing low-dimensional compositional rules. We also show that standard meta-learning approaches, in sharp contrast to humans, struggle with discrete abstract structures and prefer statistical patterns.
Our first contribution is the development of compositionally structured, rule-based task distributions for meta-learning using explicit generative grammars (Kemp & Tenenbaum, 2008). Previous work on generating compositional datasets has focused on language. We argue that using explicit gener- ative grammars has the dual advantage of being generalizable to a variety of structures, as well as being easy embed in multiple domains relevant machine learning. In this work, embed this structure into a grid-based task. Grid-hased tasks are commonly studied in reinforcementlearning easy for humans perform on online platforms, and behavior on this task is easy visual- ize, analyze, and interpret. This provides fertile ground for direct comparisons between human and machine behavior, as we demonstrate in our experiments. Previous work meta-learning composi- tionality uses performance on compositional task distribution as an indicator for meta- meta-learning this structure (Lake) 2019). However, we show that possible for meta-leaming systems to perform well using statistical patterns instead.
Our second methodological contribution is to create distributions with comparable statistical com- plexity to the structured distribution, that do not directly use the rules used to form the structured distribution. This control distribution allows us to disentangle statistical pattern matching from structured reasoning (which, in this specific case, is rule-based compositionality) and highlights the difference between actually learning and utilizing simple abstract structures (e.g. low now-dimensional compositional rules) versus using the statistical patterns that may be downstream consequence of those structures. Our method closely approximates the global statistics that emerge from the compo- sitional rules, by using neural network learn the conditional distributions and generating Gibbs
8

---

Published as a conference paper at ICLR 2021
samples from these conditionals. This approach is similar to masked language modelling (Devlin ct al., 2018), and our findings- that this procedure generates statistically similar but not explicitly compositional distributions, that are in fact easter for downstream networks to learn than the true compositional distribution-ant also relevant to understanding the representations learned by these systems more broadly (Rogers al. 2020).
In our experiments, we first show that humans have a bias toward using the compositional rule- based structure, while directly controlling for statistical complexity. This generalizes findings in the space of function learning (Schulz et al., 2017) to grid-based reinforcement learning tasks. Further, we find that agents (recurrent network trained with model-free reinforcement learning. following Wang et al. 22166] Duan et al., 2016) find the non-compositional null distribution easier to learn than the compositional one. This in direct contrast with human behavior, indicating that agents do not learn the same strategies that humans use through meta-learning. A followup experiment with con- volutional agent directly dissociates the effectiveness of statistical learning from the inductive bias toward compositional rules, and highlights learning and use of these simple, low-diment lwwddimensional rules as the key difference between humans and agents in this task. In both sets of experiments, we find a double dissociation between humans and agents: humans find the compositional task easier than the null task, while the pattern swaps for the agent. This indicates a significant difference (orthogonal overall performance) between the kinds of strategies humans and agents use solve this task. Our results therefore indicate that learning abstract structure, such as explicit compositional rules, remains difficult for artificial meta-learners and that they prefer other statistical features when possible. In other words, they do not have meta-inductive bias toward learning low-dimensional compositional rules.
Although the architecture we investigate here does not successfully meta-learn the ability to recog- nize and use abstract compositional rules, our point is not that this inductive bias cannot be meta- leamed. Rather, it is that every meta-learning architecture has its own meta-inductive bias, and we show a specific case in which standard and widely-used architecture's meta-inductive bias leads to encoding statistical features rather than the low-dimensional compositional rules used to generate the task distribution. When setting out to meta-learn a structured representation using a meta-lcarning system, it is important to consider the meta-inductive bias of that system in addition to engineering its task distribution. Graph neural networks (Battaglia et al. 2018), neurosymbolic approaches (Ellis et al. )020). as well as attention mechanisms (Mnih et al. 2014), permit abstraction by (implicitly or explicitly) decomposing the input into parts. Using these in meta-learning architectures might favor structured representations and reasoning.
Although we encourage exploring the space of architectural changes to give meta-learning agents better chance to learn such structured forms reasoning. it may also be true that simpler chitectures can acquire relevant inductive biases if given a rich enough high-dimensional training environment that rivals the environment(s) in which the species has evolved and individuals learn (Hill et al. 2019). However, the amount of data required to acquire these structured forms of rea- soning in "vanilla" architectures may be prohibitively large and largely infeasible. Further, even as training within these extraordinarily large environments becomes more feasible, the biases of the correspondingly large networks being used may continue to affect the ease at which they can learn structured representations. Therefore, it is still worthwhile to investigate the role of architectural modifications on the ability to meta-learn structured representations in smaller environments, such as the ones we present here, so that one day we may transfer those insights to training on larger, more naturalistic environments. An exciting direction for future work is to examine a range of approaches to learning structured representations with the tools set forth in this paper, and using the resulting insights to move toward closing the gap between human and machine intelligence.
5 ACKNOWLEDGEMENTS
We thank Erin Grant for providing helpful comments on the initial version of the manuscript. S.K. is supported by NIH T32MH065214. This work was supported by the DARPA L2M Program and the John Templeton Foundation. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the John Templeton Foundation.

---

Published as a conference paper at ICLR 2021
REFERENCES
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sonzaez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv: 01261, 2018.
Matthew Botvinick, Sam Ritter, Jane X Wang. Zeb Kurth-Nelson, Charles Blundell, and Demis Hassabis. Reinforcement learning, fast and slow. Trends in cognitive sciences, 23(5):488444 422, 2019.
Jeff Clune. Ai-gas:
Ai-generating algorithms, alternate paradigm for producing general artificial intelligence. arXiv preprini arXiv: 1905. 10985, 2019.
Ishita Dasgupta, Demi Guo, Andreas Stuhlmüller, Samuel J Gershman, and Noah Goodman. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv: /tt.::/80..0444 04302, 2018.
Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and Zeb Kurth-Nelson. Causal reasoning from meta-reinforcemmnt learning. arXiv preprint arXiv: 1901.08/22, 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04880 2018.
Yan Duan, John Schulman, Xi Chen, Peter L. Bartlett, llya Sutskever, and Pieter Abbeel. RI2: Fast reinforcement learning via slow reinforcement learning, arXiv preprint arXiv:1611.02777 2016.
Rachit Dubey, Pulkit Agrawal, Deepak Pathak, Thomas L Griffiths, and Alexei A Efros. Investigat- ing human priors for playing video games. arXiv preprint arXiv: /802. /02 17, 2018.
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sable-Meyer, Luc Cary, Lucas Morales, Luke Hewitt, Armando Solar-Lezama and Joshua B Tenenbaum. Dreamcoder: Growing gen- eralizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv preprint 0.X1v:2006.083811 2020.
Jerry A Fodor, Zenon W Pylyshyn, et al. Connectionism and cognitive architecture: A critical analysis. Cognition, 28(1-3)33-71 1988.
Erin Grant, Chelsea Finn, Sergey Levine, Trevor Darrell, and Thomas Griffiths. Recasting gradient- based meta-learning as hierarchical bayes. arXiv preprint 18Xiv:1801.0880 08930, 2018.
Thomas L Griffiths, Frederick Callaway, Michael B Chang, Erin Grant, Paul M Krueger, and Falk Lieder. Doing more with less: meta-reasoning and meta-learning in humans and machines. Cur- rent Opinion in Behavioral Sciences, 29:24-30, 2019.
Ashley Hill, Antonin Rafinn Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Rad- ford, John Schulman, Szymon Sidor, and Yuhuai Wu. Stable baselines. htttps://githube com/ Com/hill-a wwwhhiihlhhh st aaletaaleaaeee laslle--aaeeineee 2018.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L Mc- Clelland, and Adam Santoro. Environmental drivers of systematicity and generalization in situated agent. In International Conference on Learning Representations, 2019.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2901-2910, 2017.
Charles Kemp and Joshua B Tenenbaum. The discovery of structural form. Proceedings of the National Academy of Sciences, 105(31):10887-1055) 2008.
Brenden Lake and Marco Baroni. Generalization without systematicity: On the compositional skills of sequence-to-sequeeee recurrent networks. In International Conference on Machine Learning, pp. 2873-2882, PMLR, 2018
10

---

Published as a conference paper at ICLR 2021
Brenden M Lake. Compositional generalization through meta sowueence---ssquueeee learning. In Advances in Neural Information Processing Systems, pp. 9791-9801, 2019. 2019.
Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel Gershman. Building machines that learn and think like people. Behavioral and brain sciences, 40, 2017.
Yann LeCun, Bernhard Boser, John S Denker; Donnie Henderson, Richard Howard, Wayne Hub- bard, and Lawrence D Jackel. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541-551, 1989.
Shirley Mark, Rani Moran, Thomas Parr, Steve W Kennerley, and Timothy EJ Behrens. Transferring structural knowledge across cognitive maps in humans and models. Nature Communications, 11 (1):1-12, 2020.
R Thomas McCoy, Erin Grant, Paul Smolensky, Thomas L Griffiths, and Tal Linzen. Universal linguistic inductive biases via meta-learning, arXiv preprint arXiv:2006. 16324, 2020.
Volodymyr Mnih, Nicolas Heess, Alex Graves, et Recurrent models of visual attention. In Advances in neural information processing systems, pp. 2204-2213 2014.
Neil C Rabinowitz, Frank Perbet, H Francis Song, Chiyuan Zhang, SM Eslami, and Matthew Botvinick. Machine theory of mind. arXiv preprint arXiv: 1802.07740, 2018.
Anna Rogers, Olga Kovaleva, and Anna Rumshisky, A primer in bertology: What we know about how bert works. arXiv preprint arXiv:2002. 12327, 2020.
Eric Schulz, Joshua B Tenenbaum, David Duvenaud, Maarten Speckenbrink, and Samuel Ger- shman. Compositional inductive biases in function learning. Cognitive psychology, 99:44-79, 2017.
Joshua B Tenenbaum, Charles Kemp, Thomas Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. science, 331(6022):1299-122) 2011.
Sebastian Thrun and Lorien Pratt. Learning to learn: Introduction and overview. Learning to learn, pp. 3-17. Springer, 1998.
Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763 2016.
ZD Zhang. Conjectures on the exact solution of three-dimensional (3d) simple orthorhombic ising lattices. Philosophical Magazine, 87(34))5309-5419 2007.
A APPENDIX
A.1 HYPERPARAMETER DETAILS FOR REINFORCEMENT LEARNING
We did hyperparameter search for the following: value function coefficient, entropy coefficient, and learning rate. In particular, we evaluated each set of hyperparameters on separate validation set, selected the model with the highest performing set, and re-trained the model to be evaluated on a previously-unsed until the last, final evaluation step. Searches were independently for both task distributions (compositional and null). The final selected hyperparameters for both task distribution were: value function wtt.fiiient=00000677....35
test set of boards. Note that the final test set is not seen by the model
coefficieet=-0000077.. learning rate=0.00235.
A.2 DESCRIPTION OF COMPOSITIONAL GRAMMAR
Here provide an intuitive description of all the compositional grammar rules showin in Fig.
Start Tile Each grammar begins with a start square somewhere on or after the 3rd column/3rd row of the 7x7 grid (so grammar cannot start with tile the first or second row/column of the grid).
11

---

Published as a conference paper at ICLR 2021
Probabilistic Production Each grammar rule corresponds to partieular abstract structure. These structures can vary in size based on how many times that grammar rule is applied. Whenever a grammar rule is applied, the grammar will terminate with probability p 0.5. If a grammar rule cannot be applied again (e.g. the current tiles are on the edge and the next production would go off the 7x7 board), then the grammar automatically terminates.
Chain Production On the first chain production, the next two red tiles after the start tile (sg, sy) will be either: (s,-1, after the first one will follow the direction of the first production (for example: if the first chain production places red tiles on 11,+188 1y); (8 +1, 1,)) the second would do (82 8y); (82 + 8y).
1. sy), 19, sy or s,,, (5z sy+1). Any subsequent chain productions
Tree Production On the first tree production, the next two red tiles will either be
(sx +
(55)),22 L2 = (8 Sy + 1y),,, l2 = (Sa, By - or 4 = (82 + 1, 8y), (y - or 4 = (8, - 1, (8a, Sy + 1). The production rule always builds in two orthogonal
4 (8x 1, 1,8y),12 directions. On subsequent tree productions, one of the two added red tiles from the previous pro- duction will be picked and two orthogonal directions will be picked for the next defining characteristic in the tree structure is the "lack of loops", which means there can never be 2x2 sub-square of all red tiles. Therefore, a currently red tile chosen for the center of production such that there does exist pair of tiles f1, t2 in orthogonal directions to such that making both d1, l2 red does not create 2x2 sub-square of red tiles.
red tiles. The
Loop Production On the first loop production, 2x2 red sub sub-square will form in of four directions by coloring three tiles surrounding the start square (f1 = (82 + 1, 8y), 12 = (s. + 1, 8y
1) (8z, Sy + 1) t (sz 1, (yy)), (Sa, Sy + 1) or t (82 + 1, sy), ) l2 = (s, + 8y 1),-1,,4 1),43
(sH
Sy 1). ts
(8x, Sy 1)
(Sn - 1, 1,8 l = (82 - 1, Sy + +),
(S Sy 1). The next production rule will form another 2x2 red square surrounding
adjacent tile
to the original 2x2 red square such that the new 2x2 red square only shares one edge with the old
2x2 red square (see Fig. aand for what this exactly looks like).
A.3 REWARD OF AGENTS OVER TRAINING EPISODES
Episode Reward
-20
-40
Compositional FC
-60
Null FC
Compositional
Null Conv
-80
0.0
0.2
0.4
0.6
0.8
1.0
Training Episode Number
1e6
Figure 7: Reward received by each of the four agents trained in this work the course of the million training episodes.
12

---

Published as a conference paper at ICLR 2021
A.4
ALL PERFORMANCE DIFFERENCES ACROSS HUMANS AND AGENTS FOR ALL CONDITIONS
Human Chain
Human Tree
20
Human Loop
Human Null
10
Agent Chain
Agent Tree
T-statistic
Agent Loop
Agent Null
-10
Conv Agent Chain
Conv Agent
-20
Conv Agent Loop
Conv Agent Null
Human Chain
Human Tree
Human Loop
Human Null
Agent Chain
Agent Tree
Agent Loop
Agent Null
Conv Agent Chain
Conv Agent Tree
Conv Agent Loop
Conv Agent Null
Figure 8: We show the differences in performance (mean number of blue tiles, so lower is better) across humans and agents for the chain, tree, loop compositional conditions and the null condition. Differences are shown as t-values from testing difference in means across different participants or different agent runs. Any non-statistically significant differences are 0 (shown as the color white). Note that negative t-value indicates better performance, since the metric is mean number of blue tiles revealed.
13


Reviews:
[{"review_id": "hzoMXkeRdDN", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work is an exploration of model behaviour upon meta-learning tasks with compositional structure. The authors discover that, unlike humans, machine learning models do not readily pick up on the underlying compositional generative structure of a set of tasks, and hence cannot match the performance of humans. Conversely, when the task is structured to leverage other statistical patterns, models do well. \n\nTaken as a whole, this is a nice piece of work. The presentation is well crafted, and I believe the experiments are well planned. There are many nice analyses and some welcome statistics, such as shown in Figure 3. The authors are commended for their work. I wish to lay out a few criticisms, and I'd like the authors to know that the points are all very easily fixable. \n\nThe authors design a set of structure-learning tasks using a generative grammar. The exact details of the grammar are not given, and the reader is to rely on rough intuitions based on the figures. I encourage the authors to spell out some more details of the methods.  \n\nThe authors argue that the non-compositional boards could not have been generated by the defined grammar, which seems fair, but they also argue that these boards are necessarily non-compositional. I am having a lot of trouble with this statement, because it is not entirely clear by what the authors mean by compositional, as it hasn't been clearly defined. This is a particular problem in the machine learning field as a whole, as it pertains to research on compositionality; rarely is the term defined in any rigorous sense, and from paper to paper there are seemingly different definitions. I encourage the authors to clearly explain what entails compositionality as they refer to it here, and to explain what makes their non-compositional boards non-compositional according to their definition. The \"non-compositional\" boards might not match the generative grammar as used in the compositional setting, but it does not entail that there does not exist a compositional grammar that can produce the non-compositional boards. In fact, due to the discrete, simplified nature of the game, it would seem almost certainly true that there exists *some* generative grammar that can produce the non-compositional boards seen here. It might not be a \"simple\", \"interesting\", or human interpretable one, but it would nonetheless be a grammar, and would be compositional. This fact makes it all the more important to define compositionality.\n\nIt seems to me that what is more precisely being illustrated is the ability for humans to perceive, and infer the implications of abstract, \"simple\" *structures*, and not compositional rules or grammars per se. Without fully defining compositionality and establishing the non-compositionality in the null setting, I'm afraid that the results are misstated and misrepresented. A note to the authors: I'm fully aware that I could be misunderstanding how the MLP+Gibbs sampling method here can entail non-compositionality, and am more than open to being corrected on this matter. I look forward to a discussion in the rebuttal. \n\nThe previous point leads me to a broader point about the background presented throughout. The authors include many broad, and quite bold statements regarding humans, and how they learn, especially in regards to their capacity for \"learning compositionality\". It is claimed that humans learn rapidly, with very few samples, which is contrasted with machine models that require an enormous amount of data. This is true in some superficial sense, but does not account for the the entire evolutionary trajectory that produced humans; indeed, humans at birth are not blank slates to the degree that randomly initialized neural networks are. It's also claimed that humans learn compositional representations. While this idea is certainly in vogue in some circles in cognitive science, it is certainly not widely agreed upon. I'm not even sure how such a strong statement can be proven. The citations given point to a cople computational modeling papers, which, in my opinion, insufficiently corroborate such a claim. Moreover, it's not even clear how many non-artificial pieces of data that humans deal with are even truly compositional (for a taste of the issues and controversy surrounding one particular example --- language --- see the following entry: https://plato.stanford.edu/entries/compositionality/). Please note that this is not at all to say the views presented in this paper are necessarily false. I merely suggest that the authors take another pass at their writing, and tame a few of the broader, bolder claims about the nature of human learning, because it's not clear that they are necessarily true, either. \n\nI believe the authors have missed out on some possible interpretations to their results. They claim that a meta-learned model cannot learn the compositional structure of the tasks, since meta-learning is insufficient to establish the inductive biases required for compositional understanding. However, the inductive bias of a model is determined by more than its parameters (which in this case, are established via meta-learning). The functions a model comes to learn are also dependent on the nature of the computations, manifest through the architecture, and other such things. The authors are well aware of this, since they include a condition wherein the model uses convolutions. But the fact that the convolutional model does better entails that there might exist further architectural variants that do even better than it, and potentially, better than humans. If this were the case, then we'd no longer be able to claim that meta-learning cannot establish the proper inductive bias for compositional understanding. Therefore, the existence of a gradient in model performances warrants more careful wording in regards to the claims; we cannot so broadly categorize meta-learning as insufficient for establishing the right inductive biases for compositional understading without caveating according to the other sources of inductive bias. One piece of proof that the humans understand the compositional structure of the task is that they improve over the course of the task. But could this not simply reflect the fact that humans have a better capacity to improve behaviour over short time periods? In other words, how do we know that the problem with the models doesn't have to do with, say, working memory, rather than compositional understanding per se?\n\nAltogether, this is a well put together paper. There is a lot of interesting work here, and the authors have done well to explore various facets of the setup. My main criticisms have to do with the way the work is pitched, and the way the results are interpreted. It is very much presented with a veneer that speaks to a very particular crowd in cognitive-science inspired machine learning community. But since the views in this community are not necessarily broadly shared, many of the statements, interpretations, and claims come across as quite strong and not fully corroborated. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "BEwnKw1hCtz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper sets out to determine something about the form of the bias acquired by a standard meta-learning algorithm, and compare the form of that bias to the inherent bias that humans have.    The authors point out, rightly, that meta-learning algorithms have meta-biases and it is important to understand these, from both the scientific and engineering perspectives.  It is well written and raises good questions.\n\nThere is a cleverly constructed test domain and a set of well-executed computer and human experiments (I think---I don't really know about how to construct a human experiment.)\n\nUnfortunately, I can't end up agreeing or disagreeing with the claims made in the paper, or really understands how well they are supported by evidence, because I find that they use terms that don't seem to be sufficiently technically well defined.\n\nFor example:\n- what exactly is compositional structure?  \n- what is statistical structure?\n- what is your measure of task complexity?\n\nHow can we tell if what the agent learns is compositional?  Is that an externally measurable property of the agent's behavior and the way it generalizes to new environments?  Or is it a property of the internal representation?   (It is common to have an intuition that \"compositional\" also implies \"compact\" or \"low complexity\" in some sense.)\n\nIt feels like generalization be a way to get more clearly at the presence of a compositional representation:  could you train on small grids and have the learned agent generalize to big ones?  It seems like if a fixed-size representation can generalize to very large instances, then that is more clear evidence of compositionality (but then I'm thinking of compositionality as a property of a representation, not of externally-measurable behavior.)\n\nI also feel that I don't quite understand the meta-learning training regime.  What exactly constituted a \"task\" from the meta-learning perspective?   Is it a single board?  If so, then the meta-learning problem is to learn the task distribution, in some sense.   I was expecting something more \"meta\":  that is, to test whether the system is actually meta-learning the *idea* of compositionality, it seems like set-up would be that a task corresponds to a particular grammar with multiple boards drawn from the distribution induced by the grammar;  then we'd know that it had meta-learned compositionality if it could learn *new grammars* quickly.\n\nSmaller points\n- I didn't completely understand the production rule (nor the examples) for the loop structure.\n- It would help me understand the task set better if there were a slightly more in-depth description of the chains, trees, and loops and described how the grammar generates the compositional tasks in figure 2. Is it not the case that every connected configuration of red tiles could be described as a tree?\n- Rather than showing just one number for the final performance, It would be helpful to show learning curves for the RL algorithms so the reader can assess the stability, convergence, etc. Similarly, learning curves for humans would be interesting, but less important since I assume they just look flat.\n- \"is develop\"", "extra_scores": {}, "extra_text": {}}, {"review_id": "3wn06nwtZX-", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summarize what the paper claims to contribute. Be positive and generous.*\n\nThis paper offers an interesting comparison between humans and a meta-learning algorithm [1,2] learning to uncover structured patterns on a 7x7 grid. The patterns are either generated using simple “compositional rules” (lines, loops or trees) or sampled from a distribution that has almost identical 0th, 1st and 2nd order statistics as the ones generated through the “compositional rules”. The board is initially covered, and the agent (either a human or a RL meta-learning algorithm) is tasked with selecting one by one which tiles are to be uncovered. The aim is to guess which tiles are covered by the pattern (red tiles) and avoid the background (blue tiles), thus uncovering the red tiles as accurately as possible. The authors go on to show supporting evidence that most likely, humans are using a compositional inductive bias when trying to uncover the hidden pattern, whereas the meta-learning algorithms do not. This conclusion is arrived at using the relative performance of both humans and the algorithm on the cases where the pattern was generated using the two different schemes. They then go on to conclude that there is a strong difference between the strategy learned by the algorithm, which seems to use mostly the statistics, compared to humans that seem to make use of a compositional inductive bias.\n\n\n[1] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. Rl^2: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.02779, 2016.\n[2] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv:1611.05763, 2016.\n\n\n*List strong and weak points of the paper. Be as comprehensive as possible.*\n\n**Strong points**\n\n- The paper is very clear and very well written. I like the simplicity of the experiment and the approach taken by the authors in trying to uncover whether humans and a particular algorithm follow similar strategies when solving tasks.\n- I see no reason why this approach should not be re-used by other researchers, and I hope that it will inspire others to go to similar lengths when analysing the operation of their algorithms. Specifically, I consider this a strong point of the paper as I believe it offers an obvious and accessible set of future work.\n\n**Weak points**\n\n- It is not completely clear how the statistically similar patterns are constructed. Is it possible to get patterns that are identical to the compositionally generated ones? If so, have you checked and removed these? Please elaborate a bit more on this aspect, show more examples of the non-compositional patterns, and describe how you decided on the Compositional-passing/not compositional passing split in Figure 3C.\n- One somewhat deeper weakness of this work is that one can claim that there is no inherent notion of compositionality in the patterns of the data per-se, but rather in the manner through which they were generated. This is perhaps an underlying reason why the meta-learning algorithm never picks up the compositionality inductive bias exhibited by humans. My point does not mean to invalidate the findings and conclusion, but rather to emphasise that perhaps we shouldn’t be looking at this type of experiments, as they are “doomed to fail”. Of course this might be obvious in hindsight, but I would be interested to read what the authors think about this.\n\n\n*Clearly state your recommendation (accept or reject) with one or two key reasons for this choice. Provide supporting arguments for your recommendation.*\n\nI recommend accepting the paper, mainly for the two strong points earlier. I believe what the length at which authors have gone to for understanding what an algorithm does (or what it doesn’t do) should be an example for our field, which often lacks imagination when it comes to analysing proposed models. I do think that a condition for accepting is at least ensuring that the details of all the generated samples (compositional or not) are included in the paper. How many (unique) of each were generated/used? How did you decide the compositional-passing/not compositional passing split. Did you check whether any non-compositional pattern matched the compositional one? If so, how and did you reject that sample?\n\n*Provide additional feedback with the aim to improve the paper. Make it clear that these points are here to help, and not necessarily part of your decision assessment.*\n\n- In the intro: “Second, humans represent this learned information compositionally”. In my opinion this is a very strong statement about the nature of the representation in the human brain and it’s best avoided unless there’s neurophysiological evidence. Consider rephrasing to a softer version with a reference.\n- 4th paragraph in the introduction: “Our methodological contribution in this work is *to* develop novel tasks.”\n- Last paragraph of introduction: “Since a large swath of real-world tasks contain compositional structure..” Which ones? Please include some examples. Also, is it that they contain compositional structure (i.e. in the way they are generated) or that the compositional inductive bias that humans exhibit can more efficiently solve them?\n- Second sentence of 3.2 Results: “We demonstrate that humans have a clear bias toward compositional distributions, without extensive training and even while directly controlling for statistical complexity.” Couldn’t we claim that the patterns generated by the compositional distributions are easier/simpler? They are after all only 3 different rules. Wouldn’t that be a good enough explanation of why humans solve those much more easily than the statistically matched ones? You did touch upon the issue of spatial proximity later on. Can you touch upon this point too? I think it’s a possible (and perhaps simpler) alternative explanation of why humans would be better at them.\n- Appendix A.1. The hyperparameters are reported in an unnecessary accuracy - I would assume that results are not sensitive to that many significant figures. Please consider 2 or 3 s.f. in scientific notation for simplicity.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "0HN79lklu8r", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post revision update\n-------------------------------\n\nThe authors have been very helpful and addressed many of my concerns, and I think the revised paper is a substantial improvement. I have rated the paper as a 7, although I do have some lingering concerns.\n\nMost crucially, it's still not clear to me that the compositional rules the authors highlight are the correct way to characterize the differences in patterns of behavior, since, for example, both models significantly outperform humans at the tree rule. However, the authors do point out that humans perform better at these tasks than the null distribution. Still, I worry that the authors are focusing on the wrong dimension along which the compositional and null task distributions differ. However, I think the fact that the authors followed the suggestion to include the results in the appendix is helpful in this regard, at least future researchers will be able to see the full pattern of results to draw their own conclusions.\n\nOriginal Review\n---------------------------\n\nThis paper proposes to explore whether meta-learning approaches can exploit a compositional structure in their tasks to generalize, and compares this ability to humans. To do so, the paper introduces a grid dataset consisting of generative grammars for generating compositional grids, as well as a null task distribution which is non-compositional but matches on certain low-order statistics. These tasks are interesting and new. They have humans and agents perform a task to reveal rewarding squares on the grid, and compare to agents that meta-learn this task. Human subjects perform better at the compositional distribution, whereas models perform better at the null distribution. They conclude that \"compositional structure remains difficult for these systems and that they prefer other statistical features\" and that this \"highlights the importance of endowing artificial systems with this bias.\" While the topic is timely, and the tasks are interesting, there are a number of limitations to the model and training which I think seriously limit the conclusions. I think that the task is really only compositional for a model which is able to fixate on different locations on the grid. Thus, I recommend rejecting for now, although I think a revision with a more sophisticated model and more thorough discussion could be a valuable contribution.\n\n\nStrengths:\n\n* Interactive tasks for humans and models are a great improvement over prior toy datasets on compositionality, e.g. SCAN.  \n\n* I think the tasks are very interesting, and offer directions that aren't really address by prior compositional generalization datasets that mostly rely on composition of words, or on composition of visual properties like color and shape.\n\n* It's great to see actual comparisons to human performance, and careful thinking about how to evaluate compositional generalization vs. other types (though this could be developed further, see below).\n\n\nAreas for improvement: \n\n* Is generalization equally good for humans under each rule type? If not, this might affect the conclusions. For example, perhaps humans would be very good at inferring chains, but not the more complex structures (like trees). If so, it could be that \"compositionality\" per se is not the construct underlying their performance, but rather \"chains\" or some other, simpler construct. Because performance is not presented broken down by structure type, it is difficult to determine whether some pattern like this could explain the results. Thus, it is difficult to conclude that compositionality is the factor underlying the results. \n\n* There are a number of features of the agent that confound the comparison with the humans. These limit the ability to draw a strong conclusion about e.g. \"the importance of endowing artificial systems with [a compositional] bias.\" \n    * Why are the main comparisons run using the non-convolutional model, when the convolutional one is clearly more closely matched to humans (as the discussion acknowledges)? It seems like most (although not all) the difference in performance is due to spatial bias, rather than compositionality. \n    * Indeed, this spatial bias is partially addressed on the input (by the convolutional experiments), but it is *never* addressed on the output. Humans know that there is a spatial structure to the tiles they are clicking on, the agent can access this information only implicitly.\n    * To address this, one could build a recurrent attention model (e.g. Mnih et al, 2014; Gregor et al, 2015) which can make visual saccades around the grid before deciding whether to reveal the square at the current point of fixation, or whether to fixate to another location. This would likely match the human process better, since the humans are likely fixating their gaze on the locations they are considering, rather than fixating in the center of the grid without moving their eyes. It's also motivated by the observation that agents generalize better if they receive ego-centric input rather than visual input fixed on the grid (c.f. Hill et al, 2020; Ye et al, 2020). This is an important issue to the claims at stake. For an agent which could fixate on each location it was considering, the compositional rules would be much more consistent than for an agent that perceives the whole grid from a fixed perspective. For a fixating agent, the compositional rules would also be much more consistent than the null distribution. In fact, I would suggest that it is *only from the perspective of a model which can fixate that this distribution can be considered compositional at all.* How can we tell that the difference between the humans and the model isn't due to the human ability to fixate, rather than some abstract bias toward compositionality?\n    * The paper may not be able to address all the ways in which the model's experience of the task is unlike humans, but then the there should be a *corresponding tempering of the conclusion that the comparison to humans says something specific about the difference between the model and humans.* That is, given the current experiments, the discussion of this paper should focus at least as much on the limitations of the present model as on general conclusions about failures of the model class and the need for additional inductive biases.\n\n* Furthermore, exploring compositionality in toy tasks can be misleading. Hill et al. (2020) show that compositional generalization is significantly improved in more realistic settings (for example an RL agent that executes actions over time achieves 100% compositional generalization on a task that a feed-forward classifier only achieves 80% generalization on). They argue that toy stimuli remove one of the most important elements for training deep models — the rich environments in which humans, also, are trained. Even if the input and output of the model and humans were better matched on this dataset, it may be misleading to conclude something as general as \"the importance of endowing artificial systems with this [compositional] bias\" without giving these models training on a distribution of stimuli and tasks that more closely match the rich variety which humans experience over development. Of course, it is not feasible in practice to do so (yet). But this limitation and its relevance to the conclusions should at least be acknowledged in the discussion. \n\n* The paper could use some more discussion of the distinction between statistical patterns and compositional rules. This seems like an important point, but it wasn't entirely clear to me. Compositional rules correspond to a certain statistical distribution over grids. The fact that up to 2nd order Ising statistics are matched does not mean that the distributions of outcomes are matched \"statistically,\" it merely means they match in certain low-order statistics. It's not clear if these are the right statistics by which to compare the distributions (especially for the non-convolutional model, which has no spatial awareness). The paper would be strengthened by justifying the choice of these statistics more carefully, and articulating a clear distinction between what counts as a \"statistical\" pattern vs. a rule. \n\n\n\n\n\nReferences \n---------\n\nGregor, Karol, et al. \"Draw: A recurrent neural network for image generation.\" arXiv preprint arXiv:1502.04623 (2015).\n\nHill, Felix, et al. \"Environmental drivers of systematicity and generalization in a situated agent.\" International Conference on Learning Representations, 2020.\n\nMnih, Volodymyr, Nicolas Heess, and Alex Graves. \"Recurrent models of visual attention.\" Advances in neural information processing systems. 2014.\n\nYe, Chang, et al. \"Rotation, Translation, and Cropping for Zero-Shot Generalization.\" arXiv preprint arXiv:2001.09908 (2020).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors present a study where they investigate whether meta-learning techniques leverage the underlying task distribution. To do so, the authors come up with two conditions, in the first they generate tasks using a grammar and in the second condition, which is the null condition essentially, the tasks have the same statistical properties as the compositional task but they are not derived from a simple grammar. The authors find that while humans are better in the compositional condition, models are better in the null condition.

All reviewers have been positive with this work, but some concerns were raised regarding clarity around the use of some terms, such as compositionality. The rebuttal period has been very productive and the reviewers have acknowledged the improvements on the manuscript. 

All in all, I think this is a good study to appear on ICLR and I believe researchers would benefit from the design of the study that will perhaps open new opportunities around careful evaluation of meta-learning agents.
Decision: Accept (Poster)