Using natural language and program abstractions to instill human inductive biases in machines
Authors: Sreejan Kumar
Carlos G Correa
Ishita Dasgupta
Raja Marjieh
Michael Hu
Robert D. Hawkins
Jonathan Cohen
Nathaniel Daw
Karthik R Narasimhan
Thomas L. Griffiths
Venue: NeurIPS 2022 Accept (2022)
Keywords: meta-learning
program induction
natural language
reinforcement learning
human intelligence
cognitive science

Abstract: Strong inductive biases give humans the ability to quickly learn to perform a variety of tasks. Although meta-learning is a method to endow neural networks with useful inductive biases, agents trained by meta-learning may sometimes acquire very different strategies from humans. We show that co-training these agents on predicting representations from natural language task descriptions and programs induced to generate such tasks guides them toward more human-like inductive biases. Human-generated language descriptions and program induction models that add new learned primitives both contain abstract concepts that can compress description length. Co-training on these representations result in more human-like behavior in downstream meta-reinforcement learning agents than less abstract controls (synthetic language descriptions, program induction without learned primitives), suggesting that the abstraction supported by these representations is key. 

Using natural language and program abstractions to instill human inductive biases in machines
Sreejan Kumar!, Carlos G. Correa', Ishita Dasgupta", Raja Marjieh', Michael Y. Hu',
Robert D. Hawkins 1.3 Nathaniel D. Daw Jonathan Cohen Karthik Narasimhan", and
Thomas L. Griffiths
Princeton Neuroscience Institute
DeepMinn
Department of Psychology, Princeton University Department of Computer Science, Princeton University
Abstract
Strong inductive biases give humans the ability to quickly learn to perform variety of tasks. Although meta-learning is a method to endow neural networks with useful inductive biases, agents trained by meta-learnnng may sometimes acquire very different strategies from humans. We show that co-training these agents predicting representations from natural language task descriptions and programs induced to generate such tasks guides them toward more human- like inductive biases. Human--eneratee language descriptions and program induction models that add new learned primitives both contain abstract concepts that can compress description length. Co-training these representations result in more human-like behavior in downstream meta-reinforcement learning agents than less abstract controls (synthetic language descriptions, program induction without learned primitives), suggesting that the abstraction supported by these representations key.
Introduction
Humans are able to rapidly perform a variety of tasks without extensive experience [T] This may be because of strong inductive biases towards abstract structured knowledge (e.g. hierarchies, compositionality) 2.3] that act as strong prior knowledge, enabling generalization to novel envi- ronments with little new data. These biases present one of the most salient differences between hu- mans and neural network-based learners 4 and may be one of the keys to building artificial agents with human-like generalization capabilities. For this reason, there has been interest among ma- chine learning researchers in identifying and instilling these inductive biases into neural network- based agents (5-7).
One emerging approach to implicitly bestowing inductive biases on neural networks meta- learning [8.9). In meta-learning paradigms, agent is trained not just on single task but on a distribution of tasks, with the aim of acquiring the underlying abstractions that these tasks have in common. However, since neural networks not easily interpretable, it can be difficult tell the resulting neural networks actually acquired this abstract knowledge, or whether they have instead learned statistical artifacts correlated with abstract rules [10].
What representations can give our artificial agents access to human inductive biases? Previous work in reinforcement learning has shown that neural networks can improve performance through auxiliary tasks [33.33) We build on this insight to construct auxiliary tasks that require reproduc-
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

synthetic language
target board
natural language
the reds are in: row 3 and calumn 1, 2 and column
4 down the side,
and column row 3 and column 2, row and column 1, row 0 and column row column row and column
H
2 next to it from the bottom,
3 next to that from the bottom, like uneven shape.
and column
programs with primitives
programs with library learning
Figure 1: Four repositories of human-like priors. For a stimulus space of 2D binary grids, we investigate two classes of repositories for human-like priors: linguistic descriptions (top row) and program abstractions (bottom row). For linguistic descriptions, we consider both synthetic language (upper-left), and numan-generated language (upper-right). For program abstractions, we implemented a simple DSL consisting of an agent that can move around on the grid while filling-in red tiles. We consider both primitive programs written in the base DSL (bottom-left) as well programs written with learned library of abstractions (bottom-right).
ing different kinds of representations, and examine what kinds of representations tend to produce agents with human-like inductive biases.
Human biases toward abstract knowledge might be linked to the ability to verbalize this knowledge through natural language [5]. Human language descriptions can therefore act as a repository of this prior knowledge. Recent work in machine learning has shown that neural network repre- sentations can be shaped and structured through natural language supervision for various tasks Abstract knowledge in humans has also been modeled by program induction 23]266 where a model directly infers a structured programmatic representation from data. These program- matic representations
16 22]
constructed from primitive atoms using symbolic rules and procedures. Recent work has leveraged neural networks to make this otherwise expensive and brittle process more scalable [neurosymbolic models; 29] In these approaches, abstract knowledge is explic- itly built into the artificial agent using pre-specified Domain Specific Language (DSL), which can be restrictive. In many situations, we would like artificial agents that are implicitly guided towards human- like abstract knowledge [30 rather than explicitly building it in, since it can be hard to determine what specific concepts to build into DSL for any arbitrary environment. Therefore, in this work, we focus on how to use representations from program induction to guide artificial agents that don't have these built in concepts.
In this work, we show that language and programs can be used as repositories of abstract prior knowledge/inductive biases of humans that one can co-train with elicit human-like biases in other related tasks. The differences in behavior elicited by the use of different inductive biases are subtle,
recent framework from Kumar et al. (10 build tasks that distinguish these subtle differences. The specific domain we use is a meta-reinforcemenn learning task where the agent sequentially reveals patterns on 2D binary grids. We find that guiding meta-reiinforeement learning agents with representations from language and programs not only increases performance on task distributions humans are adept at, but also decreases performance on control task distribu- tions where humans perform poorly. We also show correspondence between numan-generann language and programs synthesized with library learning. This indicates that humans use higher- order concepts to compress their descriptions in the same way that recent library learning ap- proaches in program induction add new concepts to the DSL to compress program length [24]. Finally, we show evidence that this process of abstraction through compression in these repre- sentations (either language or program) is a key driver of human-like behavior in the downstream meta-learning agent, when co-trained on these representations.

---

human distribution
control distribution
elicited using Gibbs Sampling with People (GSP)
elicited using neural network with matched statistics
Figure 2: Example grids from human-elicited priors (left) and machine-generated priors (right).
2 Dataset and task paradigm
Our goal is to examine differences in how human-like the behavior of an agent is, as elicited by dif- ferent co-training objectives. To do this, we first need a distribution that reflects human inductive bias (as well as a control which is closely matched but doesn't 't directly reflect this inductive bias) so that human-like behavior is distinguishable. We introduce these distributions in the following sections, as well as how we build an RL task based on these distributions.
2.1 Eliciting human priors using Gibbs sampling
We collected a dataset where we directly sample from humans' prior expectations on two- dimensional binary grids (we will refer to these as "boards"). These data contain the direct human prior Knowledge/inductive biases and therefore be used to study them. To do this, we used a technique called Gibbs Sampling with People [GSP; 34 see Fig. 2 that samples internal prior distributions by putting humans "in the loop" of Gibbs sampler. The stimulus space consisted of the space of 4 x 4 boards giving 16 stimulus dimensions. Each stimulus dimension (corresponding to tile on the board) had two possible values and determined the binary color of the tile, namely, red or white. Each GSP trial consisted of human participant predicting what color single masked square in the grid should be, conditional on the colors of all other squares on the grid. The specific instructions given were "What should be the underlying color of the covered greyed tile such that the board generated by a very simple rule?" (Fig. 2}. Once a decision was made the resulting stimulus was passed to a new participant, who repeated the task with a different masked tile. When sampling from this distribution, the probability of each board is based on how frequently it occurred during the GSP sampling process, which reflects its probability under human priors over the space of 2D grids. We selected the 500 most probable boards to collect language and program data. We will refer these the "GSP boards".
2.2 Constructing matched control distribution
The control distribution of boards matches the statistics of the GSP boards but not produced by human decisions (see Fig. 2) A fully connected neural network (3 layers, 16 units each) was trained to encode the conditional distributions of the GSP boards: a random tile masked out, and the network is trained to predict its value given the other tiles (similar to masked language models; (35)). These conditional distributions contain all the relevant statistical information about the boards. The network achieved accuracy of above 99% on this task.
This network then used to generate samples, using the same process that was used to generate samples from human priors via Gibbs sampling. A board in which each tile is randomly set to red or white with probability 0.5 is initialized, and this trained network is used to predict masked out tiles. Since the conditional model is trained on the GSP boards, this generates a set of boards with similar statistical properties to original GSP boards (example: number of red tiles the GSP boards (Mean=8.4, SD=2.26) do not significantly differ from the control boards (Mean=7.4, SD=2.01), p = 0.12), but also implicitly encode the priors of the conditional model (i.e. a trained neural network instead of humans) well.
3

---

state starting state
Human
Agent
0.0
white tiles revealed (2-scored)
-0.5-
-1.5
LSTM
-2.0
human distribution
control distribution
-2.
Figure 3: Meta-RL. Task and Architecture. (A) The goal of the agent is sequentially uncover all the red tiles to reveal picture on a 2D grid while uncovering as few white tiles as possible. single board one task and distribution over boards is task distribution. (B). Performance of humans and artificial agents on the human-elicited vs machine-generatte samples (see Fig 2). The performance metric based on the number of white tiles revealed, lower is better.
2.3 A search task for meta-reinforcement learning
Task. These distributions of boards can be used to construct new reinforcement learning tasks. We use the same task, performance measures, and agent architecture as Kumar et al. [10]. In this task, the agent starts with an almost fully masked out board (all greyed tiles except for one revealed red), and has select tiles reveal. The revealed tiles are red white depending on the tile colors of preselected underlying board. One board with a fixed configuration of red tiles defines single task. The agent's s goal is to reveal all the red tiles, while revealing as few white tiles as possible. The episode ends when the agent reveals all the red tiles. There is reward for each red tile revealed, and penalty for each white tile revealed. Agents are trained on set of boards and tested held out test boards. Doing well this task requires the agent to learn and represent distribution over boards its training distribution. The primary manipulation is whether a board is generated from human priors or from the control distribution (that is closely matched in terms of statistics, but generated from machine priors). Learners that do better on boards that were directly sampled from people's prior distribution on 2D grid stimuli can be said to have human-like inductive bias, and vice versa for machine-ae machine-generated boards (see Fig. 2
Evaluation metric. The specific metric we use to track performance is as follows. An agent that does well on the task will reveal all the red tiles while revealing as few white tiles as possible. We therefore measure performance by counting the number of white tiles revealed in the episode. Since different boards will have different number of tiles and thus have varying levels of dif- ficulty, we measure the performance relative to "nearest neighbor" heuristic. This heuristic ran- domly selects covered tiles adjacent to currently uncovered red tiles. We run this heuristic 1000 times on each board and the human/agent's number of white tiles revealed is z-scored according to this distribution. z-score of below means that the human/agent did better than the mean performance of the nearest neighbor heuristic. The specific value of the z-score reflects how many standard deviations the human/agent did better than the mean performance of the nearest neighbor heuristic. We determine significant differences in performance across different task distributions and models using non parameeaetriic bootstrap independent samples statistical test.
Baseline experiments. The agent architecture is LSTM meta-learner trained with reinforce- ment learning. This architecture is inspired by Wang et [36). Our LSTM meta-learner takes the full board as input, passes it through convolutional and fully connected layers and feeds that, along with the previous action d;-1 and reward rt-1, to 120 LSTM cells. The agent had 16 pos- sible actions corresponding choosing tile (on the 4 x 4 board) to reveal. The reward function was: +1 for revealing red tiles, for white tiles, +5 for the last red tile, and for choosing an already revealed tile. The agent was trained using Proximal Policy Optimization [PPO: 37 using the Stable Baselines package 38 for one million episodes.
We trained our agent on the distribution of the human- generated GSP samples (see Fig 2). We then evaluated the agent on a set auman-generated samples as well as machine-generated samples, both of which were held out of training. Despite the fact that we did not train the agent the generatd samples, the agent still performs significantly better on the control samples than the human samples (p 0.0001) (Fig. 3B), which is in stark contrast humans who perform

---

significantly better on the human- generated samples (p < 0.0001). The fact that the meta-RL agent exhibits the opposite pattern of performance across human and srechine-generaeed tasks (compared to humans) suggests that the agent has learned a different representation than that used by the humans, reflecting a different set of inductive biases than that of the humans. This is replication of the effect reported in Kumar et al. [O] using the same architectures and task paradigm, but here the agent is trained with PPO instead of A2C. Having established that humans and agents seem to use different representations to perform these tasks presumably driven by differences in inductive bias- we now consider how to provide the agent with more human-like inductive bias(es).
3 Instilling human inductive biases with language and programs
3.1 Natural language representations
We collected Natuurl-language descriptions of 500 of the most probable GSP boards from a naive group of participants. Each participant wrote descriptions for 25 unique boards in response to the following prompt: "Your goal is to describe this pattern of red squares in words. Be detailed as possible. Someone should be able to reproduce the entire board given your description. You may be rewarded based how detailed your description is." Participants were randomly assigned boards such that each board had 9-12 descriptions from different participants. As a control for the language data, we also generated synthetic text descriptions. Synthetic descriptions used a template that said "The reds are in: row Xi and column YI...." " for every red tile location (Xa, Yn). We permuted the order in which the red tile locations were verbalized to generate multiple descriptions per board and mimic the multiple human descriptions per board for the human- generated descrip- tions. To embed each description (both numan-generater and synthetic) into a vector space with semantic meaning, we obtained the RoBERTa 39 sentence embedding for each description using the Sentence' Transformer package (https: / /wwww sbert. abeert.et/.,.. net/,, based on Reimers and Gurevych 40). The Sentence' Trannnerrrrrrrrrrrr.rr model maps text into a 768 dimensional dense semantically meaningful vector space (i.e. text samples that are semantically similar are close to each other in vector space). Reimers and Gurevych [40 do this by using a contrastive objective on a dataset of semantically-paire text where embeddings from the same pair are pushed closer and embeddings from different pairs are pushed further apart.
3.2 Program representations
We used DreamCoder 24 to parse the boards from the GSP experiment into programs that can generate them. Given flmminn-specffi language (DSL) and way to score program comprised of DSL functions, DreamCoder enumerates and scores programs during its wake phase. This process is accelerated in two ways during the sleep phase: first, by learning neural-networfs based recognition model that guides searching the space of programs during wake, and second by augmenting the DSL with subprograms abstracted from repeated expressions in existing programs using library learning. DreamCoder alternates between wake and sleep phases for multiple cycles.
We now describe our domain and DSL. In this domain, the goal is to produce the target board on a grid by controlling a pointer that marks locations with a pen (see Fig. []. The pointer state consists of a location on the grid, an orientation in one of the four cardinal directions, and whether the pointer's 's pen is down (i.e. whether it will mark at subsequent future locations). The DSL is comprised of the following primitives; move which moves the pointer forward from the current location based on orientation; left, right which turn the pointer, changing its orientation; pen-up. pen-down which change the state of the pointer's 's pen; and fork which stores the pointer state, executes program, and then restores the agent state (except that grid locations marked in the subroutine are not reset).
The score <pll) of a program p when executed from an initial location is the sum of several penalties that encourage the program to replicate the target board faithfully in an efficient manner. First, there a penalty of - 10 for each marked location that in the target board and not in the generated program's 's board, and penalty of oo for each location that is in the generated pro-
5

---

A
Language
B
Programs
C
Text Description Length
Program Description Length
Human Generated Symthiii
Primittes
IAn
'This very tricky perhaps hammer'
Mon Absract
'The left most column is fully red. The two middle columns only have their top squares red'
D
Human Language vs Library Learning Programs R=0.18, pç0.001
Human Language vs Primitive-Only Programs R=0.02, p
4 by grid of squares num-
1 it
bered 1 to 4 top left to top right and 5 to top left to bottom left red squares at locations 1.5 to inclusive; 2,5 and 2,6;
3,5 and 3,6'
beeteee trigh
Figure 4: Properties of Language and Program Data. (A) Example board with corresponding human-generated language descriptions. Some people write more abstract descriptions than others. (B) Examples showcasing compression of programs with learned library concepts. We show the library-learning program (top grids) and non-library learning programs (bottom grids). For each program, we show the execution path (right, star indicates start of path) and the program stack (left). Black indicates main stack whereas blue indicates use of learned library function. Non- library learning programs do not use learned library functions and thus function only on the base stack. (C) Human-generated language descriptions typically have lower description length than the synthetic descriptions. Analogously, programs with library learning are on average shorter than programs with just the base DSL. (D). Boards that elicit shorter more abstract descriptions from humans also elicit shorter programs, indicating correspondence between the abstractions human in language and those from with library learning.
gram's board and not in the target board I The program is therefore strongly encouraged to mark all the tiles that marked in the target, and strongly discouraged from marking additional tiles that are not in the target. Second, there are penalties that encourage the program to replicate the target board efficiently. There is penalty of 1 for each instance of move, left, and right when the pen was down, encouraging programs to draw boards in few steps. The score of a program is set as the best score achieved by the program starting from any initial location max; s(pll).
In the first sleep phase, DreamCoder trains a neural network that is used to guide program search this recognition model does by taking the board input and predicting programs that are likely to solve that board based on previous successful solutions. The recognition model consists of a 16-channel 3 x 3 convolutional layer, followed by two fully-connected layers with 64 units each, and ending with a linear layer. To generate a vector-based representation of the boards that contains the information necessary for program induction, model as the representation of the board. This embeds each board into a dense vector, similar to the natural language embedding of each board.
the final hidden layer of the recognition
In the library learning sleep phase, DreamCoder grows the DSL. by adding library functions based on reused components of successful programs found during wake (see Fig. 4 for an example). Specifically, it adds program components that minimize the description length of both the current program library and each successful program, once rewritten to this new library component. These learned library abstractions serve to compress programs found in previous wake cycles and better enable the search for future programs in future wake cycles. We execute DreamCoder with and without library learning to test the usefulness of program representations with abstractions.
'This penalty is stronger because incorrectly marked locations can't 'tn't be unmarked, "What's done cannot be undone." Lady Macbeth

---

3.3 Comparing programs and language descriptions
We first consider the descriptions we obtained from humans. We found a wide range in the level of abstraction in human natural language descriptions. Some more literal and verbose in their descriptions whereas others use more abstract concepts to compress their descriptions (Fig. [ On average, human-generated descriptions
shorter than synthetically-genentt tions, which listed every red tile location (Fig. C.. By analogy, library-learned concepts allow for compression of programs (Fig. 4B). Both numan-generale language (vs synthetic) and library- learned programs (vs only primitives) have the same effect on enabling shorter description lengths (Fig.EC)
descrip-
We further found that the description lengths in human language and library-learned programs are significantly correlated (Fig. 4D) across boards. In other words, board that is compress- ible using human language abstractions was also more compressible with library learning over programs. This indicates that the abstractions used by humans language overlap with the abstrac- tions learned by Dreamcoder during library learning. This correlation is not significant when com- paring human generated language description length with description lengths of programs without library learning (Fig. 4D.. This indicates that the correlation may be due to the abstractions cap- tured through library learning. An illustrative example can be seen Fig. | where the program's used library function and the phrase "U shape" within in the human language description are ac- tually the same concept. We verify this further by testing for the difference between these two correlations using panpaaraettric permutation test. The correlation between human-generann generated language description length and library learning program description length statistically signifi- cantly higher (p 0.0034) than the correlation between human-gent -enaannte description length and non-library learning program description length.
Not only do the representations between language and programs have correlated description lengths, but the actual representations learned can be similar well. We consider sample of three relatively simple boards in Fig. 5 and conduct a representational similarity analysis. We find that the similarity structure of these boards overlap across modalities i. they are similar between human- generraed language and library-learen programs, as well as between synthetic language and primitive programs. Conversely, these matrices don't look very similar within modality. That is, the matrices for abstract
language don't look very similar to the matrices for the
non-abstract synthetic language, and similarly for programs. This indicates that modality key driver of similarity in representation space; rather, the level of abstraction can be more salient. This also indicates that the process abstraction changes the representation in similar ways across language and programs.
Representational Similarity (Correlation)
Figure 5: Representational Sim- ilarity Between Language and Programs. Similarity matrices (correlation matrices for respec- tive embeddings) for sample of human numan-generated boards with line motif's for numan-generann language, programs with library learning, synthetic language, and programs without library learn- ing. Abstract representations' similarity profiles (human lan- guage, library programs) across modalities are more similar than those of Aithin-modality representations (human lan- guage+synthetic language, library learning programs-non-liben learning programs).
-075
-350
050
0.5
Language
Program Programs Librar Learning
Language
More Abstract
Synthetic Language
Pragrams with no Library Lasmnig
Less Abstract

---

B
0.0
buseline
-0.5
x(()
1.0
LSTH
white tiles
-1.5
2.0
Human Generated (GSP)
Machine Generated (Control)
diff. from control (essore)
0.5-
program objective
language objective
autoencader objective
0.0-
RoBERTs
-0.5-
Printive
Program
Libraey Preg
Synthetic
Langunge
Human
Langaage
Autoenoccio
Boseline
Human
down on side.
Figure 6: Grounding Mechanism (A) Mechanism for grounding agents using specific repre- sentations. We have the agent's 's encoder output simultancously predict a task embedding while performing the task. These task embeddings can be from programs (embeddings from the recogni- tion model of DreamCoder). language (embeddings from RoBERTa), or the flattened board itself (autoencoder baseline). (B). Performance on previously unseen GSP (colored bars) and control tasks (white bars with colored borders) from grounding agents with different task representations; lower better (since performance based on white tiles revealed). Error bars are 95% confidence intervals across different 15 trained agents (for humans, across 50 different participants), (C). Gap between performance on human-gen aeaaanngeerraeed and machine-gennrater tasks. Higher gap indicates more human-likk like behavior of the model.
3.4 Grounding agents in language and program representations
In order to guide the agent to learn human-like inductive bias, we introduce a task grounding term to the loss function: loss LPPO (0) + Ctaak (yo, 1)). Here (P) is the original PPO loss function, weights the task embedding loss fask (see Fig. [ for the types of task embeddings we used), v the task embedding, and big is the agent's prediction of the task embedding. The task embedding loss task is the Mean Squared Error (MSE) between the agent's predicted and actual task embed- ding. This forces the agent simultaneously predict embedding of the hidden task state (i.e. the underlying board) as well as come up with the best possible action for the current observation (see Fig. (A).
the current model's parameters, Crask is hyperparameter coefficient that
Like the Baseline RI agent Fig. JA we trained our learners on a training set from the human- generated boards and evaluated its performance
held-out test set of nlman--geneatee and maahine-generated boards. We trained different agents to predict different kinds of task embed- language)
dings during training (Fig. (AA based language (both synthetic and and programs (both no library learning and library learning). We refer to co-training on predicting these auxiliary representations
"grounding" the agent the corresponding representation. The term "grounding" is often used work that trains agents to associate data in one modality (i.e. text instructions) to another (i.e. visual images) 47]. We also co-trained directly predicting flattened version of the underlying board stimulus, which is equivalent to co-training the agent on autoencoder loss.
We set out to test whether grounding meta-learning agents on these task embeddings will result in them producing more human-lite performance. We know humans perform better on GSP boards than control boards (Fig. BB,, while the generic meta-RL agent does the opposite. Performing better the GSP boards and worse the control boards would therefore indicate more human- like behavior. To compare task performance across different task distributions or different models, use parammettt bootstrap independent samples test to test for significance.
see that grounding with human-generated descriptions leads to a human-like inductive bias (Fig. (). Training on this auxillary loss led to an increase in performance at the GSP boards relative to the baseline agent (p < 0.0001). The haman-language grounded agent also performed significantly better at the GSP boards than the control boards (p < 0.0001), just like humans do. In

---

contrast, this auxiliary loss significantly reduced performance on the machine- machine-generated control boards (as compared the baseline agent, p = 0.021). This indicates that the auxiliary loss did not generically improve performance across the board, but rather improves performance selectively in the numan-generated boards, and worsens it in the machine generated boards. In other words, makes the agent more human-like. The human-language grounded agent also does the best at the human- human-generated tasks across all models.
The autoencoder agent is substantially better on the GSP boards than the original agent (p < 0.001). However, its performance does not differ significantly between the human and machine- generated tasks (p 0.252), performance pattern different from that of humans. Similarly, we also find that grounding on synthetic descriptions does not lead human-like bias and instead has an effect similar to that of the autoencoder loss it improves performance uniformly rather than selectively. In fact, the agent using synthetic text does better in the control distribution than the GSP distribution (p- = 0.0016), which is qualitatively similar to the performance pattern of the original agent (Fig.) B) and distinctly dissimilar from human behavior.
We now move to examining the agents co-trained with program representations. We found that grounding on library learning programs leads to a human-like inductive bias (Fig. 6B). Specifically, although grounding on non-library program task embeddings does numerically bet- ter on human- human-generater generated tasks than machine-generale
tasks, this difference is not significant (p = 0.11). This difference, however, is highly significant when grounding on library- program task embeddings (p < 0.0001). In addition, grounding on library learning program task embeddings does significantly better at the GSP tasks than grounding on non-library learning pro- gram embeddings (p- < 0.0001). Just as grounding on human language produces more human-like inductive bias than synthetic language, grounding on library learning programs produces more human-like inductive bias than grounding on programs without library learning.
This points to evidence that the level of abstraction of the task embeddings influences the ex- tent to which the agent acquires human inductive bias, as indicated by the differences in performance among low (synthetic for language, no library learning for programs) and high-level (human-generated language and library learning programs) agents. In human- -ennratde descrip- tions, humans write about abstract concepts (e.g. lines, shapes, letters, etc). Similarly abstract concepts can be learned as library functions (see Fig. B). The process of abstraction in language and programs allows for compression of the respective description and induced program (Fig. see fC). The embeddings of these representations that were compressed through abstract concepts may therefore be distilling useful concepts into our meta-learning agent that enables it to acquire human-like inductive bias.
4 Discussion
In this work, we argue that language descriptions and program abstractions can act as repositories for human inductive biases that may be distilled into artificial neural networks. To test this idea, we considered a tile-revealing task with 2D grids that were directly sampled from human priors. We found that grounding on either numan-generatie not only improves machine performance on tasks that people perform well at, but also impairs performance on tasks that people perform more poorly at. Although the idea of co-training agents on language to shape their representations has been explored before, most works utilize synthetic descriptions 21). Our work suggests human-gene samman-generated language can lead to more human-like performance than more literal synthetic language descriptions [16]. because human-generated descriptions contain information about abstract concepts (c.g. lines, shapes, letters, ctc) that com- press description length and are reflected in tasks directly sampled from human priors, therefore better capturing human inductive biases. Although we used synthetic descriptions that are as lit- eral/devoid of abstractions as possible to test this point, synthetic descriptions built to contain such abstractions could work as well. The challenge, then, in using such synthetic descriptions would be coming up with the correct abstractions to build in, rather than crowdsourcing such abstractions from human participants as we do in this work.
language or programs with library learning
We also show evidence that co-training artificial agents on representations from program induction results in learning human inductive biases just as co-training on language does. Programs induced with library-learned learned concepts [24] are analogous to language in that adding library-learned con-

---

cepts to the initial DSL enables compression of description length just as how humans use abstract concepts to compress description length. Like human language, programs with library abstractions better enable learning of human-like inductive biases. Program induction can sometimes be dif- ficult due to the combinatorial explosion of the program space 4X] and the non-trivial decision of picking DSL. Co-supervision of tabula rasa artificial agents using representations from pro- gram induction can be a useful way to build agents with both the flexibility of neural networks and the human-like priors that program induction can provide. Additionally, although we investigate language and programs as separate but analogous representations that can guide the training of artificial agents, recent work has also learned joint compositional generative models over program libraries and natural language descriptions [18]. An exciting direction for future work is to com- bine our objectives, utilizing joint program and language representations to instill human inductive biases in artificial agents.
In this work, we use constrained task paradigm on 2D grids, which gives us the advantage of being able to directly study samples of human and machine priors on the task space (Fig. 2). Al- though some abstractions we found may be specific to this domain, the literature in cognitive science [4] provides good reason to expect that the effectiveness of our approach in this more ar- tificial domain could translate to abstractions in real-world domains that are natural for humans. Some specific domains, such as mathematical reasoning. planning, or puzzle solving, may even more obviously necessitate abstractions [49]. It will be exciting future work to use approach to empirically validate this conjecture by determining the space of tasks for which co- vo-training with language and program abstractions improves an artificial neural agent's 's performance. This future work will be an exciting opportunity both for studying human intelligence by getting a bet- ter context for what the role of abstraction is in everyday cognition as well as improving machine intelligence by expanding the capabilities of current neural agents. However, there is still work to do to determine how language and programs can influence acquisition of human inductive biases in more scaled up, naturalistic, and real-world settings.
On the language side, the main bottleneck is that collecting human descriptions with sufficient coverage can become prohibitively expensive as the state space grows larger. One potential remedy would be to employ active learning methods to focus language elicitation the most relevant abstractions in the environment and to explore data augmentation approaches to make these de- the human descriptions to generate sufficiently high-quality descriptions for new states. There is some new work showing that even small amount of language can be used for models to perform visual grounding on large environments [51]. This approach to scaling language may require scoring descriptions for their quality in describing the relevant abstractions in the environment, which is less time-consuming for human participants than producing such descriptions.
scriptions go further. For example, a vision-language model (50) could be fine-tuned
On the program side, the need to define ad hoc base DSLs for each domain is an important limi- tation of any approach relying on program abstractions. An important direction for this area is to develop 'canonical' basis for program representations that may apply across many domains. For example, a graphics engine may provide one such general-purpose DSL to synthesize state abstrac- tions for larger and more realistic visual environments. This kind of base DSL could be seen as an extension to the generic 2D vector graphics used by Ellis et al. [52) to induce graphics programs from hand-drawn images. A potential bottleneck here is that library learning can be especially computationally expensive for larger environments, but several algorithmic improvements have already emerged that make this direction more promising (53]
Strong inductive biases towards abstract structure allow humans to generalize to novel environ- ments without much experience [4]. By instilling these biases in our artificial agents, we can work towards enabling machines to demonstrate human-like general intelligence.
Acknowledgements
S.K. was supported by NIH T32MH065214 during the duration of this work. R.D.H is supported by a C.V Starr Fellowship. This work was supported by the the U.S. Army Research Office grant W91 16---66--44 the DARPA L2M Program, ONR grant 180014-18-1-2881 and the John Templeton Foundation. The opinions expressed in this publication are those of the authors and do not necessarily reflect the views of the funders.
10

---

References
[1] Jeanne Ellis Ormrod. Human Learning, eBook. Pearson Higher Ed, 2016.
[2] Joshua B Tenenbaum, Charles Kemp. Thomas Griffiths, and Noah D Goodman. How to grow a mind: Statistics, structure, and abstraction. Science, 131(6022) 2011.
[3] Thomas L. Griffiths, Nick Chater, Charles Kemp, Amy Perfors, and Joshua B Tenenbaum. Probabilistic models of cognition: exploring representations and inductive biases. Trends in Cognitive Sciences, 14(8):357-3644 2010.
[4] Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. Building machines that learn and think like people. Behavioral and Brain Sciences, 40, 2017.
[5] Tom M Mitchell. The Need for Biases in Learning Generalizations, Department of Computer Science, Laboratory for Computer Science Research 1980.
[6] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
[7] Anirudh Goyal and Yoshua Bengio. Inductive biases for deep learning of higher-level cogni- tion. arXiv preprint arXiv:2011. 1. 15091 2020.
[8] Thomas L Griffiths, Frederick Callaway, Michael B Chang, Erin Grant, Paul M Krueger, and Falk Lieder. Doing more with less: meta-reasoning and meta-learning in humans and machines. Current Opinion in Behavioral Sciences, 29:24-30, 2019.
[9] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. arXiv preprint arXiv:2004.05445 05439, 2020.
[10] Sreejan Kumar, Ishita Dasgupta, Jonathan Cohen, Nathaniel Daw, and Thomas Griffiths. Meta- learning of structured task distributions in humans and machines. In International Conference on Learning Representations, 2021.
[11] Piotr Mirowski, Razvan Pascanu, Fabio Viola, Hubert Soyer, Andrew J Ballard, Andrea Banino, Misha Denil, Ross Goroshin, Laurent Sifre, Koray Kavukcuoglu, et al. Learning navigate in complex environments. arXiv preprint arXiv:1611.0366 2016.
[12] Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. Loss is own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv: 16/2.0737/ 2016.
[13] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Leibo, David Silver, and Koray Kavukcuoglu. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.085 1.05397, 2016.
[14] Gary Lupyan and Benjamin Bergen. How language programs the mind. Topics in Cognitive Science, 8(2):488-4 -24, 2016.
[15] Elizabeth S Spelke. What makes smart? core knowledge and natural language. Language in Mind: Advances in the Study of Language and Thought, page 277, 2003.
[16] Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 2166-21799 2018.
[17] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Ed- ward Grefenstette, Shimon Whiteson, and Tim Rocktäschel. A survey of reinforcement learn- ing informed by natural language. In International Joint Conferences on Artificial Intelligence, 2019.

---

[18] Catherine Wong, Kevin M Ellis, Joshua Tenenbaum, and Jacob Andreas. Leveraging language learn program abstractions and search heuristics, In International Conference on Machine Learning, pages 11 11193-1 1204. PMIR, 2021.
[19] Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Rexearch, 63:849-874, 201 8.
[20] Jesse Mu, Percy Liang, and Noah Goodman. Shaping visual representations with language for few-shot classification. In Proceedings of the 58th Anmual Meeting of the Association for Compulational Linguistics, pages 4823-4830, 2020.
[21] Andrew K Lampinen, Nicholas A Roy, Ishita Dasgupta, Stephanie CY Chan, Allison C Tam, James L McClelland, Chen Yan, Adam Santoro, Neil C Rabinowitz, Jane X Wang, et al. Tell why! - explanations support learning of relational and causal structure. arXiv preprint urXiv:2112.0375: 2021.
[22] Austin W Hanjie, Ameet Deshpande, and Karthik Narasimhan. Semantic supervision: Enabling generalization over output spaces. arXiv preprint (rXiv:2202.13/00 2022.
[23] Brenden M Lake, Ruslan Salakhutdinov, and Joshua B Tenenbaum. Human-level concept learning through probabilistic program induction. Science, 350062666:1333-11388 2015.
[24]
Kevin Ellis, Catherine Wong, Maxwell Nye, Mathias Sablé-Meyer, Lucas Morales, Luke He- witt, Luc Cary, Armando Solar-Lezama, and Joshua B Tenenbaum. Dreamcoder: bootstrapping inductive program synthesis with wake-sleep library learning. In Proceedings of the 42nd ACM SIGPLAN International Conference Programming Language Design and Implementation, pages 835-850, 2021.
[25] Catherine Wong, William P McCarthy, Gabriel Grand, Yoni Friedman, Joshua B Tenenbaum, Jacob Andreas, Robert D Hawkins, and Judith E Fan. Identifying concept libraries from language about object structure. arXiv preprint arXiv:2205.0500 2022.
[26] William P McCarthy, Robert D Hawkins, Haoliang Wang, Cameron Holdaway, and Ju- dith E Fan. Learning to communicate about shared procedural abstractions. arXiv preprint a7Xiv:2107.00077, 2021.
[27] Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B Tenenbaum, and Jiajun Wu. The neuro- symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. arXiv preprint arXiv: 1904. 12584, 2019.
[28] Maxwell Nye, Armando Solar-Lezama, Josh Tenenbaum, and Brenden M Lake. Learning compositional rules via neural program synthesis. Advances in Neural Information Processing 2020.
Systems,
[29] Lucas Tian, Kevin Ellis, Marta Kryven, and Josh Tenenbaum. Learning abstract structure for drawing by efficient motor program induction. Advances Neural Information Processing Systems, 33:2686-248 2697, 2020.
[30] James I. McClelland, Matthew M Botvinick, David C Noelle, David C Plaut, Timothy T Rogers, Mark S Seidenherg, and Linda B Smith. Letting structure emerge: Connectionist and dynamical systems approaches to cognition. Trends in Cognitive Sciences, 14(8):348-356, 2010.
[31] R Thomas McCoy, Erin Grant, Paul Smolensky, Thomas I. Griffiths, and Tal Linzen. Universal linguistic inductive biases via meta-learnin. arXiv preprint arXiv:2006. 16324, 2020.
[32] Ishita Dasgupta, Jane Wang, Silvia Chiappa, Jovana Mitrovic, Pedro Ortega, David Raposo, Edward Hughes, Peter Battaglia, Matthew Botvinick, and 7eb Kurth-Nelson. Causal reasoning from meta-reinforcement learning. arXiv preprint 1rXiiv/900.000 162, 2019.
[33] Neil Rabinowitz, Frank Perbet, Francis Song, Chiyuan Zhang, SM Ali Eslami, and Matthew Botvinick. Machine theory of mind. In International conference on machine learning, pages 4218 11884427. PMLR, 2018
12

---

[34] Peter Harrison, Raja Marjich, Federico Adolfi, Pol van Rijn, Manuel Anglada- Tort, Ofer Tchernichovski, Pauline Larrouy-Maesta and Nori Jacoby. Gibbs sampling with people. Advances in Neural Information Processing Systems, 33: 10:5066-11 2020.
[35] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprini 2018.
[36] Jane X Wang, Zeb Kurth-Nelson, Dharshan Kumaran, Dhruva Tirumala, Hubert Soyer, Joel Z Leibo, Demis Hassabis, and Matthew Botvinick. Prefrontal cortex as a meta-reinforcement learning system. Nature neuroscience, 21(6):860- 868, 2018.
[37] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: (ttd::/777.63447,7 2017.
[38]
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, and Noah Dormann. Stable baselines3, 2019.
[39] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer. and Veselin Stoyamov. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint 1rXiv:190.111692 2019.
[40] Nils Reimers and Iryna Gurevych. Sentence-BERG Sentence embeddings using siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 3982-3992, 2019.
[41] Deb K Roy and Alex P Pentland. Learning words from sights and sounds: A computational model. Cognitive science, 26(1): 13-)1113-46,,11 2002.
[42] Haris Dindo and Daniele Zambuto. A probabilistic approach to learning a visually grounded language model through human-robot interaction. In 2010 JEEE/RS/ International Conference on Intelligent Robots and Systems, pages 790-796. IEEE, 2010.
[43] Adam Vogel, Karthik Raghunathan, and Dan Jurafsky. Eye spy: Improving vision through dialog. In 2010 AAAI Fall Symposium Series, 2010.
[44] Changsong Liu, Lanbo She, Rui Fang, and Joyce Chai Probabilistic labeling for efficient referential grounding based on collaborative discourse. In Proceedings of the 52nd Anmual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 13-18, 2014.
[45] Hanwang Zhang, Yulei Niu, and Shih-Fu Chang. Grounding referring expressions in images by variational context. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 4158-4166, 2018
[46] Haonan Yu, Xiaochen Lian, Haichao Zhang, and Wei Xu. Guided feature transformation (gft): A neural language grounding module for embodied agents. In Conference on Robot Learning, pages 81-98. PMLR, 2018.
[47] Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems, 34:200-212, 2021.
[48] Sumit Gulwani, Oleksandr Polozov, Rishabh Singh, al. Program synthesis. Foundations and Trends@ in Programming Languages, 4(1 4(1-2):1-119 2017.
[49] Allen Newell, Herbert Alexander Simon, et al. Human problem solving, volume 104, Prentice- hall Englewood Cliffs, NJ, 1972.
[50] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8766 PMLR, 2021.
13

---

[51] Chen Yan, Federico Carnevale, Petko Georgiev, Adam Santoro, Aurelia Guy, Alistair Muldal, Chia- Hung, Josh Abramson, Timothy Lillicrap, and Gregory Wayne. Intra-agent speech permits zero-shot task acquisition. arXiv preprint arXiv:2206.031332 2022.
[52] Kevin Ellis, Daniel Ritchie, Armando Solar-Lezama, and Josh Tenenbaum. Learning to infer graphics programs from hand-drawn images. Advances in neural information processing systems, 31, 2018.
[53] MATTHEW BOWERS, THEO X OLAUSSON, CATHERINE WONG, GABRIEL GRAND, JOSHUA B TENENBAUM, KEVIN ELLIS, and ARMANDO SOLAR-LEZAMA, Top-down synthesis for library learning.
14


Reviews:
[{"review_id": "UAarKPlDOJS", "reviewer": "Reviewer_QkpB", "summary": "The aim of this paper is twofold: (1) showing humans and machines use different inductive biases, and (2) showing that machines can learn to generalise more like humans do by distilling human inductive biases into them during training.\n\nAs a proxy for human-like inductive bias, the authors use the setup from [11] with two distributions of tile revealing tasks: one reflecting human inductive biases and one so-called \"metamer\" task, that is similar to the human-biased task in statistics, but is generated differently. The task reflecting human inductive biases (called GSP boards) are 2D grids with shapes sampled directly from humans. The metamer task uses the exact same sampling procedure, but instead of a human in the loop, there is a machine in the loop (called machine-generated boards). The task the agent is evaluated on is revealing tiles on a 2D grid to uncover a certain shape of red tiles, while revealing as little white tiles as possible. For the human-like tasks, imagine U-like, I-like, or pyramid-like shapes on a board, and the machine-generated boards are similar except that it's not symmetric and smooth shapes. Humans perform better on the GSP boards than on the machine-generated boards. \n\nTowards aim (1), showing humans and machines use different inductive biases, the authors train an RL agent with PPO on a set of GSP boards (i.e. red shapes on a 2D grid), and test on a held-out set of GSP boards as well as on machine-generated boards. Even though the agent has been trained on GSP boards, the agent performs better on the machine-generated boards than on the GSP boards. The authors hypothesise that this is because humans and machines use different biases to solve the task, and even though the machine is trained on the human-generated boards reflecting the human inductive biases, because the machine-generated boards have similar statistics to the GSP boards, they are better at those\n\nTowards aim (2) the authors train the baseline model again but with an auxiliary term added to the loss. The encoder of the baseline model is, besides learning the policy, trained to reconstruct a representation of the grid. Four different representations are experimented with: (1) synthetic language describing the board (red tile on row x1 col y1, red tile on row x2 col y2, etc.), (2) crowdsourced natural language describing the board (u-like shape in the bottom left, etc.), (3) a program constructed of a bunch of primitives from a DSL learned with DreamCoder (primitives like move, pen-up, pen-down, fork, left, right, so programs end up following the shape), (4) a program  learned with DreamCoder + library learning (meaning certain re-usable learned programs are added to the DSL). The authors show that the descriptions lengths of natural language descriptions and programs with library learning are correlated, and that some abstractions are even similar (i.e. humans say U-shape, and DreamCoder adds a U-shaped-program to the DSL). Additionally, they train the encoder to simply reconstruct the flattened grid, called the autoencoder objective.\nResults show that for the autoencoder objective, synthetic language objective, and primitive program objective (DreamCoder without library learning) don't distill human-like inductive biases, but natural language descriptions and programs with re-used components do. This indicates that the level of abstraction of the task embeddings is important for distilling human inductive bias. Human-generated language is important and works better than synthetically generated language. ", "questions": "- I don't see under what definition this method uses meta-learning. As far as I know [2], meta-learning always has some meta-phase (i.e. doing some learning over the task distribution) to learn a meta representation (e.g. initialisation of parameters like in MAML) and an inner phase on a task sampled from the task distribution. This method does have a task distribution, but there's no notion of meta-learning as far as I can see. I could have an outdated idea of what constitutes meta-learning nowadays, so adding it to the questions :)\n\n- Nit 1: Can you add how many runs you did to get the performance in variance, and add in the bar chart figures that the bars are variance over different runs / different humans?\n\n- Nit 2: I can't parse line 249-250\n\n- Nit 3: I personally found library learning in the abstract slightly confusing because I didn't know the term. Generally, I think the abstract could use some work on clarity.\n\n[2] Meta-Learning in Neural Networks: A Survey; Timothy Hospedales, Antreas Antoniou, Paul Micaelli, Amos Storkey", "limitations": "The two main limitations are properly addressed by the authors in the discussion, but I would like to see some more speculation on the direction of the future work and the scalability of this method. Party copied from the strengths&weaknesses part above my suggestions are: \n\n- Would like to see more discussion on scaling this approach to real-world tasks. There are many tasks for which humans are thought to use program-like representations (e.g. mathematics, counting, addition, etc., see [1] below). \n\n- Perhaps vision-language models could be used to generate the language descriptions to get around the need for crowdsourcing them.\n\n- I wonder whether these results transfer to problems where the solution is less obviously constructed from the abstractions, perhaps you could discuss this", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This paper is very carefully and clearly written, with clear figures supplementing understanding. \n- The experimental setup is sound and convinces the reader of the main points.\n- The result that reproduces Kumar et al. in showing that humans have different inductive biases is an interesting and (to me) surprising result, and even though it's not a novel result it's still important in RL to replicate results from other papers with different algorithms (PPO vs A2C). Different training algorithms can give very different results in RL, and the fact that this result is stable under PPO and A2C strengthens it.\n- Interesting main findings; natural language descriptions and programs learned with DreamCoder + library learning are correlated in certain features, and even share abstractions, and these representations are also the ones that make the model generalise more like humans do. \n- I'm very excited about the potential of using the abstractions that are naturally present in human language for better generalisation, this is a very important research direction and I would like to see more in this area, this paper is a great step in that direction\n\nWeaknesses:\n- It is mentioned in the discussion, but the weakness of this method is that either a DSL needs to be created or natural language descriptions of each task in the dataset. The former isn't as expensive but might be difficult for more realistic tasks, and the latter is expensive. It would be nice to see some more discussion on the feasibility of scaling this approach to real-world tasks. There are many tasks for which humans are thought to use program-like representations (e.g. mathematics, counting, addition, etc., see [1] below). Also, perhaps vision-language models could be used to generate the language descriptions to get around the need for crowdsourcing them.\n- Related; the tile-revealing task is a task very much tailored to literally visualising human-like inductive biases. Basically, in this task the solution is a sort of sum of the abstractions (e.g. U-like shape plus some red tile somewhere). Would these results transfer to problems where the solution is less obviously constructed from the abstractions?\n- This is a very minor point, and maybe more of an opinion than a weakness, but in the discussion you claim that your work suggests \"not all language is created equal: human-generated language leads to more human-like performance than synthetic language descriptions\", but this seems like more of an artefact of the particular type of synthetic language you chose, namely one without any abstractions, simply listing the tiles. Probably a synthetically generated description that would take into account shapes like lines (which are easily automatically detected) would work as well as natural language. In any case, I do agree that natural language is more suited to this, I just think the comparison shouldn't be synthetic versus natural but rather learned/natural abstractions (language/library learning) versus hardcoded synthetic abstractions (that will cover less of the abstractions in the data).\n\n[1] The Child as Hacker, Joshua S. Rule, Joshua B. Tenenbaum, and Steven T. Piantadosi", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9D2RU7P-fQa", "reviewer": "Reviewer_zreP", "summary": "This paper presents an experimental setup and set of results that analyze (1) whether RL agents exhibit similar behavior to people in guessing which squares on a board are colored, in two different distributions of boards: human-elicited, and machine-produced, and (2) whether more human-like behavior (good performance on human-elicited; poor on machine-produced) is obtained by the RL agent when co-training the it using an auxiliary loss that predicts one of 4 types of representation: human-written board descriptions, low-level synthetic language board descriptions, program abstractions induced by the DreamCoder algorithm, or program primitives (which span 2 modalities and 2 rough levels of abstraction). The paper's main finding is that regularizing RL agent training with either human-generated language or induced program abstractions make the agent's behavior correspond more closely to human behavior.", "questions": "Q1) 123- I was pretty unclear on how meta-learning is used in the agent training. Is a meta-learning algorithm used (it doesn't seem to be; PPO seems to be applied in each individual task), or is it a meta-learning architecture? If the second, does the LSTM model the temporal nature of individual tasks (recurrence is across timesteps for individual agent actions) or the temporal nature of learning across tasks?\n\nQ2) What are 120 LSTM units? Does this mean an LSTM with hidden dimension 120, or 120 different LSTM cells, or 120 unrolled time steps (or something else)?\n\n*Suggestions*\n\nThe paper shows similar effects from natural language abstractions & learned program abstractions, but it would be interesting to also investigate whether there is a direct correspondence between these modalities, e.g. using alignment methods similar to those in Andreas et al. 2017, Translating Neuralese or Andreas 2019, Measuring compositionality in representation learning [I feel I should disclaim here that I am not an author of these works :) ].\n\nClaims in the results descriptions / discussion that I'd suggest revising/rewording:\n- 275: \"improves performance uniformly\": if I'm reading the results correctly, autoencoder seems to decrease the performance of the RL baseline on the machine-generated boards\n- 302-303: I may have misunderstood, but the library learning isn't grounded in language in the current work, is it (i.e. the paper doesn't do a LAPS-like, Wong et al. approach) -- it's just the RL agent that is grounded?\n- Mu et al. does use human-generated language for the Birds dataset. This would also be a good place to cite Andreas et al., Learning with Latent Language\n- 312: this is describing the results in Fig 5B, right? there are really only 2 bits in the comparison (relative to the RL agent with no representation prediction, GSP goes up; control goes down), and so this wasn't totally convincing to me\n\n\n*Minor suggestions*\n- line 51: is *a* meta-learning\n- 59: define the auxiliary representation here\n- 93-95: this seems intuitive, but it could help to give some stats to make it concrete -- what does it mean to respect most of the stat properties?\n- 240: it wasn't clear until seeing Fig 6 that \"task attribute\" can include representations of the language or program.\n- 250-252: I confess that having only these citations bugged me a bit, given the long tradition of work on language grounding before 2018. \n- Fig 5 caption: \"across modalities\" was a bit confusing to me: it's the correlation matrices that are being compared across modalities, right, not the individual representations? (i.e. language embeddings are never compared to program embeddings)", "limitations": "The discussion of the limitations at the end of the paper was overall thorough and clear. The main things I would suggest addressing are in the weaknesses section above.\n\nOne additional point of uncertainty for me was what typifies the \"machine prior\". It's not at all clear to me that an agent should do better on the control distribution than on the human distribution, particularly given what seem to be differences in the architecture used to produce the distribution and the one used in the RL agent. It would strengthen the paper to explore this more. The nearest neighbor heuristic also encodes a human-like prior in some sense, and since many results are reported relative to this heuristic, I wonder if this could be stacking the cards against the agent. \n", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "*Strengths*\n\nS1) I found the paper thought-provoking -- it's likely to stimulate future work on program induction and grounding language to abstraction. I'm likely to recommend this paper to others working in these areas, as well as to point to it as a nice example of a paper that combines cognitive science investigation with solid ML techniques.\n\nS2) The experiments and analysis were overall very well-designed and carefully controlled (but see below for some minor suggestions) and thorough, anticipating and heading off several concerns that I had. \n\nS3) I thought the choice of setting was a strength -- the grid world environment seems rich enough to produce sophisticated natural language and programs at varying levels of abstraction, while still being simple enough to afford controlled experiments, in particular collecting the GSP boards.\n\nS4) The paper was extremely well-written and clear. I enjoyed reading it.\n\n\n*Weaknesses*\n\nW1) The use of language in the paper is not quite as well-developed as the use of program abstractions. There's a stark contrast between the nature of the synthetic language and human-written language -- not just in the level of abstraction, but also in being human-written versus synthetic. The synthetic language is likely far out-of-distribution for the RoBERTa model, and the language representations (if I understood correctly) weren't induced for any of the tasks in this paper (the RoBERTa model was frozen). I would have appreciated a finer-grained analysis that compared different levels of abstraction in the human-written language. But I don't think this is a crucial weakness.\n\n*Edit after response:* In the response, the authors authors point out differential effects of synthetic language on human vs model boards, which I agree give evidence that improvements from human language aren't just due to being OOD for RoBERTa. So I'm even less concerned about this now.\n\nW2) I worry a bit that the findings here are bound to depend on the program abstractions and the language elicited. I don't think this is a crucial weakness because the paper was pretty careful here (primitives and synthetic language, while similar in design, are natural for this domain; and the language is human-elicited and program abstractions are learned from just a few low-level primitives) -- but I'd be more convinced if the main findings were also shown to hold in another domain. But again I don't think this is a crucial weakness, there's enough here that that could be followup work.\n\n*Edit after response:* I agree with the authors' points that these would be exciting for future work but are out-of-scope for the submission.\n\nI've updated my score to an 8 (from a 7).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "8k3BbpTDx28", "reviewer": "Reviewer_czvj", "summary": "This paper explores the intersection of abstraction, induction, language, and behavior. The goal of the paper is to instill human-like inductive biases into neural networks, in an attempt to improve generalization and performance. This is accomplished by using auxiliary tasks in an RL framework that attempt to elicit structured internal representations that match human representations, as measured by language-based representations of tasks.  The paper presents careful experimentation that involves a variety of tests of human and synthetic descriptions, and show that asking the agent to match human language generally induces more human like inductive biases.", "questions": "- Is it possible that the fact that DreamCoder has a DNN inside of it presents a confound that might result in its libraries and procedures being too similar to what an agent would learn directly anyway?", "limitations": "This paper suitably address both limitations and potential ethical concerns (which are minor).", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Wow. I really, really enjoyed this paper. It was thoughtful, extraordinarily well-written, and intellectually stimulating.\n\n+ The idea of inducing human-like abstractions in general DNNs is important and potentially impactful\n\n+ The paper is one of the most clearly written papers I've read in a long time\n\n+ The experimental setup is complex, but clean, with each step naturally building on previous steps\n\n+ The paper is intellectually enlarging, proposed new methodologies and tasks to make progress against a hard problem\n\n- I wish there was some sort of clearer path to \"scaling this up\" to the next step -- a more complex agent, a larger DNN, a richer task.\n\n- Ultimately, the paper only provides an /indirect/, behavioral measure of whether or not the internal representations of the DNN are human-like. It would be intriguing to see if any sort of /direct/ measure is possible (maybe by analyzing weights directly?).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept