Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper explores the intersection of abstraction, induction, language, and behavior. The goal of the paper is to instill human-like inductive biases into neural networks, in an attempt to improve generalization and performance. This is accomplished by using auxiliary tasks in an RL framework that attempt to elicit structured internal representations that match human representations, as measured by language-based representations of tasks.  The paper presents careful experimentation that involves a variety of tests of human and synthetic descriptions, and show that asking the agent to match human language generally induces more human like inductive biases.
strengths_and_weaknesses: Wow. I really, really enjoyed this paper. It was thoughtful, extraordinarily well-written, and intellectually stimulating.

+ The idea of inducing human-like abstractions in general DNNs is important and potentially impactful

+ The paper is one of the most clearly written papers I've read in a long time

+ The experimental setup is complex, but clean, with each step naturally building on previous steps

+ The paper is intellectually enlarging, proposed new methodologies and tasks to make progress against a hard problem

- I wish there was some sort of clearer path to "scaling this up" to the next step -- a more complex agent, a larger DNN, a richer task.

- Ultimately, the paper only provides an /indirect/, behavioral measure of whether or not the internal representations of the DNN are human-like. It would be intriguing to see if any sort of /direct/ measure is possible (maybe by analyzing weights directly?).
questions: - Is it possible that the fact that DreamCoder has a DNN inside of it presents a confound that might result in its libraries and procedures being too similar to what an agent would learn directly anyway?
limitations: This paper suitably address both limitations and potential ethical concerns (which are minor).
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The aim of this paper is twofold: (1) showing humans and machines use different inductive biases, and (2) showing that machines can learn to generalise more like humans do by distilling human inductive biases into them during training.

As a proxy for human-like inductive bias, the authors use the setup from [11] with two distributions of tile revealing tasks: one reflecting human inductive biases and one so-called "metamer" task, that is similar to the human-biased task in statistics, but is generated differently. The task reflecting human inductive biases (called GSP boards) are 2D grids with shapes sampled directly from humans. The metamer task uses the exact same sampling procedure, but instead of a human in the loop, there is a machine in the loop (called machine-generated boards). The task the agent is evaluated on is revealing tiles on a 2D grid to uncover a certain shape of red tiles, while revealing as little white tiles as possible. For the human-like tasks, imagine U-like, I-like, or pyramid-like shapes on a board, and the machine-generated boards are similar except that it's not symmetric and smooth shapes. Humans perform better on the GSP boards than on the machine-generated boards. 

Towards aim (1), showing humans and machines use different inductive biases, the authors train an RL agent with PPO on a set of GSP boards (i.e. red shapes on a 2D grid), and test on a held-out set of GSP boards as well as on machine-generated boards. Even though the agent has been trained on GSP boards, the agent performs better on the machine-generated boards than on the GSP boards. The authors hypothesise that this is because humans and machines use different biases to solve the task, and even though the machine is trained on the human-generated boards reflecting the human inductive biases, because the machine-generated boards have similar statistics to the GSP boards, they are better at those

Towards aim (2) the authors train the baseline model again but with an auxiliary term added to the loss. The encoder of the baseline model is, besides learning the policy, trained to reconstruct a representation of the grid. Four different representations are experimented with: (1) synthetic language describing the board (red tile on row x1 col y1, red tile on row x2 col y2, etc.), (2) crowdsourced natural language describing the board (u-like shape in the bottom left, etc.), (3) a program constructed of a bunch of primitives from a DSL learned with DreamCoder (primitives like move, pen-up, pen-down, fork, left, right, so programs end up following the shape), (4) a program  learned with DreamCoder + library learning (meaning certain re-usable learned programs are added to the DSL). The authors show that the descriptions lengths of natural language descriptions and programs with library learning are correlated, and that some abstractions are even similar (i.e. humans say U-shape, and DreamCoder adds a U-shaped-program to the DSL). Additionally, they train the encoder to simply reconstruct the flattened grid, called the autoencoder objective.
Results show that for the autoencoder objective, synthetic language objective, and primitive program objective (DreamCoder without library learning) don't distill human-like inductive biases, but natural language descriptions and programs with re-used components do. This indicates that the level of abstraction of the task embeddings is important for distilling human inductive bias. Human-generated language is important and works better than synthetically generated language. 
strengths_and_weaknesses: Strengths:
- This paper is very carefully and clearly written, with clear figures supplementing understanding. 
- The experimental setup is sound and convinces the reader of the main points.
- The result that reproduces Kumar et al. in showing that humans have different inductive biases is an interesting and (to me) surprising result, and even though it's not a novel result it's still important in RL to replicate results from other papers with different algorithms (PPO vs A2C). Different training algorithms can give very different results in RL, and the fact that this result is stable under PPO and A2C strengthens it.
- Interesting main findings; natural language descriptions and programs learned with DreamCoder + library learning are correlated in certain features, and even share abstractions, and these representations are also the ones that make the model generalise more like humans do. 
- I'm very excited about the potential of using the abstractions that are naturally present in human language for better generalisation, this is a very important research direction and I would like to see more in this area, this paper is a great step in that direction

Weaknesses:
- It is mentioned in the discussion, but the weakness of this method is that either a DSL needs to be created or natural language descriptions of each task in the dataset. The former isn't as expensive but might be difficult for more realistic tasks, and the latter is expensive. It would be nice to see some more discussion on the feasibility of scaling this approach to real-world tasks. There are many tasks for which humans are thought to use program-like representations (e.g. mathematics, counting, addition, etc., see [1] below). Also, perhaps vision-language models could be used to generate the language descriptions to get around the need for crowdsourcing them.
- Related; the tile-revealing task is a task very much tailored to literally visualising human-like inductive biases. Basically, in this task the solution is a sort of sum of the abstractions (e.g. U-like shape plus some red tile somewhere). Would these results transfer to problems where the solution is less obviously constructed from the abstractions?
- This is a very minor point, and maybe more of an opinion than a weakness, but in the discussion you claim that your work suggests "not all language is created equal: human-generated language leads to more human-like performance than synthetic language descriptions", but this seems like more of an artefact of the particular type of synthetic language you chose, namely one without any abstractions, simply listing the tiles. Probably a synthetically generated description that would take into account shapes like lines (which are easily automatically detected) would work as well as natural language. In any case, I do agree that natural language is more suited to this, I just think the comparison shouldn't be synthetic versus natural but rather learned/natural abstractions (language/library learning) versus hardcoded synthetic abstractions (that will cover less of the abstractions in the data).

[1] The Child as Hacker, Joshua S. Rule, Joshua B. Tenenbaum, and Steven T. Piantadosi
questions: - I don't see under what definition this method uses meta-learning. As far as I know [2], meta-learning always has some meta-phase (i.e. doing some learning over the task distribution) to learn a meta representation (e.g. initialisation of parameters like in MAML) and an inner phase on a task sampled from the task distribution. This method does have a task distribution, but there's no notion of meta-learning as far as I can see. I could have an outdated idea of what constitutes meta-learning nowadays, so adding it to the questions :)

- Nit 1: Can you add how many runs you did to get the performance in variance, and add in the bar chart figures that the bars are variance over different runs / different humans?

- Nit 2: I can't parse line 249-250

- Nit 3: I personally found library learning in the abstract slightly confusing because I didn't know the term. Generally, I think the abstract could use some work on clarity.

[2] Meta-Learning in Neural Networks: A Survey; Timothy Hospedales, Antreas Antoniou, Paul Micaelli, Amos Storkey
limitations: The two main limitations are properly addressed by the authors in the discussion, but I would like to see some more speculation on the direction of the future work and the scalability of this method. Party copied from the strengths&weaknesses part above my suggestions are: 

- Would like to see more discussion on scaling this approach to real-world tasks. There are many tasks for which humans are thought to use program-like representations (e.g. mathematics, counting, addition, etc., see [1] below). 

- Perhaps vision-language models could be used to generate the language descriptions to get around the need for crowdsourcing them.

- I wonder whether these results transfer to problems where the solution is less obviously constructed from the abstractions, perhaps you could discuss this
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an experimental setup and set of results that analyze (1) whether RL agents exhibit similar behavior to people in guessing which squares on a board are colored, in two different distributions of boards: human-elicited, and machine-produced, and (2) whether more human-like behavior (good performance on human-elicited; poor on machine-produced) is obtained by the RL agent when co-training the it using an auxiliary loss that predicts one of 4 types of representation: human-written board descriptions, low-level synthetic language board descriptions, program abstractions induced by the DreamCoder algorithm, or program primitives (which span 2 modalities and 2 rough levels of abstraction). The paper's main finding is that regularizing RL agent training with either human-generated language or induced program abstractions make the agent's behavior correspond more closely to human behavior.
strengths_and_weaknesses: *Strengths*

S1) I found the paper thought-provoking -- it's likely to stimulate future work on program induction and grounding language to abstraction. I'm likely to recommend this paper to others working in these areas, as well as to point to it as a nice example of a paper that combines cognitive science investigation with solid ML techniques.

S2) The experiments and analysis were overall very well-designed and carefully controlled (but see below for some minor suggestions) and thorough, anticipating and heading off several concerns that I had. 

S3) I thought the choice of setting was a strength -- the grid world environment seems rich enough to produce sophisticated natural language and programs at varying levels of abstraction, while still being simple enough to afford controlled experiments, in particular collecting the GSP boards.

S4) The paper was extremely well-written and clear. I enjoyed reading it.


*Weaknesses*

W1) The use of language in the paper is not quite as well-developed as the use of program abstractions. There's a stark contrast between the nature of the synthetic language and human-written language -- not just in the level of abstraction, but also in being human-written versus synthetic. The synthetic language is likely far out-of-distribution for the RoBERTa model, and the language representations (if I understood correctly) weren't induced for any of the tasks in this paper (the RoBERTa model was frozen). I would have appreciated a finer-grained analysis that compared different levels of abstraction in the human-written language. But I don't think this is a crucial weakness.

*Edit after response:* In the response, the authors authors point out differential effects of synthetic language on human vs model boards, which I agree give evidence that improvements from human language aren't just due to being OOD for RoBERTa. So I'm even less concerned about this now.

W2) I worry a bit that the findings here are bound to depend on the program abstractions and the language elicited. I don't think this is a crucial weakness because the paper was pretty careful here (primitives and synthetic language, while similar in design, are natural for this domain; and the language is human-elicited and program abstractions are learned from just a few low-level primitives) -- but I'd be more convinced if the main findings were also shown to hold in another domain. But again I don't think this is a crucial weakness, there's enough here that that could be followup work.

*Edit after response:* I agree with the authors' points that these would be exciting for future work but are out-of-scope for the submission.

I've updated my score to an 8 (from a 7).

questions: Q1) 123- I was pretty unclear on how meta-learning is used in the agent training. Is a meta-learning algorithm used (it doesn't seem to be; PPO seems to be applied in each individual task), or is it a meta-learning architecture? If the second, does the LSTM model the temporal nature of individual tasks (recurrence is across timesteps for individual agent actions) or the temporal nature of learning across tasks?

Q2) What are 120 LSTM units? Does this mean an LSTM with hidden dimension 120, or 120 different LSTM cells, or 120 unrolled time steps (or something else)?

*Suggestions*

The paper shows similar effects from natural language abstractions & learned program abstractions, but it would be interesting to also investigate whether there is a direct correspondence between these modalities, e.g. using alignment methods similar to those in Andreas et al. 2017, Translating Neuralese or Andreas 2019, Measuring compositionality in representation learning [I feel I should disclaim here that I am not an author of these works :) ].

Claims in the results descriptions / discussion that I'd suggest revising/rewording:
- 275: "improves performance uniformly": if I'm reading the results correctly, autoencoder seems to decrease the performance of the RL baseline on the machine-generated boards
- 302-303: I may have misunderstood, but the library learning isn't grounded in language in the current work, is it (i.e. the paper doesn't do a LAPS-like, Wong et al. approach) -- it's just the RL agent that is grounded?
- Mu et al. does use human-generated language for the Birds dataset. This would also be a good place to cite Andreas et al., Learning with Latent Language
- 312: this is describing the results in Fig 5B, right? there are really only 2 bits in the comparison (relative to the RL agent with no representation prediction, GSP goes up; control goes down), and so this wasn't totally convincing to me


*Minor suggestions*
- line 51: is *a* meta-learning
- 59: define the auxiliary representation here
- 93-95: this seems intuitive, but it could help to give some stats to make it concrete -- what does it mean to respect most of the stat properties?
- 240: it wasn't clear until seeing Fig 6 that "task attribute" can include representations of the language or program.
- 250-252: I confess that having only these citations bugged me a bit, given the long tradition of work on language grounding before 2018. 
- Fig 5 caption: "across modalities" was a bit confusing to me: it's the correlation matrices that are being compared across modalities, right, not the individual representations? (i.e. language embeddings are never compared to program embeddings)
limitations: The discussion of the limitations at the end of the paper was overall thorough and clear. The main things I would suggest addressing are in the weaknesses section above.

One additional point of uncertainty for me was what typifies the "machine prior". It's not at all clear to me that an agent should do better on the control distribution than on the human distribution, particularly given what seem to be differences in the architecture used to produce the distribution and the one used in the RL agent. It would strengthen the paper to explore this more. The nearest neighbor heuristic also encodes a human-like prior in some sense, and since many results are reported relative to this heuristic, I wonder if this could be stacking the cards against the agent.
review_title: null

Meta-review: metareview: The submission explores differences in human and machine inductive biases. Using the task of generating a pattern in a grid, the authors first show that models trained on human inputs generalize better to machine generated inputs than other human inputs, suggesting that models lack the correct inductive bias. However, by exploiting representations of natural language descriptions or programs during training, models can be given a human-like inductive bias. The reviewers agree that this is creative, thought provoking work backed up by thorough experiments, and the paper it is particularly well written. The main area for improvement in future work would be showing that these results would generalize to other, more complex domains.
confidence: Certain
Decision: Accept