Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: 
Summary
---

Children generalize to new sights that combine known perceptual elements.
Children generalize to new instances of known abstract concepts like order and number.
Children know what actions they can take in new scenarios, because those actions have been available in similar contexts.
Machines should be able to generalize in the same ways.

This paper proposes a new environment, HAMLA, where machines are tested on their
ability to generalize in all of these ways.
Agents navigate through a maze to a goal location using only carefully designed
signals extracted from carefully constructed mazes.
They must learn perception (MNIST digits, color, location), abstract concepts
(number, order), affordances (move up/down/left/right as available), and
efficient exploration strategies at once, like humans seem to be able to.

Evaluation is dynamic, estimating what an agent already knows then generating
new test instances that test the model on something slightly outside what it knows.
There is no static test set or test environment.

TD3 is used to train various agents based on different NN architectures
that incorporate more or less structure.
Agents often fail to navigate to the goal and do so efficiently in scenarios basically similar to those it was successful at.
Failure happens more often as the generalization gap becomes larger and when the agents must also learn perception in addition to concepts and exploration.


Strengths
---

The motivation is ambitious, interesting, and relevant. It makes sense. It pulls strongly from cognitive science. Explicitly attacking multiple specific modes of generalization at once is an interesting direction.

This could establish a new baseline task that tests multiple kinds of generalization in a toy manner.

The dynamically generated evaluation strategy is new and interesting. It may make it harder to compare performance across models, but clearer about how well a single model is actually performing. That could inspire a significant shift in evaluation methodology.

The paper is highly sylized and polished.


Weaknesses
---

1) The paper is hard to understand in just 8 pages.

The writing is so dense and so many details are left to the 20 page appendix that it is hard to understand the approach or experiments at more than a very high level by reading just the main 8 pages of the paper.

The main example here is the notation. Much of it is non-standard. It is also used frequently and most definitions are left for the appendix.


2) Novelty relative to some related work isn't clear.

How does this compare to point-goal nav? [1] I think the test procedure and available senses make it different, but it's still fundamentally navigating to a goal location. Will scaling training help solve this problem as evaluated by the proposed metric?

[1]: Wijmans, Erik et al. “DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames.” ICLR (2020).


3) Impact may be limited by difficulty. Tackling this whole problem may be too difficult right now. All approaches completely fail to solve the full problem including vision, as indicated by the bottom right of table 1, which is filled with 0s. If progress is limited to the symbolic setting then the problem is significantly less interesting.


Preliminary Evaluation
---

At the moment the paper is not very clear. That makes it hard to evaluate the quality of the experiments. The quality and novelty of the motivation is high, being fairly novel and interesting. Its significance is highly uncertain because of the paper's clarity and potential difficulty. The paper might be significant as either 1) a central reference for applying some cognitive science concepts to AI, 2) a benchmark that spurs new agent designs, or 3) inspiration for designing new evaluation metrics.

My main uncertainty in this evaluation is because I haven't understood the paper in its full 30 pages of depth.


review_title: Dense, but well motivated

final_score: rating: 6: Marginally above acceptance threshold
review_content: ---Post rebuttal---

Thank you for the detailed response. Overall, I think the proposed work provides a valuable benchmark for testing generalization ability of RL agents. However, I agree with R3 regarding the writing being dense/difficult to follow. I keep my rating unchanged (Weak Accept).

----

This work proposes a benchmark for evaluating the task-solving capabilities of agents on three levels: perceptual, conceptual, and algorithmic. The tasks are procedurally-generated contextual 2D gridworld environments.

I believe there is a lack of RL benchmarks on evaluating the agent’s understanding of object affordances, so this is a useful benchmark for the RL community.

Clarity: The paper is well-written and well-motivated.

Suggestions:

1. The paper provides empirical evaluations of TD3 with various encoder/decoder architectures. However, there does not seem to be evaluations of model-based/planning methods, despite the task requiring planning & reasoning. I think the comparison of model-free vs. model-based on this benchmark would be valuable.

2. For future work, I think it would be valuable to add continuous control to the benchmark tasks for more “humanlike abstraction learning”. For example, use locomotion actions instead of gridworld actions; or a robotic arm learning to play a logical puzzle game.

3. The related works section (Appendix H) can also add prior work on visual semantic navigation, which connects visual and semantic understanding with control:
[1] VIsual Semantic Navigation using Scene Priors https://arxiv.org/pdf/1810.06543.pdf
[2] Embodied Multimodal Multitask Learning https://arxiv.org/pdf/1902.01385.pdf

review_title: Useful benchmark that evaluates the task-solving capabilities of agents using the notion of affordances

final_score: rating: 5: Marginally below acceptance threshold
review_content: Disclaimer: Apologies I have very little background in the area of this paper and should have probably opted out. However I did read the text twice with interest. 

The authors propose a task HALMA which involves a grid-world maze that is partially observed, and includes visual panels that contain conceptual reasoning tasks which the agent must solve to find an optimal path. The aim of this environment is to test three proposed 'levels of generalisation': 'perceptual', 'conceptual' and 'algorithmic'. The authors propose several tests for generalisation: 'semantic', 'affordance', 'analogy'. The authors propose a dynamic manner of evaluating the agents by showing them problems that they appear to not yet have demonstrably understood. The authors conclude that existing agents are woefully inadequate at these generalisation tests and invite researchers to approach this novel task. 

I found the paper had an enjoyable positive style of discourse and built up the problem area quite nicely in the introduction, but was unfortunately very verbose and many important details were relegated to the appendix. For example the actual algorithm used to dynamically produce test-set elements was relegated to Appendix D --- however this seemed to be one of the central contributions to the paper. Furthermore the environment of HALMA was not very succinctly described. Possibly this paper has too much information for a conference submission and should fit better in a journal, otherwise it should be edited down to make room for a succinct description of the environment, and the evaluation approaches *in the main text*.

I was curious to what effect many of the research findings depended on the arbitrary size of the training set; and would have been interested to see some generalisation metrics reported as a function of training set size (100 mazes seems very small). I would have also liked to see a human baseline and then to have had human-normalised scores.
review_title: Naive review

final_score: rating: 7: Good paper, accept
review_content: ##########################################################################

Summary:

The paper introduces a new benchmark which measures agent reasoning abilities and their generalization of 3 kinds: perceptual, conceptual and algorithmic. The paper extensively motivates this benchmark and shows experiments with training RL agents on it. 



##########################################################################

Reasons for score:

New benchmarks for measuring intelligence are important for driving the field. This one seems interesting.


##########################################################################

Pros:

1. New benchmark which tries to provide a comprehensive measure of generalization, which is a very important topic.
2. Paper is nicely written and illustrated - pleasure to read!
3. Benchmark/code will be published upon acceptance, so the whole community will be able to profit from it.

##########################################################################

Cons:

1. The history of artificial domains as AGI playgrounds has taught us a few bitter lessons. To give one very recent example, the winner of Abstraction and Reasoning Challenge (yeah, that one proposed by Francois Chollet to measure general intelligence https://arxiv.org/abs/1911.01547) wrote "Unfortunately, I don't feel like my solution itself brings us closer to AGI." https://www.kaggle.com/c/abstraction-and-reasoning-challenge/discussion/154597. The solution was some handcrafted search algorithm, no machine learning. To be fair, the challenge is very far from being solved - and yet the winning solution from 914 teams didn't teach us much. Is there anything in the design of HALMA which we expect will protect it from similar problems?

2. No human baselines are given. How do we know that our agents achieved human performance?

##########################################################################

Questions during rebuttal period: 

I would be curious to know the authors' opinion on Cons above.

#########################################################################

Minor suggestions and typos: 

(1) affordnace -> affordance
review_title: Nice new benchmark

Meta-review: This paper proposes a new task domain for learning-based AI agents, HALMA, a game that is designed to bring together multiple areas of research in AI. Perception, in the form of recognition of MNIST digits, learning mathematics - in the form of arithmetic operations on the natural numbers, and navigation and planning. When combined into a game, these elements are argued to require various important properties of human cognition, such as abstraction, analogy and affordance. 

I commend the ambitious goals of this work, and its multidisciplinary motivations. I believe the benchmark can indeed eventually be an important challenge for the community. I think that the dynamic testing aspect is particularly interesting, where the environment produces trials designed to go beyond the agent's experience to that point. However, having considered the views of the reviewers and read the (main body) paper entirely myself, I unfortunately cannot recommend acceptance in its current form. 

The main reason for the decision is simply that it is prohibitively challenging for me to grasp exactly how the game actually works after a thorough reading and considerable thought. The authors spend two pages motivating the approach with (arguably excessively grandiose) allusions to Marr's levels of analysis, Gibson's affordances, Holyoak's analogy and various other famous works from the history of AI and philosophy of mind; as well as to the board game HALMA. But as a reader I can't myself start to make any of these connections because the game proposed by the authors has not been explained to me! It is finally introduced on the fourth page - with reference to Figure 2 which is too small to consult and very hard to interpret. After consulting the appendix (where the idea is a bit clearer) I was able to decipher the way in which the numbers related to the maze itself, but was (and am still) unclear on the actions available to the agent. These are explained as follows: 

""The direction set is t , , , u. The primitive action set, in terms of the numberof moves, is t , , , u; this design of primitive numbers with a maximum of three aligns withthe doctrine of core knowledge in developmental psychology (Feigenson & Carey, 2003; Dehaene,2011). If an option is selected, consecutive hops as in Halma are simulated; all observations fromintermediate states will be skipped, and only the observation of the final state is provided. A movewould fail if a wall stops the agent, leaving the agent’s position unchanged; failure moves bringpenalties to the agent. The agent would receive a positive reward when reaching the goal""

From reading this I am left with the following questions: 
- Are directions primitive actions? 
- How can a primitive action also be a number of moves? 
- What does it mean to select an option? 
- Can I select an option and an action at the same timestep? 

Most importantly, I still don't really know how the game works. 

This example is intended to illustrate the difficulty faced by readers of this paper in general.  

I note that the reviewers awarded this work scores that place it on the borderline for acceptance, but with consistently low confidence. On consulting with the reviewers it is clear that this is not because they lack expertise but because they too did not understand the full details of how this domain/task works. This is also clear from the lack of detail in their reviews; only reviewer 3 engaged with any of the details of the task itself. 

To summarise, I think there is potentially a very interesting and important contribution in this dataset. However, the work will only have impact in the community if it can be understood and adopted after a single read of the paper. I therefore recommend that the authors resubmit this work to a different venue taking account of the following: 

- Explain how the game works *then* connect it to the literature on human learning *not* vice versa (from the concrete to the abstract)
- Be very concrete, perhaps guide the reader through a single particular episode explaining the observations available to the agent and the options open to it at each important point-
 Get to the point of your contribution. Tenuous connections to cogsci etc can go in the discussion
- Make all diagrams and illustrations extremely simple to interpret and large enough to easily read
- Avoid use of subjective adjectives, and particularly describing one's own contributions as "ingenious solutions" and "impeccable"
- Avoid rhetorical flourishes and latin
- Submission to a journal may allow the authors greater space to draw the desired connections to disparate fields without compromising on readability or exposition of their methods
Decision: Reject