Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes a framework (Scoring-Aggregating-Planning (SAP)) for learning task-agnostic priors that allow generalization to new tasks without finetuning. The motivation for this is very clear - humans can perform much better than machines in zero-shot conditions because humans have learned priors about objects, semantics, physics, etc. This is achieved by learning a scoring function based on the final reward and a self-supervised learned dynamics model.

Overall, the paper is very clear and easy to follow.
The presented task is realistic and important, and the paper seems to address it in a reasonable approach. 
However, the evaluation seems lacking to me - the evaluation convinced me that SAP works, but I am not convinced that it works better than existing approaches (see below), and especially did not convince me that it is better in the zero-shot test environment.
The (anonymized) website contains nice videos that support the submission.

Questions for the authors:

1. Page 3, 3rd paragraph of Section 3: the paper says that "The proposed formulation requires much less information and thus more realistic and feasible" - I agree that this is more realistic, but is it really more feasible? The requirement of much less information makes the proposed formulation much more sparse.

2. A basic assumption in the SAP framework is that a local region score is a sum of all the sub-regions. As phrased in the paper: "in the physical world, there is usually some level of rotational or transnational invariance". I'm not sure that this assumption makes sense neither in the Mario case or in other tasks, e.g., robotics. Doesn't it matter if you have a "turtle" right in front of you (which means that the turtle is going to hit you), or below you (which means that you are going hit the turtle)?

3. A question about the planning phase - page 5 says: "We select the action sequence that gives us the best-aggregated score and execute the first action". Do you select the entire sequence of actions in the new environment in advance? Can the agent observe the new state after every action, and decide on the next action based on the actual step that the action has reached, rather than on the state that was approximated in advance?
In other words - what happens if the first action in the new test environment yields an unexpected state, that was not predicted well by the dynamics model; does the agent continue on the initial planned trajectory (that ignores the "surprise"), or does it compute its next action based on the unexpected state?
 
4. Experiments: in Gridworld and Mario - are there any stronger baselines in the literature, or reductions of known baselines to the zero-shot scenario? Are the chosen "Human Priors", BC-random and BC-SAP just strawmen? 
Since the main goal of this paper is the zero-shot task, what would convince me is a state-of-the-art model that does possibly *better than SAP on the training level*, but *worse than SAP in generalizing to the new level*. Additionally, are there other baselines that specifically address the zero-shot task in the literature?

Minor (did not impact score):
Page 2, 1st paragraph: "... we show that how an intelligent agent"...
Page 3, 3rd paragraph: "... in model-free RL problem" - missing an "a" or "problem*s*"?
Page 3, 3rd paragraph: ". Model based method ..." - missing an "a" as well?
Page 4, 1st paragraph:: "... utilizing the to get the ..."
Page 4, last row: missing a dot after the loss equation, before the word "In".
Page 7, Table 1: "BC-random" is called "BC-data" in the text. Aren't they the same thing?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper describes a method that aims to learn task-agnostic priors for zero-shot generalization. The main idea is to employ the following modeling approach on top of the model-based RL framework: a local convolution network is used to compute a score for each local state action pair, and then another network is used to aggregate all the scores. While the problem being studied is important and the experimental results seem positive, there are a few concerns.

First, the baselines presented in the experiments are relatively weak. In Related Work, the authors discuss the differences between the proposed method and the related methods, but few of the related methods are used as baselines for comparison with the proposed method. Moreover, the experiments are quite insufficient in terms of ablating different components of the proposed methods.

Second, essentially the proposed method is trying to solve the zero-shot generalization by parameter initialization; a model is pretrained on related tasks and used as initializations for target tasks. The authors claim that it is different from prior work mainly because of the neural architecture that deals with sparse rewards via score aggregation. While the proposed architecture might be more suitable for solving tasks with sparse rewards, it is not intuitive whether it has something to do with learning zero-shot generalization. And apparently, the method will also rely on the similarity between the pretrained task and the target task, and such a scope constraint is not discussed in the paper. In other words, I'm not quite sure a better architecture is fundamental progress towards zero-shot RL.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: I am not from this area and don't know much about reinforcement learning.

The paper discusses zero shot generalization (adaptation) into new environments. The authors propose an approach and then show results on Grid-World, Super Mario Bros, and 3D Robotics. 

In the training environment E1 = (S, A, p) the algorithm sees a bank of exploratory trajectories \tau_i = {(s_t, a_t)}_{t=1}^{T} but not rewards. The authors  then say that algorithm is tested on the test environment E2. They " propose to only inform the new task per trajectory terminal evaluation r(τ ) in E1" to give the training signal (where r is the reward).

I am a bit confused by this setting. The model never sees any rewards for E1 but it does see rewards for E2? How is this zero shot?

The authors then propose their approach, I wish some of it had been described more rigorously with math (e.g. the loss etc.) so it was easier to understand for people not in the domain and familiar with some of the terminology. 

Empirically the authors show results for 3 datasets and this seems thorough. 
review_title: Official Blind Review #1

Meta-review: The paper proposes an algorithm for zero-shot generalization in RL via learning a scoring a function from.

The reviewers had mixed feelings, and many were not from the area. A shared theme was doubts about the significance of the experimental setting, and also the generality of the approach.

As this is my field, I read the paper, and recommend rejection at this time. The proposed method is quite laborious and requires quite a bit of assumptions on the environments to work, as well as fine tuning parameters for each considered task (number of regions, etc). I also agree that the evaluation is not convincing -- stronger baselines need to be considered and the experiments to better address the zero-shot transfer aspect that the paper is motivated by. I encourage the authors to take the review feedback into account and submit a future version to another venue.
Decision: Reject