Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper focuses on a very interesting problem, that of pre-training deep RL solutions from observational data.   The specific angle selected for tackling this problem is through meta-learning, where a set of Q-functions / policies are learned during pre-training, and during testing the network identifies the training set MDP matching the data to extract a transferable solution.

The main strength of the paper is to draw attention to the issue of pre-training in RL, which is much less studied than in supervised learning, where it has been shown to have tremendous impact.   The paper also provides reasonable coverage of a large amount of related work.

Unfortunately I really struggled (despite careful reading) to understand several aspects of the proposed methods.   The training of function f() is not clearly explained; is this done as per Sec.2.4?  What is the loss function for this?  Is it done end-to-end as per the pipeline in Fig.1 (right), so using a gradient propagated back from Q?   What is the purpose of Proposition 1?  The more interesting point seems to be that solutions can “converge to a degenerate solution”, but this is not formally defined (i.e. how do you assess degeneracy, and how is that information used?)  Furthermore Proposition 1 seems limited to discrete state/action spaces. Is this the case for TIME in general?  The results on Mujoco suggest not.  Regarding the second phase of the pipeline, it is briefly mentioned that “P has low capacity” (bottom of p.4), but this is not explained further.  Is this due to a generalization issue, or a computational issue?  Why would P be low capacity but not E?  How does this actually impact the implementation?

As a higher-level comment: is it really necessary (preferable) to infer the identity of a specific train MDP (using the function f)?  This is used as a premise in this work, but I am not convinced this is desirable (for good generalization) or scalable (in the case of several observed meta-train MDPs).   What is the advantage of proceeding in this way?  Much of the work on pre-training in supervised learning just exposes the learner to large amounts of observational data to pre-condition the solution.

Finally, I have some concerns with the results as presented in the paper.  There are some details lacking, for example how specifically are the meta-test MDPs chosen for the Mujoco experiments?  How similar/different from the meta-train MDPs?  This is an issue because in Sec.5.1 the meta-test MDPs are chosen to coincide with meta-train MDPs.  So I am definitely interested in seeing how well the method actually generalizes to unseen MDPs, so need more detail on this part of the experiment.   I would also like to see a few additional naïve baselines.  First, what is the result if you do the pre-training as specified, and then at test time you randomly sample one of the pre-trained MDPs (rather than use the identification function).  Second, what is the result if you put all the meta-train data into a single batch, train a solution with SAC, then use this as a pre-trained solution (rather than the current “SAC trained from scratch”), allowing more training at test time.   Both these are useful sanity checks to verify the effectiveness of the proposed approach.

============
Post-rebuttal comments:

1.  My question, as stated in the review is: " how specifically are the meta-test MDPs chosen for the Mujoco experiments".  I read that they are tested on unseen MDPs.  I want to know how those unseen MDPs are selected / specified, and again as per my review: "How similar/different from the meta-train MDPs?"

2.  You should entertain the possibility that perhaps Sec.3 is not as clear as you think it is.  For example the MDP (S,A,\hat{T}, \hat{R}, \gama) is not defined as "the wrong MDP" - which I assume is what you mean by degenerate solution?  

3.  I would like to know what are the results from the 2 naive baselines I described, as good sanity check.

In general, the rebuttal is intended to be a conversation to clarify understanding of the work. It is insulting to the reviewer to say that they did not read the paper carefully when they indicate they did.  It is much more productive to assume that many of your other (future) readers might have the same need for clarification, and you should be thankful for help provided by the reviewers to achieve this.
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper studies the meta-RL problem in the off-policy, batch learning setting. Batch-RL is the setting in which a policy is learned entirely offline, that is, without interaction with the environment and given only trajectories collected by some policy. Compared to RL, Meta-RL involves the additional challenge of task-inference; the goal of Meta-RL is to train a policy that can generalize to a distribution of tasks (i.e. a distribution of MDPs), without actually being given a description of the task (unlike contextual multi-task policy learning). A simple approach for solving the meta-RL problem thus might be to first perform task inference by encoding data from some task into a task description, and then condition a contextual policy on this task description. 

The authors state that the straight-forward application of such an idea in the Batch-RL setting fails due to an issue they term the MDP “misidentification” problem, wherein having multiple tasks in a single batch results in confusion between tasks. This issue really only arises in the setting where task inference is jointly learned with the multi-tasking contextual policy. Thus, they propose a stage-wise algorithm wherein first 1) the N tasks in some dataset of trajectories are learned by N separate policies, and then 2) these N separate policies are distilled into a single master policy, wherein the subpolicy enacted by the master policy is modulated by some sort of task description. The distillation procedure thus involves task description (i.e. mapping from task data to a task embedding) and supervised learning (i.e. imitation) of each sub policy when conditioned on respective task embedding.

* Pros
	* Demonstrates the effectiveness of a straightforward stage-wise approach for off-policy meta-RL (albeit without comparison to alternatives)
	* The method seems to perform well on the tasks considered, approaching the level of performance of SOTA model-free algorithms.
	* The approach is general insofar as it could presumably be used with other batch-RL methods (for first stage of algorithm), or even non-batch RL methods. In this sense, the main contribution might be seen as an approach for distilling multiple policies into a single policy in a way that allows for interpolating between them.
	
* Cons
	* Technical novelty: while they address a problem that has not yet seen much attention, their solution is combination existing solutions. I say this because I am not fully convinced that the more novel aspects of their distillation procedure (i.e. the auxiliary task) are absolutely necessary. I would be willing to change my stance on this, given evidence.
		* Ablations of i.e. the auxiliary task would help to clarify this. In all tasks considered, the transition function T does not change. Therefore, the function composite P(E(s_j, a_j)) should actually not learn any task-specific information when predicting s’_j. If the method works without the auxiliary loss, it is very similar to a stagewise version of PEARL, with Batch-RL. 
	* Limited comparisons to relevant alternatives, simple baselines.	 Do not compare to anything other than SAC.
	* Only tasks considered are pointmass, hopper, and half-cheetah; other work (i.e. PEARL) has also been demonstrated on Ant in Mujoco. 
	* Argues that it is more stable than variants that involve interaction with the environment / use critic loss for MDP identification, which is a somewhat unfair since in their case 1) they assume they have data that is good enough to learn a good policy in the batch-RL setting, and 2) the stability is by virtue of the fact that they do things in a stage-wise manner because batch-RL works, i.e. they are doing supervised learning without having to bootstrap.
	* Not sure if the use of term “pre-train” is appropriate, insofar as the test tasks are assumed to come from the same distribution as training tasks. It seems to be more about Batch-Meta-RL. 
	* Not much attention given to attempts to solve the MDP mis-identification problem with simple solutions like giving side-information at training time (i.e. task-ID). 

* Would be helpful to see:
	* More baselines
		* A simple task classifier. This would basically do a nearest neighbor lookup over the training tasks (given a test task), but this might perform well under the reward function.
		* Comparison to something like PEARL, or some method that does involve interaction with the environment -> this would help shed light on whether interaction is necessary for task inference in the the tasks they consider, and if so, such methods would in some sense be oracles.
	* Ablations
		* Remove auxiliary forward prediction loss: if it works without auxiliary loss, this is very similar to a stagewise version of PEARL, with Batch-RL.
		* How much data per task is needed?
	* Harder tasks, where the method can’t approach SAC (as one would expect for sufficiently challenging tasks)
	* Experiment where they study zero-shot generalization by considering disjoint parts of task space
		* Claim zero-shot generalization but in this case they study tasks where resets are not crucial for exploration needed for task inference
		
* Minor comments:
	* I found the discussion about inductive bias in RL at the end of section 3 (last few sentences of last paragraph) to be a bit vague.

I've given a weak reject mainly because 1) auxiliary loss has not been experimentally shown to be crucial and therefore the technical novelty may be relatively limited, and 2) more comparisons are needed, to alternatives or other baselines.  I would be willing to change my decision on this, given supporting evidence.
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper studies batch meta learning, i.e. the problem of using a fixed experience from past tasks to learn a policy which can quickly adapt to a new related task. The proposed method combines the techniques of Fujimoto et al. (2018) for stabilizing batch off-policy learning with ideas from Rakelly et al. (2019) for learning a set invariant task embedding using task-specific datasets of transitions. They learn task-specific Q-values which are then distilled into a new Q function which is conditioned on the task embedding instead of the task ID. The embedding is further shaped using a next-state prediction auxiliary loss. The algorithmic ideas feel a bit too incremental and the experimental evaluation could be stronger--I'd recommend trying the method on more complicated environments and including ablation studies.

Specific comments:
1. I disagree that the cheetah and hopper environments are "challenging"--they're one of the simplest MuJoCo environments. 
2. The problem of adapting to run at a specific speed when the meta-learner observes the dense rewards is actually not a meta learning problem because the meta learner can uniquely identify the target speed from a single transition. This is because the current speed is part of the observation, and so given the value of the dense reward at this state, it is simple to calculate the target speed. Hence these environments are effectively the same as directly giving the agent the target speed as an input. Given this interpretation, I'm not sure what is "meta" about this environment. The problem then reduces to the question of whether the agent can generalize from the 16 or 29 training tasks. That this should be the case is not surprising considering the one-dimensional nature of the task space.
3. It would also be useful to see some ablations. For example, is the auxiliary prediction task necessary? Would it be possible to side step the distillation process and directly learn Q_S from the buffer as done e.g. in Rakelly et al. (2019)? Could you show some data that the corrections from Fujimoto et al. (2018) are important in the batch setting?

------------------------------------------------------------------------------------------------------------
Thanks for your comments. I still think this is too incremental, and my concerns regarding the environments and using the dense reward as a feature which identifies the task haven't changed and so I'm keeping my score as is.
review_title: Official Blind Review #3

Meta-review: The reviewers reached a unanimous consensus that the paper could not be accepted for publication in its current form. There were a number of concerns raised regarding (1) the clarity of the writing; (2) the comparisons, especially to prior work; (3) the details of the experimental setup.
Decision: Reject