Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: *EDIT: Score increased after discussion with authors clarified many concerns raised below*

Summary:
This paper presents an algorithmic approach toward learning Bayes Optimal policies under the uncertainty of the environment. Leveraging meta-learning, the proposed variBAD approximate inference procedure is capable of adapting within the first episode at, what the authors term, meta-test time. 

Comments:
It is my estimation that this paper is well positioned to further current state-of-the-art adaptive RL frameworks or methodologies, whether they are meta-learned, transferred or directly inferred through probabilistic mechanisms. The primary contribution of this paper is in how variBAD learns the variational inference procedure. As noted in the related work section, many contemporary policy learning approaches via variational inference are limited by their construction, selection of prior distributions, etc. The advantage of the proposed methodology is that it is capable of efficiently inferring the current environment and adapting the policy learning procedure accordingly. The experiments successfully compare with relevant baselines and prior approaches. The discussion is well framed in highlighting the benefits and limitations of the proposed methodology in relief to prior approaches. One possible weakness in the experimentation, given how closely RL^2 matches the performance of variBAD, is that the specific contributions of each architectural choice or optimization protocol are unclear. There are a few areas in the paper where the authors suggest that ablating their model in specific ways would recover the core approaches present in RL^2. It would be instructive to see how/if performance degrades or converges toward that of RL^2 as the variBAD methodology is ablated.

The paper is well grounded in the literature, albeit skewed perhaps a bit too far toward recent meta-learning results. This is understandable given the focus of this paper, however there are other approaches that might deserve a mention as they similarly parameterize variation over possible MDPs with some latent variables. Namely, I have in mind a few lines of research such as Contextual MDPs (Hallak, et al, 2015; Jiang, et al, 2017; Dann, et al, 2018; etc.), Successor Features (Barretto, et al, 2016,2017,2019; Lehnert, et al, 2017; etc.)  and HiP-MDPs (Doshi-Velez and Konidaris, 2016; Killian, et al, 2017). In particular use of the HiP-MDP framework, Yao, et al (2018) also use the inferred latent variable used to identify the task to condition the policy. While it's always easy to dig into the rabbit holes of related research and distract from the overall objective of a paper, I thought that there was sufficient overlap with these other lines of research that the authors may find interesting. I do not claim that any one of these additional sources of prior work have been overlooked to the detriment of the current paper, they are offered as merely a suggestion to broaden the author's anchoring in the literature.

Now, some more specific questions about the paper and proposed approach. Further clarity along any of these questions would greatly improve the presentation of the paper as well as further convince me of the paper's suitability for publication.
1) What is the advantage of decoding the entire trajectory? It is well understood that this is advantageous in training as that data is available and allows for better inference of the variational parameters. However, under test conditions where the framework may be operating in environments that lie outside the distribution of MDPs it was trained on, I can imagine that errors in trajectory prediction may compound and throw off the entire inference procedure. The experimental set-up did not allay these concerns as there was no mention for holding out-of-distribution tasks/environments aside and the variation in environments is pretty narrow. 
2) How is the proposed trajectory decoding more stable than model predictive control? Is stability a large consideration for variBAD when exploring? How much can one trust the exploratory actions under variBAD? 
3) The visualization and careful explanation in Section 5.1 of how variBAD executes inference and learning was greatly appreciated. However, are these intuitions valid when extending beyond discrete state and action spaces? Can one make the same claims about the overall approach or procedure in the MuJoCo domains? It was mildly disappointing that a similar explanatory effort was not made in more complex environments. Even an acknowledgement of this being unreasonable would help round out the discussion in Section 5.2.
4) It is not clear what the connection is between the horizon H and the number of rollouts used for evaluation/inference/training. I spent a bit more time than necessary going over and over these items in the paper to where I think that I may understand but I'm still not 100% confident about what is impacted by the number of rollouts used.

review_title: Official Blind Review #4

final_score: rating: 6: Weak Accept
review_content: Summary: 
This paper considers a version of reinforcement learning problem where an unknown prior distribution over Markov decision processes are assumed and the learner can sample from it. After sampling a MDP, a standard reinforcement learning is done. Then the paper investigates the Bayes-optimal strategy for such meta-learning setting. The experiments are done for an artificial maze solving tasks. 

Comments: 
Considering a Bayesian setting of reinforcement learning is sound and well-motivated in a mathematical or statistical sense. On the other hand, I wonder what kind of practical applications motivate such formulation. Unfortunately, I don’t have any examples in mind and the paper also shows some artificial experiments. So, the formulation seems, so far, not to be convincing in a practical sense. 

Another concern in my mind is that the proposed methods are not supported by any theoretical analyses. I think mathematical papers without practical applications are acceptable if they contain strong mathematical analyses. The present paper, however, does not contain such analyses. 

As a summary, I feel that the paper is not strong for theoretical analyses nor practical usefulness, and thus further investigation for either side is necessary.  


Comments after Rebuttal:
I modified my score according to authors' comments. 
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: This paper presents a new deep reinforcement learning method that can efficiently trade-off exploration and exploitation. An optimal policy for this trade-off can be solved under the Bayesian-adaptive MDP framework, but in practice, the computation is often intractable. To solve the challenge and approximate a Bayesian-optimal policy, the proposed method VariBAD combines meta-learning, variational inference, and bayesian RL. Specifically, the algorithm learns latent representations of task embeddings and performs tractable approximate inference by optimizing a tractable lower bound of the objective.

The paper is well-written and easy to follow. The combination of meta-learning, variational inference and BAMDP is a clear and neat way to approximate Bayes-optimal policy. The idea also sounds practical for RL as it can approximately solve larger tasks with unknown priors. Experiments on Gridworld and Mujoco show the effectiveness of the proposed method. On Gridworld the performance of the proposed algorithm is close to the performance of the Bayes-optimal policy.

One concern for this paper is the level of novelty, as each major component of the proposed solution has been explored quite extensively in the existing literature (as mentioned in the related work section). 

In addition, since comparing many existing Bayesian RL methods, VariBAD meta-learns the inference procedure. This can add additional computation complexity to Bayesian RL, which is not explained or mentioned in neither the method part nor the experiment. I hope the authors can add some discussions on this aspects
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: The proposed method represents a single MDP using a learned, low-dimensional stochastic latent variable. On these grounds, given a set of tasks sampled from a distribution, the method jointly trains: (1) a variational auto-encoder that can infer the posterior distribution over the postulated latent variable when it encounters a new task while interacting with the environment, and (2) a policy that conditions on this posterior distribution over the MDP embeddings, and thus learns how to trade off exploration and exploitation when selecting actions.

Such variational inference arguments for transfer learning in the context of MDPs are not new. The authors have not made a good job reviewing the related literature. Most importantly, their experimental evaluations lack substantial comparison to such related methods. This is totally disappointing.
review_title: Official Blind Review #1

Meta-review: This paper considers the problem of transfer learning among families of MDP, and proposes a variational Bayesian approach to learn a probabilistic model of a new problem drawn from the same distribution as previous tasks, which is then leveraged during action selection. 

After discussion, the three respondent reviewers converged to the opinion that the paper is novel and interesting, and well evaluated. (Reviewer 1 never responded to any questions the authors or me, so I have disregarded their review.) I am therefore recommending an accept.
Decision: Accept (Poster)