Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: Overall the paper is written quite well and addresses a relevant topic. The experiments are thorough and make sense given the research question.  The paper furthermore provide additional ablation studies and does a good job in model analysis.

There are some critical points I have about the paper summarized below:


Introduction:
The 3 categories to solve POMDPs appear artificial. All 3  (whether using a window directly, using a RNN or building a surrogate model) aim to transform the POMDP to an MDP by taking history of observations (either explicit, or implicit).

Figure 1 is too  complicated for an illustrative figure  and the presentation and clarity needs to be improved.


Method:
The idea is sound and described well. However, some design decisions appear ad-hoc  (only justified by empirical observations). For instance, the authors argue it is better to keep 2 models (the "first-impression" and "keep-learning" model) and show an ablation study in appendix C.  
Now, one could wonder if the proposed method would still perform better than the baselines if with, say, the just using a single VRM.   The improvement compared to SAC-LSTM is not very large except in a few cases, so possibly just using the VRM would perform no better.

This could then defeat the theoretical argument of the method that "[..] the actor network of SLAC
did not take advantage of the latent variable, but instead used some steps of raw observations as input,
which creates problems in achieving long-term memorization of reward-related state-transitions."


Reference that should be included and possibly compared to:
[1] Watter, Manuel, et al. "Embed to control: A locally linear latent dynamics model for control from raw images." Advances in neural information processing systems. 2015.


Questions:
1. What is the scalability, such as wall-clock time and #parameters  of the approach compared to the baselines? (When using 2 VRMs the # of parameters is doubled)
2. Why the need to input the original observation x (and not just the latent representation) into the RL controller?

review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: The paper proposes that for partially observable reinforcement learning tasks it might be simpler to decompose the problem in two parts: a recurrent world model and a feedforward agent, as opposed to using just a recurrent agent. Intuitively the decomposition makes sense, though it's not very clear to me that the problems encountered training a recurrent agent should be dramatically simpler when training a recurrent world model instead.

For the world model the paper proposes using a variational recurrent state-transition model, which is essentially a VRNN which also conditions on the actions. It is not clear to me that this is easier to learn than a recurrent agent because the probability distribution of the data used to train the modified VRNN is highly dependent on the current policy in the same way that the non-iid training data makes it tricky for the RNN policy models to converge.

While the experimental results seem to show the new algorithm outperforming the existing ones over a large set of random seeds the paper does not specify how the fixed hyperparameters for all models were chosen, leaving open the possibility that different hyperparameter settings would have shown reversals in the experimental results. It's also not clear that the complexity of the alternate models was adequately accounted for (specially since two VRNNs are required to match the performance of a single RNN agent). That said the incompleteness of the experimental results is my only reservation against accepting this paper.


----

The author response somewhat addressed my major concern, which was the lack of hyperparameter tuning in the baselines, as well as clarifying some of the other questions I had. Given that my score has been updated.
review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: * Summary

The work considers reinforcement learning in partially observed environments, targeting in particular the case when the agent needs to remember some important information over a longer period of time. 

The work gives an interesting connection between variational recurrent neural networks and reinforcement learning. The paper is well written and technically sound. The method achieves improved performance in several (simpler) benchmark problems in comparison to strong baselines.

* Detailed discussion

The work allocates latent variables to describe unobserved part of the state and learns the state transition probability model (the action-dependent generator (10)). Learning these state transitions becomes an unsupervised learning task, for which the lower bound on the likelihood of the observed data is optimized. This learning of the state transition model is actually similar to classical model-based RL. The policy and value functions are then learned on top of this model using SoTA techniques. From this perspective I would say this is a classical approach to partially observed reinforcement learning with advanced models for transition matrix and value and policy functions.

The work combines together variational recurrent neural network (VRNN) and soft actor critic (SAC) models into a rather advanced system for reinforcement learning. These parts are somewhat extended to fit together, but it is not clear whether all components are needed. For example, why z and d need to be distinguished, cannot they be combined into one latent variable “zd” in order to simplify the number of connections? Or I am missing some important property necessary for the inference?

In (7),(8) are the expectations over the same state / actions or different? The notation is not clear about it.

After (2), given x_t and d_{t-1} instead of d_t?

The experiments are conducted in the setting that the observations were all relevant measurements for the agent (positions, velocities). From Fig 7 RoboscoolHooper (close loop) we see that actually all measurements can be well predicted by generative model up to 8 steps ahead (in dim 2 that stays zero the noise is amplified). This is a nice and desirable property and is the consequence of the generative modeling and the marginal likelihood ELBO optimization. However the paper seem to argue (Appendix E) that the algorithm does not require the model to make accurate predictions and relies on the encoding capacity, which I find confusing. Furthermore, it would be interesting to see how this approach should scale to the setting when there are more sensory inputs, some less relevant than the other. In the end, is a good generative model to predict the future observation needed or not needed in this approach?

The exploration seems to be addressed just by randomizing the policy,  are there any better options?

review_title: Official Blind Review #2

Meta-review: The authors propose to decompose control in a POMDP into learning a model of the environment (via a VRNN) and learning a feed-forward policy that has access to both the environment and environment model. They argue that learning the recurrent environment model is easier than learning a recurrent policy. They demonstrate improved performance over existing state-of-the-art approaches on several PO tasks.

Reviewers found the motivation for the proposed approach convincing and the experimental results proved the effectiveness of the method. The authors response resolved reviewers concerns, so as a result, I recommend acceptance.
Decision: Accept (Poster)