Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes SVQN, an algorithm for POMDPs based on the soft Q-learning framework which uses recurrent neural networks to capture historical information for the latent state inference. In order to obtain this formulation, the author first derive the variational bound for POMDPs and then present a practical algorithm.

The key idea of the paper is to replace DQN with Soft Q-learning that already demonstrated better performance on a variety of tasks. This seems to be an obvious extension of DRQNs (Hausknecht & Stone, 2015) even though it did not appear in the literature.

The authors evaluate the final algorithm on a set of ALE and DoomViz tasks. The algorithm outperforms the previous methods, in particular, DRQNs. The set of tasks and prior methods is adequate.

Overall, the contribution of the paper is not significant enough to be accepted to ICLR.

review_title: Official Blind Review #1

final_score: rating: 8: Accept
review_content: This paper proposes a new sequential model-free Q-learning methodology for POMDPs that relies on variational autoencoders to represent the hidden state. The approach is generic, well-motivated and has  clear applicability in the presence of partial observability. The idea is to create a joint model for optimizing the hidden-state inference and planning jointly. For that reason variational inference is used to optimize the ELBO objective in this particular setting. All this is combined with a recurrent architecture that makes the whole process feasible and efficient.

The work is novel and it comes with the theoretical derivation of a variational lower bound for POMDPs in general. This intuition is exploited to create a VAE based recurrent architecture. One motivation comes from maximal entropy reinforcement learning (MERL), but which has the ad hoc objective of maximizing the policy entropy. On the other hand SVQN optimizes both a variational approximation of the policy and that of the hidden state. Here the rest terms of the ELBO objective can be approximated generatively and some of them are conditioned on the previous state which calls for a recurrent architecture. The other parts are modeled by a VAE.

The paper also explores two different recurrent models in this context: GRU and LSTM are both evaluated. Besides the nice theoretical derivation the paper presents compelling evidence by comparing this approach to competing approaches on four games of the flickering ATARI benchmark and outperforming the baselines significantly. Also both the GRU and LSTM version outperforms the baseline methods on various tasks of the VIZDoom benchmark as well.

In general, I find that this well written paper presents a significant progress in modelling POMDPS in a model-free manner with nice theoretical justification and compelling empirical evidence.





review_title: Official Blind Review #3

Meta-review: The paper proposes a novel model-free solution to POMDPs, which proposes a unified graphical model for hidden state inference and max entropy RL. The method is principled and provides good empirical results on a set of experiments that relatively comprehensive. I would have liked to see more POMDP tasks instead of Atari, but the results are good. Overall this is good work.
Decision: Accept (Poster)