Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: The paper focuses on the problem learning state representations that can effectively capture historical information
in POMDPs. Specifically, the paper proposes an alternative approach to using RNN's for capturing such history - the authors adopt a variant of recently proposed Invariant Information Clustering approach to discover important events in past observation and then use these events to learn a probability distribution over observations to represent the state information. The goal here is to address the unstable and inefficient training issues associated with RNN based architectures. The authors validate their approach with experiments on seven tasks on Atari 57 benchmark and Obstacle Tower, both with discrete action space and compare their performance against both original and RNN versions of PPO. 

The paper addresses an interesting problem to help learning better state representation in absence of  complete information about the environment. The approach of using IIC clustering (or any clustering approach) for learning state representations is novel. The overall goal of replacing RNN based methods in order to achieve stable, efficient learning in presence of budget constraints is very useful and hence this approach is potentially a good step in that direction. Although not adequate, the results in figure 3 provides good insight into effectiveness of the method in making training easier. 

However, I am inclining to reject this paper for the following reasons:
(1) The motivation for using proposed clustering approach for history representation is neither clear  and nor well exposed. MI based methods are inherently difficult to learn and hence this approach needs  rigorous analysis on why it works when it does and how it fails.
(2) The paper fails to position the new approach in comparison to related works both in discussions and experiments.
(3) The experiments are very limited in nature and fails to demonstrate the efficacy of the proposed approach effectively. Further, the key contribution focusing on learning effective representations under constrained budget is not adequately tested.

Major concerns:

Motivation
-------------
- It is not clear how the proposed clustering mechanism to discover events allows to successfully capture information that an RNN based approach does. For instance, RNN helps to capture long-term dependencies but it is very hard to interpret the proposed model from that aspect. Also, why is this particular clustering approach (Invariant Information Clustering) chosen? The motivation for using this approach is not clear and overall combination appears adhoc.
- Further, RNN based methods can retain order information in sequential history of observations. However,  this method appears to not consider that information. Is this is true or am I mistaken here? If this is true, why would this not create issues in learning good representations for task where order is indeed important? 
- The paper discusses several articles that provide background on the methods used however it fails to position the exposition in comparison to existing RNN based approaches which is a big miss as the goal of the paper is to replace RNN based approaches [1,2,3,4].

Method
------
- In the event discovery stage, it is not clear why using consecutive observations is not useful. Also, if one does use L=1 (consecutive observations), can one reduce this method to RNN as you end of capturing all the previous history? Also, why L=3 is good across all different tasks? does it have any relation with the use of 4 frames in I(t)? 
- The authors mention tat H(t) matrix is highly sparse but also low-dimensional in all their experiments.  Would this is be the case for any other task? If not, is this a limitation of the method that you need S and C  to be small? 
- The authors attempt to use clustering based approach with the hope of recovering important information, however
mention that H(t) stores all the past events. Isn't this contradictory? Also, why can RNN with an attention based mechanism not achieve similar effect?
- It is also useful to analyse how will this method work in presence of long vs short history? Will the clustering itself and hence the learned representations get affected by length of available history?

Experiments
----------
- Focusing only on PPO and designing an RNN version of PPO constrains the effectiveness of experiments in validating the approach. Authors mention difficulty of training with RNN as one reason for PPO with RNN's under performance but this is not convincing Could the performance of PPO with RNN be limited only due to specific RNN architecture used?
- Authors must compare with other methods that use RNN approaches (e.g. [1]). Also, if methods such [2],[3],[4] are not  directly applicable, they must atleast, use their RNN based architectures to modify PPO and compare several baselines.
- Why do the authors not report PPO with RNN for Obstacle Tower? 
- For the experiments, authors use a specific 10000 steps budget but this seems to be highly curated. The experiments
would be stronger if the authors show experiment over a range of budget. This will also give insights on when does RNN becomes better and is there a budget after which both methods perform equally well or RNN based approaches
surpass the current approaches.
- Figure 3: Training dynamics seem to favor RNN approach for BreakOut, Gravitar. Do the authors have insight on why
this is the case?
 
Minor points to improve submission not affecting the score:

- The paper needs to be proofread for various typos and sentence construction issues
- Table 1: For Qbert, original PPO (Sch.) is best performing, not EDHR
- [5] talks about difficulties in MI based methods and I encourage the authors to look at the analysis in
this paper. As it is only an arXiv version and not published yet, I have not based my assessment on the
existence of this paper but still connection to such analysis will make this paper stronger.


[1] Deep Variational Reinforcement Learning for POMDPs, Igl et. al.
[2] On improving deep reinforcement learning for POMDPs, Zhu et. al.
[3] Policy Learning with continuous memory states in partially observed robotic control, Zhang et. al.
[4] Memory-based control with recurrent neural networks, Heess et. al. 
[5] On Mutual Information Maximization and Representation Learning, Tschannen et. al.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new way to represent past history as input to an RL agent, that consists in clustering states and providing the (soft) cluster assignment of past states in the input. The clustering algorithm comes from previous work based on mutual information, where close (in time) observations are assumed to be semantically similar. The proposed scheme, named EDHR (Event Discovery History Representation), is shown to perform better than PPO and an RNN variant of PPO on (most of) 7 representative Atari games, and better than PPO on the Obstacle Tower benchmark.

I would like to see this paper eventually published as I find the proposed technique original and quite relevant to current RL research, however I feel like its empirical evaluation is too weak at this time, which is why I am recommending rejection. I hope the results can be strengthened in a revised version so that I can increase my rating.

The main limitations of the current empirical evaluation are:
•	Only 7 Atari games are used (vs 49 in the PPO paper the proposed technique is compared to), without justification for how they were chosen, and it seems like only 1 run is performed on each game (while RL algorithms are well known to exhibit high variance)
•	On Obstacle Tower there seems to be also only one run of each algorithm (more runs could be done with different training & testing seeds in order to get an idea of the variance)
•	There is no comparison to PPO+RNN on Obstacle Tower
•	I think a natural and important baseline to compare to is using the same architecture as in Fig. 2 but where the mapping Phi(o_t) is learned through regular backprop (using the same loss as when learning the mapping I(t)). This would validate that the advanced self-supervised clustering technique from Ji et al. (2018) is actually useful, and thus that the observed improvements are not simply due to providing 32 frames of history vs. 4 as in vanilla PPO.

Other (more minor) remarks:
•	Please explain better how the clustering technique from Ji et al. (2018) works, possibly in the Appendix if there is no room in the main body of the paper. This will make the paper more self-contained. 
•	Without fully understanding how this clustering technique works, it is difficult to me to get an intuition on how the clusters evolve during training, especially as new types of states are discovered by the agent. Some discussion on this topic would be appreciated.
•	It would also be interesting to analyze the impact of varying the various new hyper-parameters (in particular S, C and L)
•	On the first line of the last paragraph on p. 4, there is a missing reference « (Sec. ) »
•	Overall there are a bunch of typos throughout the paper that could easily be fixed


Follow-up on author response: I remain inclined to stick to my rejection recommendation. I appreciate the efforts in providing more results, but I find them too limited and not entirely convincing: Table 4 is only done on 3 games, and the only one where the proposed method has a clear edge over "Without self-supervision" is MsPacMan. In addition, the fact that PPO+RNN shows much better performance than the proposed method on Obstacle Tower is also worrying.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The authors study the problem of RL under partially observed settings. While most current (D)RL approaches use RNNs to tackle this problem, RNNS are trickier to optimise than FFNNs - in practice RNN-based DRL agents can perform well on partially observed problems, may require more effort to optimise, and may underperform FFNNs on domains with no/less partial observability. The proposed solution is to use a FFNN, but provide a "history representation", which is a set of feature vectors for previous timesteps that is extracted from a second network. The second network is trained separately using self-supervision (specifically, IIC, but adapted to use temporal consistency of observations rather than data augmentation). The proposed algorithm outperforms both PPO with a FFNN and PPO with an RNN on 5/7 Atari games (mainly games where partial observability is higher), as well as on the new and challenging Obstacle Tower benchmark - though PPO with RNN results are conspicuously missing on the latter! Given the promising approach (which also has a 2x better wall-clock training time than PPO with an RNN) and results, I would give this paper a weak accept. Some nice properties are that the instantaneous feature extractor is trained using RL, while the history feature extractor is trained using self-supervision at a high level (not pixel level), so that they are probably complementary; in addition it appears that in practice the resulting history features are sparse and usually binary, which is an intriguing and potentially useful property for future work.

There are several things to be done to improve the paper however. First and foremost, the results should be run over several seeds with standard deviation/error reported (it appears that this might be the case for Obstacle Tower, but no error is reported). I believe that the large improvements in some of the domains are significant, but it would be best to have this confirmed empirically. The authors could improve the presentation of background material by giving techniques names instead of using just author names. Although it would be expensive to show how changing L affects performance on all domains, some quantitative results on this hyperparameter would be useful - the same applies to C. The authors should also provide more clarity on choosing the history head - is the head identity fixed after pretraining? It would be useful to know how the more standard PPO with RNN architecture performs, but the authors' choice of architecture is a fitting comparison to their method, and does perform well in practice on the more partially observed domains, so the current setup is satisfactory.
review_title: Official Blind Review #1

Meta-review: The authors propose approaches to handle partial observability in reinforcement learning. The reviewers agree that the paper does not sufficiently justify the methods that are proposed and even the experimental performance shows that the proposed method is not always better than baselines.
Decision: Reject