Venue: ICLR 2018 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes one RL architecture using external memory for previous states, with the purpose of solving the non-markov tasks. The essential problems here are how to identify which states should be stored and how to retrieve memory during action prediction. The proposed architecture could identify the ‘key’ states through assigning higher weights for important states, and applied reservoir sampling to control write and read on memory. The weight assigning (write) network is optimized for maximize the expected rewards. This article focuses on the calculation of gradient for write network, and provides some mathematical clues for that.

This article compares their proposed architecture with RNN (GRU with 10 hidden unit) in few toy tasks. They demonstrate that proposed model could work better and rational of write network could be observed. However, it seems that hyper-parameters for RNN haven’t been tuned enough. It is because the toy task author demonstrates is actually quite similar to copy tasks, that previous state should be remembered. To my knowledge, copy task could be solved easily for super long sequence through RNN model. Therefore, empirically, it is really hard to justify whether this proposed method could work better. Also, intuitively, this episodic memory method should work better on long-term dependencies task, while this article only shows the task with 10 timesteps. 

According to that, the experiments they demonstrated in this article are not well designed so that the conclusion they made in this article is not robust enough. 
review_title: A new RL model with episodic memory - robust experiments needed to justify the conclusion

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper considers a new way to incorporate episodic memory with shallow-neural-nets RL using reservoir sampling. The authors propose a reservoir sampling algorithm for drawing samples from the memory. Some theoretical guarantees for the efficiency of reservoir sampling are provided. The whole algorithm is tested on a toy problem with 3 repeats. The comparisons between this episodic approach and recurrent neural net with basic GRU memory show the advantage of proposed algorithm.

The paper is well written and easy to understand. Typos didn't influence reading. It is a novel setup to consider reservoir sampling for episodic memory. The theory part focuses on effectiveness of drawing samples from the reservoir. Physical meanings of Theorem 1 are not well represented. What are the theoretical advantages of using reservoir sampling? 

Four simple, shallow neural nets are built as query, write, value, and policy networks. The proposed architecture is only compared with a recurrent baseline with 10-unit GRU network. It is not clear the better performance comes from reservoir sampling or other differences. Moreover, the hyperparameters are not optimized on different architectures. It is hard to justify the empirically better performance without hyperparameter tuning. The authors mentioned that the experiments are done on a toy problem, only three repeats for each experiment. The technically soundness of this work is weakened by the experiments.

review_title: A new version of episodic memory for DRL with less convincing results

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes a modified approach to RL, where an additional "episodic memory" is kept by the agent. What this means is that the agent has a reservoir of n "states" in which states encountered in the past can be stored. There are then of course two main questions to address (i) which states should be stored and how (ii) how to make use of the episodic memory when deciding what action to take. 

For the latter question, the authors propose using a "query network" that based on the current state, pulls out one state from the memory according to certain probability distribution. This network has many tunable parameters, but the main point is that the policy then can condition on this state drawn from the memory. Intuitively, one can see why this may be advantageous as one gets some information from the past. (As an aside, the authors of course acknowledge that recurrent neural networks have been used for this purpose with varying degrees of success.)

The first question, had a quite an interesting and cute answer. There is a (non-negative) importance weight associated with each state and a collection of states has weight that is simply the product of the weights. The authors claim (with some degree of mathematical backing) that sampling a memory of n states where the distribution over the subsets of past states of size n is proportional to the product of the weights is desired. And they give a cute online algorithm for this purpose. However, the weights themselves are given by a network and so weights may change (even for states that have been observed in the past). There is no easy way to fix this and for the purpose of sampling the paper simply treats the weights as immutable. 

There is also a toy example created to show that this approach works well compared to the RNN based approaches.

Positives:

- An interesting new idea that has potential to be useful in RL
- An elegant algorithm to solve at least part of the problem properly (the rest of course relies on standard SGD methods to train the various networks)

Negatives:
- The math is fudged around quite a bit with approximations that are not always justified
- While overall the writing is clear, in some places I feel it could be improved. I had a very hard time understanding the set-up of the problem in Figure 2. [In general, I also recommend against using figure captions to describe the setup.]
- The experiments only demonstrate the superiority of this method on an example chosen artificially to work well with this approach.
review_title: Interesting new ideas - more work needed to justify approach

Meta-review: This paper presents a memory architecture for RL based on reservoir sampling, and is meant to be an alternative to RNNs. The reviewers consider the idea to be potentially interesting and useful, but have concerns about the mathematical justification. They also point out limitations in the experiments: in particular, use of artificial toy problems, and a lack of strong baselines. I don't think the paper is ready for ICLR publication in its current form.


Decision: Reject