Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: # Summary
This paper presents a new external-memory-based neural network (Neural Map) for handling partial observability in reinforcement learning. The proposed memory architecture is spatially-structured so that the agent can read/write from/to specific positions in the memory. The results on several memory-related tasks in 2D and 3D environments show that the proposed method outperforms existing baselines such as LSTM and MQN/FRMQN. 

[Pros]
- The overall direction toward more flexible/scalable memory is an important research direction in RL.
- The proposed memory architecture is new. 
- The paper is well-written.

[Cons]
- The proposed memory architecture is new but a bit limited to 2D/3D navigation tasks.
- Lack of analysis of the learned memory behavior.

# Novelty and Significance
The proposed idea is novel in general. Though [Gupta et al.] proposed an ego-centric neural memory in the RL context, the proposed memory architecture is still new in that read/write operations are flexible enough for the agent to write any information to the memory, whereas [Gupta et al.] designed the memory specifically for predicting free space. On the other hand, the proposed method is also specific to navigation tasks in 2D or 3D environment, which is hard to apply to more general memory-related tasks in non-spatial environments. But, it is still interesting to see that the ego-centric neural memory works well on challenging tasks in a 3D environment.

# Quality
The experiment does not show any analysis of the learned memory read/write behavior especially for ego-centric neural map and the 3D environment. It is hard to understand how the agent utilizes the external memory without such an analysis. 

# Clarity
The paper is overall clear and easy-to-follow except for the following. In the introduction section, the paper claims that "the expert must set M to a value that is larger than the time horizon of the currently considered task" when mentioning the limitation of the previous work. In some sense, however, Neural Map also requires an expert to specify the proper size of the memory based on prior knowledge about the task. 
review_title: Review

final_score: rating: 7: Good paper, accept
review_content: The paper introduces a new memory mechanism specifically tailored for agent navigation in 2D environments. The memory consists of a 2D array and includes trainable read/write mechanisms. The RL agent's policy is a function of the context read, read, and next step write vectors (which are functions of the observation). The effectiveness of the proposed architecture is evaluated via reinforcement learning (% of mazes solved). The evaluation included 1000 test mazes--which sets a good precedent for evaluation in this subfield. 

My main concern is the lack of experiments to test whether the agent really learned to localize and plan routes using it's memory architecture. The downsampling experiment in Section 5.1 seems to indicate the contrary: downsampling the memory should lead to position aliasing which seems to indicate that the agent is not using its memory to store the map and its own location. I'm concerned whether the proposed agent is actually employing a navigation strategy, as seems to be suggested, or is simply a good agent architecture for this task (e.g. for optimization reasons). The short experiment in Appendix E seems to try and answer this question, but it's results are anecdotal at best. 

If good RL performance on navigation tasks is the ultimate goal then one can imagine an agent that directly copies the raw map observation (world centric) into memory and use something like a value iteration network or shortest path planning to plan routes. My point is that there are classical algorithms to solve navigation even in partially observable 2D grid worlds, why bother with deep RL here? 
review_title: OK paper--needs further ablation experiments 

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper presents a fully differentiable neural architecture for mapping and path planning for navigation in previously unseen environments, assuming near perfect* relative localization provided by velocity. The model is more general than the cognitive maps (Gupta et al, 2017) and builds on the NTM/DNC or related architectures (Graves et al, 2014, 2016, Rae et al, 2017) thanks to the 2D spatial structure of the associative memory. Basically, it consists of a 2D-indexed grid of features (the map) M_t that can be summarized at each time point into read vector r_t, and used for extracting a context c_t for the current agent state s_t, compute (thanks to an LSTM/GRU) an updated write vector w_{t+1}^{x,y} at the current position and update the map using that write vector. The position {x,y} is a binned representation of discrete or continuous coordinates. The absolute coordinate map can be replaced by a relative ego-centric map that is shifted (just like in Gupta et al, 2017) as the agent moves.

The experiments are exhaustive and include remembering the goal location with or without cues (similarly to Mirowski et al, 2017, not cited) in simple mazes of size 4x4 up to 8x8 in the 3D Doom environment. The most important aspect is the capability to build a feature map of previously unseen environments.

This paper, showing excellent and important work, has already been published on arXiv 9 months ago and widely cited. It has been improved since, through different sets of experiments and apparently a clearer presentation, but the ideas are the same. I wonder how it is possible that the paper has not been accepted at ICML or NIPS (assuming that it was actually submitted there). What are the motivations of the reviewers who rejected the paper - are they trying to slow down competing research, or are they ignorant, and is the peer review system broken? I quite like the formulation of the NIPS ratings: "if this paper does not get accepted, I am considering boycotting the conference".

* The noise model experiment in Appendix D is commendable, but the noise model is somewhat unrealistic (very small variance, zero mean Gaussian) and assumes only drift in x and y, not along the orientation. While this makes sense in grid world environments or rectilinear mazes, it does not correspond to realistic robotic navigation scenarios with wheel skid, missing measurements, etc... Perhaps showing examples of trajectories with drift added would help convince the reader (there is no space restriction in the appendix).
review_title: Important paper about structured memory for navigation

Meta-review: Biological memory systems are grounded in spatial representation and spatial memory, so neural methods for spatial memory are highly interesting. The proposed method is novel, well-designed and the empirical results are good on unseen environments, although the noise model may be too weak. Moreover, it would have been great to evaluate this method on real data rather than in simulation. 
Decision: Accept (Poster)