Venue: ICLR 2018 conference

Reviews:

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper presents a method for navigating in an unknown and partially observed environment is presented. The proposed approach splits planning into two levels: 1) local planning based on the observed space and 2) a global planner which receives the local plan, observation features, and access to an addressable memory to decide on which action to select and what to write into memory. 

The contribution of this work is the use of value iteration networks (VINs) for local planning on a locally observed map that is fed into a learned global controller that references history and a differential neural computer (DNC), local policy, and observation features select an action and update the memory. The core concept of learned local planner providing additional cues for a global, memory-based planner is a clever idea and the thorough analysis clearly demonstrates the benefit of the approach.

The proposed method is tested against three problems: a gridworld, a graph search, and a robot environment. In each case the proposed method is more performant than the baseline methods.  The ablation study of using LSTM instead of the DNC and the direct comparison of CNN + LSTM support the authors’ hypothesis about the benefits of the two components of their method. While the author’s compare to DRL methods with limited horizon (length 4), there is no comparison to memory-based RL techniques. Furthermore, a comparison of related memory-based visual navigation techniques on domains for which they are applicable should be considered as such an analysis would illuminate the relative performance over the overlapping portions problem domains  For example, analysis of the metric map approaches on the grid world or of MACN on their tested environments.

Prior work in visual navigation in partially observed and unknown environments have used addressable memory (e.g., Oh et al.) and used VINs (e.g., Gupta et al.) to plan as noted. In discussing these methods, the authors state that these works are not comparable as they operate strictly on discretized 2d spaces. However, it appears to the reviewer that several of these methods can be adapted to higher dimensions and be applicable at least a subclass (for the euclidean/metric map approaches) or the full class of the problems (for Oh et al.), which appears to be capable to solve non-euclidean tasks like the graph search problem. If this assessment is correct, the authors should differentiate between these approaches more thoroughly and consider empirical comparisons. The authors should further consider contrasting their approach with “Neural SLAM” by Zhang et al.

A limitation of the presented method is requirement that the observation “reveals the labeling of nearby states.” This assumption holds in each of the examples presented: the neighborhood map in the gridworld and graph examples and the lidar sensor in the robot navigation example. It would be informative for the authors to highlight this limitation and/or identify how to adapt the proposed method under weaker assumptions such as a sensor that doesn’t provide direct metric or connectivity information such as a RGB camera. 

Many details of the paper are missing and should be included to clarify the approach and ensure reproducible results. The reviewer suggests providing both more details in the main section of the paper and providing the precise architecture including hyperparameters in the supplementary materials section. 

review_title: Clever idea with strong supporting experimental evidence, but paper is missing key details about the approach

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:

A method is proposed for robot navigation in partially observable scenarios. E.g. 2D navigation in a grid world from start to goal but the robot can only sense obstacles in a certain radius around it. A learning-based method is proposed here which takes the currently discovered partial map as input to convolutional layers and then passes through K-iterations of a VIN module to a final controller. The controller takes as input both the convolutional features, the VIN module and has access to a differential memory module. A linear layer takes inputs from both the controller and memory and predicts the next step of the robot. This architecture is termed as MACN.

In experiments on 2D randomly generated grid worlds, general graphs and a simulated ground robot with a lidar, it is shown that memory is important for navigating partially observable environments and that the VIN module is important to the architecture since a CNN replacement doesn't perform as well. Also larger start-goal distances can be better handled by increasing the memory available.

Comments:

- My main concern is that there are no non-learning based obvious baselines like A*, D*, D*-Lite and related motion planners which have been used for this exact task very successfully and run on real-world robots like the Mars rover. In comparison to the size of problems that can be handled by such planners the experiments shown here are much smaller and crucially the network can output actions which collide with obstacles while the search-based planners by definition will always produce feasible paths and require no training data. I would like to see in the experimental tables, comparison to path lengths produced by MACN vs. those produced by D*-Lite or Multi-Heuristic A*. While it is true that motion-planning will keep the entire discovered map in memory for the problem sizes shown here (2D maps: 16x16, 32x32, 64x64 bitmaps, general graphs: 9, 16, 25, 36 nodes) that is on the order of a few kB memory only. For the 3D simulated robot which is actually still treated as a 2D task due to the line lidar scanner MxN bitmap is not specified but even a few Mb is easily handled by modern day embedded systems. I can see that perhaps when map sizes exceed say tens of Gbs then perhaps MACN's memory will be smaller to obtain similar performance since it may learn better map compression to better utilize the smaller budget available to it. But experiments at that scale have not been shown currently.

- Figure 1: There is no sensor (lidar or camera or kinect or radar) which can produce the kind of sensor observations shown in 1(b) since they can't look beyond occlusions. So such observations are pretty unrealistic.

- "The parts of the map that lie within the range of the laser scanner are converted to obstacle-free ...": How are occluded regions marked?
review_title: What about strong motion-planning baselines?

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper addresses the important problem of planning in partially observable environments with sparse rewards, and the empirical verification over several domains is convincing. My main concern is that the structure of these domains is very similar - essentially, a graph where only neighboring vertices are directly observable, and because of this, the proposed architecture might not be applicable to planning in general POMDPs (or, in their continuous counterparts, state-space models). The authors claim that what is remembered by the planner does not take the form of a map, but isn't the map estimate \hat{m} introduced at the end of Section 2.1 precisely such a map? From Section 2.4, it appears that these map estimates are essential in computing the low-level policies from which the final, high-level policy is computed. If the ability to maintain and use such local maps is essential for this method, its applicability is likely restricted to this specific geometric structure of domains and their observability. 

Some additional comments:

P. 2, Section 2.1: does H(s) contain 0s for non-observable and 1s for observable states? If yes, please state it.

P. 3: the concatenation of state and observation histories is missing from the definition of the transition function.

P. 3, Eq. 1: overloaded notation - if T is the transition function for the large MDP on histories, it should not be used for the transition function between states. Maybe the authors meant to use f() for that transition?

P. 3, Eq. 3: the sum is over i, but it is not clear what i indexes.

P.3, end of Section 2.1: when computing the map estimate \hat{m}, shouldn't the operator be min, that is, a state is assumed to be open (0), unless one or more observations show that it is blocked (-1)?

P.5: the description of the reward function is inconsistent - is it 0 at the goal state, or >0?

P. 11, above Fig. 9: typo, "we observe that the in the robot world"
 
 
review_title: The paper proposes a novel neural network architecture for planning in partially observable environments with sparse rewards. It uses a differentiable memory module to maintain an estimate of the geometry of the partially observable state, and splits planning into a two-level hierarchical process. Experiments in several domains demonstrate the validity of the proposed architecture. 

Meta-review: The authors have proposed an architecture that incorporates a VIN with a DNC to combine low level planning with high level memory-based optimization, resulting in a single policy for navigation and other similar problems that is trained end-to-end with sparse rewards. The reviews are mixed, but the authors did allay the concerns of the most negative reviewer by adding a comparison to traditional motion planning (A*) algorithms. 
Decision: Accept (Poster)