Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This very well written and executed paper synthesizes several ideas recently published in the field of deep reinforcement learning-based goal-driven navigation. It elegantly combines these ideas together by presenting a neural agent architecture that consists of:
* a perception module (e.g. a convnet) that extracts coarse visual feature maps s_t from an RGBD image
* a differentiable map canvas M_t that is rotated at each step based on affine egocentric velocity (dx_t, dy_y, d \phi_t)
* differentiable inverse projection mapping, which uses known camera parameters, projective geometry and the depth channel of the image to project the visual feature vectors s_t onto a 2D map and add it to the existing canvas M_t
* a recurrent module (GRU) for update a state h_t that is used for computing the policy distribution and value function
* additional inputs to the policy and value function, that include a global map read r_t, as well as a query q_t (produced by the policy head) based retrieval of features from the map
* position indexing of features retrieved from the map

The algorithm is trained end-to-end, without extra supervision, using Advantage Actor-Critic (A2C) RL. Based on the strong inductive biases regarding the map, namely affine transforms of the map given information about relative movement, and projective geometry transformations of visual features in the map frame, it seems that the question of where to write is solved, and that the network only needs to learn what to write in the differentiable map. Evaluation is done on 3 games in VizDoom: finding the exit of the Labyrinth, object retrieval and find and return / Minotaur. 

Criticism:

The authors could justify better the choice of using the projective geometry inductive prior. They use sentences like "We argue that projective geometry is a strong law imposed on any vision system working from egocentric observations" (not quite related to grid and place cells, despite being in that section) and "this inverse mapping operation is second nature to many organisms" without giving any reference.

Several papers have been published in the last two years, focusing on differential memory architectures with a 2D map structure, projective geometry. This paper goes further by building and iteratively updating a 2D occupancy map using visual features and image geometry, just like RGBD-SLAM (which would merit a citation, e.g., [1] and [2]). This paper essentially combines existing ideas (see table 1): projective geometry, reward-based learning of M_t, RL, multitask navigation, semantic features. While this is not novel, seeing all this combined in a single technique does have merit.

What is disappointing, given that this is a combination paper, is that the environment is so simply, and that photorealistic environments were not tested. For example, the VizDoom environment uses 2D sprites for objects, making the visual feature extraction from objects much simpler. Would the method work equally well with the objects in DeepMind Lab, which are seen from multiple view points? And what in an environment like AdobeIndoorNav?

[1] Henry et al (2010) "RGB-D mapping: Using depth cameras for dense 3D modeling of indoor environments"
[2] Izadi et al (2011) "KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera"
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper studies how to build semantic spatial maps for the purpose of navigation in 3D environments. The paper presents a differentiable policy network that pastes together semantic map predictions into a spatial map. Information is read out from this map using a global read operation (that looks at the entire map) and a self-attention read operation. This information is used to produce actions. The paper presents experimental results in 3D VizDoom scenarios and reports improvements over a vanilla LSTM, and another spatial memory based method (Neural Map).

Strengths:
1. I very much like the proposed formulation for tackling navigation problems. Using learning to leverage semantic reasoning, and structuring the computation spatially makes a lot of sense.
2. In my view, the proposed formulation advances current models in the following ways:
a. Maintaining and updating allocentric maps, and reading off egocentric maps. This alleviates need for repeated rotations of the map, and thus prevents aliasing.
3. The paper provides ablations for the various parts of the system and provides qualitative analysis of the learned spatial representations.
4. Very good placement of work in current literature. I really like Table 1.

Shortcomings:
1. The central contribution of the paper is the design of the egocentric spatial memory, how to build and maintain it over time, and its use in deep RL. The paper does this by using components from previous papers and presents a very nice summary of this in Table 1. Unfortunately, modulo the component described above (that of maintaining allocentric maps and reading off egocentric maps as and when needed), all other components are borrowed from existing papers, as can be seen in Table 1 already. The paper lists its contributions in Introduction on page 2, and each of those contributions has been studied in previous papers (though I will that admit no single paper does all these things together). Thus, I believe the paper falls short in terms of technical contributions.

2. Following on from point above, putting everything together and showing that it works, could also be a reasonable contribution, though it would warrant more extensive and systematic experiments for the different design choices, possibly in more realistic environments. For example, a) is the projective projection important, or could that have been learned, b) do repeated rotations indeed lead to blurred representations, c) what is critical to get such models work with RL, that past models that used imitation learning couldn't, d) other claimed differences from past works in this space.

3. Experiments and analysis:
a. The paper compares against NeuralMap, and reports improvements, but doesn't give a reason as to why this happens.
b. Past works have demonstrated these ideas in visually realistic environments (similar to those in Gibson / Habitat, see semantic tasks in CogMap). Current paper only investigates proposed ideas in VizDoom environments.

Thus, while I like the direction of research and the fact that the paper presents an architectures that uses latest techniques in the area, I believe the paper doesn't have enough technical contribution of its own, and experiments are limited to synthetic VizDoom environments.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: The paper proposes a novel architecture for spatially structured memory. The main idea is to incorporate inductive bias/invariance derived from projective geometry arguments. The experiments seem to clearly show that this new architecture improves previous approaches to tasks which require spatial reasoning and memory, and the ablations studies and visualizations provide useful insights into the workings of the agent. One thing I'm missing is an experiment showing that this inductive bias also doesn't degrade performance on tasks where spatial reasoning is not necessary (as compared to vanilla GRU/LSTM).
review_title: Official Blind Review #3

Meta-review: This paper presents a spatially structured neural memory architecture that supports navigation tasks.  The paper describes a complex neural architecture that integrates visual information, camera parameters, egocentric velocities, and a differentiable 2D map canvas.  This structure is trained end-to-end with A2C in the VizDoom environment.  The strong inductive priors captured by these geometric transformations is demonstrated to be effective on navigation-related tasks in the experiments in this environment.

The reviewers found many strengths and a few weaknesses in this paper.  One strength is that the paper pulls together many related ideas in the mapping literature and combines them in one integrated system.  The reviewers liked the method's ability to leverage semantic reasoning and spatial computation.  They liked the careful updating of the maps and the use of projective geometry.  

The reviewers were less convinced of the generality of this method.  The lack of realism in these simulated environments left the reviewers unconvinced that the benefits observed from using projective geometry in this setting will continue to hold in more realistic environments.   The use of fixed geometric transformations with RGBD inputs instead of learned transformations also makes this approach less general than a system that could handle RGB inputs.  Finally, the reviewers noted that the contributions of this paper are not well aligned with the paper's claims.

This paper is not yet ready for publication as the paper's claims and experiments were not sufficiently convincing to the reviewers. 
Decision: Reject