Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper introduced Egocentric Spatial Memory Networks (ESMN), a novel learning paradigm and architecture for encoding spatial memory in a sphere representation. The representation can be used for several downstream applications ranging from semantic segmentation as well as action learning for robotics applications.

The formulation of the proposed approach builds on a Kalman Filter setting and encodes memory in a spherical structure. Both the formulation and design makes sense, and the downstream applications show that the proposed approach is indeed plausible.

While I think the value of this paper is well justified, I do have a few comments on the paper:
- I think it'll be worthwhile to elaborate on the "Mono" method, including the network architectures as well as training details and such. It will also help understand which part of ESMN improves over this baseline method.
- In the related work section, one argument made by authors is that ESMN and MemNN/NTM are two different paradigms for learning spatial memory. While the proposed approach ESMN is reasonable, I wonder if it is possible to have a concrete experiment comparing those two different designs.
- Could the authors elaborate on Sec 4.1.1 Fig 5? What's the meaning of the colorings in this figure, and how to interpret that?
review_title: Reivew

final_score: rating: 7: Good paper, accept
review_content: The paper considers the problem of creating spatial memory representations, which play important roles in robotics and are crucial for real-world applications of intelligent agents. The paper proposes an ego-centric representation that stores depth values and features at each pixel in a panorama. Given the relative pose between frames, the representation from the previous frame is transformed via forward warping (using known depth values) to the viewpoint of the current frame. The proposed approach has no learnable parameters. Experiments on a wide range of tasks show that the proposed approach outperforms baselines such as LSTM and NTM.

On the positive side, the approach is positively simple, in the sense that it relies on known techniques (EKF, forward warping, etc.) that ensure that it is easy to implement while achieving good results in the experiments. Up to Sec. 4, I found the paper easy to follow, although some design choices could be better motivated (e.g., I assume that diagonal covariances are assumed for simplicity).
The paper evaluates the proposed approach on multiple tasks and in various configurations, which is another strength of the paper.

While I like the proposed approach, I also see multiple significant weaknesses:
1) I found the experimental evaluation nearly impossible to understand. My main problem is that I don't understand what the different method that are evaluated are:
 * Given the abbreviation ESMN introduced in the abstract, I assume that ESMN is the proposed approach. ESM seems to be a variant of ESMN, but I am not sure how ESM and ESMN differ as the difference is never clearly described (or if it is, I seem to have missed it). Sec. 4.1.1 mentions training with a convolutional encoder in the context of ESMN, Sec. 4.1.3 and Sec. 4.1.4 only evaluate ESM but not ESMN, while Sec. 4.2 states that "ESM represents map-only inference, while ESMN includes convolutions for both the image-level and map-level inference". Unfortunately, the term "map-level" inference is not well-defined. Overall, I don't understand the difference between ESM and ESMN. As a result, it is unclear to me why ESM performs worse than ESMN in Tab. 1 for DR-Ego-S but comparable for all other tasks in the table, or why ESM and not ESMN is used for some of the experiments. Similarly, what is the "ESM-DepthAvoid" baseline? Does it only use depth and no features?
 * Tab. 1 contains a baseline called "PO" and I don't understand how it works. The abstract introduces PO as an abbreviation for partial observability, but that does not seem to be an explanation for a baseline. The PO baseline performs similarly well as ESM and ESMN in Tab. 1, which makes me wonder whether Tab. 1 really shows the superiority of the proposed approach.
 * I am confused by the statement "In contrast, ESM by design stores features in the memory with meaningful indexing. The inclusion of relative cartesian co-ordinates in the memory image also effectively aligns each pixel with an associated relative translation." since the inclusion of such coordinates is never mentioned before. I assume ESM uses some form of handcrafted features?
2) Some of the statements made in the paper seem too strong:
 * I don't see how the claim "Our memory is much more expressive than these 2D examples, with the ability to represent detailed 3D geometry in all directions around the agent." (Sec. 2.2)  holds. I agree that being able to store information in 2.5D (panorama and depth) is more powerful than storing only top-down 2D maps. However, the allocentric maps of the references can store larger and more complicated scene parts. E.g., if an agent would turn around a corner, ESMN would essentially be forced to forget about everything that is not directly visible anymore as it is occluded and thus not included in the memory structure anymore. As such, a point can be made that ESMN is much less expressive than for example Henriques & Vedaldi. Given that the latter evaluate on significantly more complex scenes compared to the ones used in this paper (which do not have strong occlusions) strengthens this impression. 
 * Regarding the statement "Although the most recent depth frame is also a strong signal for local obstacle avoidance, we show that the avoidance based on the full ESM geometry results in fewer collisions when tested on a variant of the drone task with the inclusion of 25 obstacles.": Looking at Tab. 2, it seems to me that the standard deviation is so large for both ESM and ESM-DepthAvoid that it is unclear to me whether one is consistently better than the other. 
3) The CodeSLAM approach from Bloesch et al. also provides a form of memory representation and I don't understand why the paper, and its follow-up (Zhi et al., SceneCode: Monocular Dense Semantic Reconstruction using Learned Encoded
Scene Representations, CVPR 2019), is not discussed in the related work section.

Overall, I believe that the paper has potential. My main criticism is that I do not feel able to properly understand the experimental evaluation. As such, it is hard to recommend acceptance. However, I am willing to increase my score if the information necessary to understand this part of the paper is provided.

### After rebuttal phase ###
The answers provided by the authors and the revised version of the paper sufficiently address my concerns. As such, I recommend to accept the paper. I am still concerned that the experimental evaluation is very packed with multiple experiments while lacking details on the experimental setup and explanations of the baselines. Still, I feel that in the current form, the paper can be accepted.
review_title: Interesting approach to egocentric memory based on forward warping; experiments cover a wide range of tasks, but are very hard to understand as there is little / no explanation of the evaluated methods; some claims seem too strong

final_score: rating: 4: Ok but not good enough - rejection
review_content: # Summary
-------
This paper presents a method to build an ego-centric spatial memory map from an agent's viewpoint. This map module is differentiable and can be used for a variety of tasks, such as object segmentation, or image-to-action learning in different control tasks.

# Pros
----
+ The idea of combining the ego-centric representation seems interesting and novel. 
+ Evaluation in Image-to-Action learning shows good performance compared to baselines. 

# Cons
----
   *  _[Major]_ The Method section is very confusing. The method is based on the EKF pipeline and modifies the particular parts of the pipeline. The authors only briefly outline the overview of the method and then jump into specific details.  The text mostly focuses on improvement, and hence it is hard to judge which parts are novel and what the contribution is. Furthermore, there are errors in notation and variables that are not properly explained. u_t is first defined as incremental pose measurements and then as a control vector. Kamera intrinsic matrix K_1 is never introduced. The method section would benefit from restructuring, especially giving a clear overview of the method would help the reader to understand the method better.

   * _[Major]_ Experiments are not well explained, and more baselines are necessary for proper evaluation. In all experiments, the experimental set-up is not well defined. For example, in the first experiment, we find out that the authors use imitation learning only from the appendix. In the second experiment, the environment used for the evaluation is not introduced. The experimental set-up needs to be clearly outlined, and the goal of the experiment needs to be provided. The choice of baselines is not well motivated. Why were LSTM and NTM chosen in the first experiment? There is very little rationale for the choice in the submission. Do these approaches show the state-of-the-art performance on the examined tasks? In the second experiment, the authors only use the LSTM network to train the RL agent. Various methods tackle the navigation problem in the RL domain, and hence the authors should choose better baselines for comparison. I suppose that the PO task is some form of navigation task, although the authors never explain what the PO task is. 

   * _[Minor]_ The simple approach to directly quantize pixel projections leads to artifacts in the map. It is a bit unclear why this option was chosen, especially now that progress on differentiable renderers has been made.

review_title: Ego-centric Spatial Memory Networks - Review

Meta-review: In this paper, the authors combine ideas from SLAM (using an Extended Kalman Filter and a state with nonlinear transitions and warping) and differentiable memory networks that store a spherical representation of the state (from the ego-centric point of view of an RL agent moving in an environment) with depth and visual features stored at each pixel and dynamics transitions corresponding to warping.

The main idea in the paper is very simple and elegant, but I will concur with the reviewers that the writing of the first version of the paper was extremely hard to understand and that the experimental section was too dense. Two subsequent revisions of the paper have dramatically improved the paper.

Given the spread of scores (R1: 6, R2: 7 and R3: 4) and the fact that only R1 and R2 have acknowledged the revisions, I will veer towards acceptance.

Decision: Accept (Poster)