Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents a map-based method for Vision-and-Language Navigation (VLN). The approach builds a semantic map which includes object categories and properties such as color, texture and shape. The semantic map is used to localize the objects and their properties mentioned in the instruction. The localization, map and instruction is used to predict waypoints for a local policy. Experiments onVision-and-Language Navigation in Continuous Environments (VLN-CE) benchmark show competitive results.
strengths_and_weaknesses: Strengths:
- The paper presents of novel application of map-based navigation techniques to VLN-CE. The use of maps encoding both object categories and properties for navigation is also novel to the best of my knowledge.
- The use of ground truth trajectories for language-to-map grounding is innovative. It allows the authors to have a localization objective without requiring map annotations.
- The ablation experiments show the importance of both kinds of map.

Weaknesses:
- There is a published method (CWP-CMA) which performs better than the proposed approach on the public VLN-CE benchmark. The benefit of the proposed approach is that it doesn't require the agent to turn-around to get a panorama between waypoints. However, it is unclear which approach would be better given access to a panoramic camera.
questions: - I do not understand why the authors use the term "Multi-granularity" for describing maps representing objects and their properties. It seems a bit misleading to me because the granularity of the map for both object and properties is the same (denoted by m L143 and L154). What parts of the map have multiple granularities?
- A potential benefit of a modular map-based approach which is trained offline vs end-to-end RL approaches is the sample efficiency as shown in https://arxiv.org/abs/2201.10029. Did you observe such benefits for the proposed approach as compared to the baselines?
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper tackles the Visual & Language Navigation task. The paper proposes a new "multi-granular" map representation for encoding coarse-grained (semantic) and fine-grained (textures, colors, etc) details about the 3D space. It also proposes a self-supervised loss to localize "instruction-relevant" objects on the map. Experiments are performed on the VLN-CE benchmark on MP3D dataset. The work demonstrate strong improvement over prior methods for VLN-CE while using limited field-of-view sensing (as opposed to panoramic sensing).
strengths_and_weaknesses: # ----------------------- Post-rebuttal update --------------------
The author's responses during the rebuttal were very convincing and addressed my concerns. The main weaknesses pre-rebuttal were the novelty of the multi-grain map representation and the auxiliary loss. In response, the authors compared and contrasted with the cited works, and even showed quantitative benefits in relation to them. I've raised my rating to 7 (accept).

# ----------------------- Pre-rebuttal review --------------------
# Strengths

- The paper is well motivated in building multi-granular maps for wholistic object representations and grounding instructions to relevant objects on the map. 
- The paper writing is good and it was easy to read and understand the ideas and contributions.
- Experiments are well designed. A lot of relevant and appropriate baselines have been compared with. The experimental results show good improvements over comparable methods. Ablation studies confirm the value of the contributions. Following traditions from prior work on this task, the authors also present results on an online leaderboard, which verifies the claims made under standardized settings. 

# Weaknesses
## Novelty of multi-granular maps for embodied AI
- The proposed multi-granular maps does the following: (1) use image segmentation features to capture fine-grained details, (2) project the features to get a top-down 2.5D fine-grained map, (3) obtain coarse-grained segmentation maps with potential expansion beyond observed cells, and (4) jointly merging fine-grained and coarse-grained to obtain final representations. This particular pipeline itself appears novel (particularly in VLN).
- However, prior work [1, 2, 3, 4] has extensively studied mapping in high-dimensional feature spaces (as opposed to occupancy / semantic maps). These have not been discussed in Sec. 2.2.
- More importantly, they might be doing exactly what the proposed method intends to do. The general idea is to get visual feature maps of the egocentric inputs, and project them in a geometric / learned fashion to top-down maps / voxel grids. That is, the multi-grained map may be obtained implicitly here as opposed to explicitly as done in the Fig. 2. 
- For example, Semantic MapNet [2] builds such maps from visual inputs in the encoder, and decodes them to obtain semantic maps. In L145-146, the authors suggest that the last layers of an image segmentation model (Unet) contain a good mix of textural and semantic features. It is reasonable to expect the same to hold true for the last layers of Semantic MapNet to encode something similar, but in the top-down map space. 


## Novelty of auxiliary task
- L174-176: "The principle for generating coarse ground-truth is that the regions closer to the path have a higher probability of containing instruction-relevant objects." --- It is unclear to me whether this supervision is resulting in localizing "instruction-relevant" objects. 
- This objective could also plausibly be viewed as a form of imitation learning for waypoint prediction (like [5]). For example, in Figure 3, the model could be learning to predict the locations of future waypoints to follow instead of referring to the carpet (i.e., the carpet being along the waypoints could just be a coincidence). 
- If the model were to just predict future waypoints, then the method will no longer be self-supervised. The self-supervised component only refers to how "instruction-relevant" objects are localized without explicit supervision.


## Need clarifications in approach and experiments
- L152-156 - is hallucination of maps beyond the field of view helpful quantitatively?
- Eqn 1 - is it a pixelwise loss?
- L173 "Because we do not know the exact locations of these instruction objects" --- how often does this type of instruction-object ambiguity occur? For example, the instruction object being a long-bench and there are multiple benches nearby. I can imagine that in quite a few cases, the objects might be unique and can be localized accurately. 
- L177-184 - does P remain the same for all time-steps of the trajectory? If so, is it possible for the model to just ignore visual inputs and map the fixed language instruction to the fixed P (regardless of the agent states)?
- In Tab. 3 ablation study - do all rows benefit from the auxiliary task? For example, the result without auxiliary task in Tab. 4 is close to semantic map (row 3 in Tab. 3). If semantic map does not use auxiliary task, then the gain from multi-granular representations itself is limited.
- **[Minor point, not weakness]** L231 - how is the DAgger training performed without a human in the loop? Might be useful to add a section in the supplementary to explain this.
- **[Minor point, not weakness]** In Tab. 5 ablation study - classification layer logits are better than semantic map from Tab. 3. Could the authors comment on this?


# Typos and grammatical errors
L11 - "Moreover, we propose" -> "We further propose"   
L64 - followings -> follows  
L87 - "disables the agent to ground … well"  -> "prevents the agent from effectively grounding …"  
L114 - "access to predefined" -> "access the predefined"  
L122 - "larger number" -> "large amount"  
L164 - procuding -> producing  
L168 - "reason precise semantics" -> "reason precisely about semantic"  
L231 - "teaching force" -> "teacher forcing"  
L322 - "carper" -> "carpet"  


	
[1] **MapNet:** Henriques, Joao F., and Andrea Vedaldi. "Mapnet: An allocentric spatial memory for mapping environments." proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.  
[2] **Semantic MapNet:** Cartillier, Vincent, et al. "Semantic mapnet: Building allocentric semantic maps and representations from egocentric views." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 2. 2021.  
[3] **Cognitive Mapping and Planning**: Gupta, Saurabh, et al. "Cognitive mapping and planning for visual navigation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.  
[4] **Geometry-Aware RNNs**: Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. "Geometry-aware recurrent neural networks for active visual recognition." Advances in Neural Information Processing Systems 31 (2018).  
[5] **Cross-Modal Map Learning**: Georgakis, Georgios, et al. "Cross-modal Map Learning for Vision and Language Navigation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.  
questions: # Novelty of multi-granular maps

- Is the proposed mapping method achieving something different compared to prior work [1, 2, 3, 4]? 
- If so, can the authors quantitatively compare against map representations from [2] (and maybe even [1, 3, 4]) and verify it is indeed better? That is, replace the proposed multi-granular maps while keeping the auxiliary task and waypoint navigation as it is.

# Novelty of auxiliary task

The method from [5] picks waypoints along the demonstration trajectory, and places gaussians at each waypoint to obtain the target heatmap for prediction. 
- Is the heatmap P in L177-182 different from the heatmap in [5]? 
- If they are indeed different (i.e., P focuses more on instruction-relevant objects rather than waypoints), could the authors replace the heatmap P with the heatmap from [5] and verify that the proposed method is indeed better? (note: the multi-granular maps and waypoint navigation pipeline remain the same)
- Is the auxiliary task really self-supervised? That is, is it learning to localize instruction-relevant objects? Is the heatmap from [5] not resulting in localizing instruction-relevant objects in that case?

# Other clarifications in approach and experiments
- Could the authors clarify the questions raised in the weaknesses?

[1] **MapNet:** Henriques, Joao F., and Andrea Vedaldi. "Mapnet: An allocentric spatial memory for mapping environments." proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.  
[2] **Semantic MapNet:** Cartillier, Vincent, et al. "Semantic mapnet: Building allocentric semantic maps and representations from egocentric views." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 2. 2021.  
[3] **Cognitive Mapping and Planning**: Gupta, Saurabh, et al. "Cognitive mapping and planning for visual navigation." Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.  
[4] **Geometry-Aware RNNs**: Cheng, Ricson, Ziyan Wang, and Katerina Fragkiadaki. "Geometry-aware recurrent neural networks for active visual recognition." Advances in Neural Information Processing Systems 31 (2018).  
[5] **Cross-Modal Map Learning**: Georgakis, Georgios, et al. "Cross-modal Map Learning for Vision and Language Navigation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.  
limitations: Yes. They have been sufficiently addressed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: ## Post Rebuttal

I thank the authors for their response. My main concern was addressed and the zero-shot evaluation on RxR is helpful. I encourage the authors to include a full evaluation on RxR. I have increased my rating.

## Pre-rebuttal
This paper presents a method for building an environment map towards the end of performing natural language instruction following in continuous environments (VLN-CE). In the proposed method, the map has multiple granularities of information. A coarse granularity with features projected from UNet, and a global semantic map created by hallucinating semantic labels for the entire environment based on the coarse granularity map.

The encourage the map contain features relevant to instruction following, the authors also propose an additional objective that predicts the location of the ground-truth path on the environment map. This predicted path location is then used to attend to the map.

The method is evaluated on the VLN-CE dataset and outperforms all existing single camera methods. It is also competitive with panoramic methods.
strengths_and_weaknesses: ### Strengths

The proposed method is well motivated. It makes sense to have multiple levels of information in a semantic map and to augment what is given to the agent based on the instruction.

The method works well. VLN-CE is a challenging task and a +4.5% SR increase is meaningful.

The ablation studies are well-done and provide useful insight.

### Weaknesses

No evaluation on RxR-Habitat (https://ai.google.com/research/rxr/habitat). This uses the more recent RxR dataset that fixed some of the issues with the original VLN dataset and thus evaluation on this dataset would be very helpful in evaluated the method.

The intro to the task should cite + mention VLN-CE since that's the variant of the task being studied (and the less common one). I recommend referring to the task as VLN-CE to avoid confusion.

L28-29: We don't know why humans (or any animal for that matter) build cognitive maps. Neither Tolman nor O'Keefe claim to know why cognitive maps were learned to be built (other than that they are useful for survival). They are an explanation of observed behavior, seen, but we don't know why that mechanism exists other than it's useful for survival (and thus is something that was learned implicitly). I recommend weakening this statement.

Self-supervised map representation learning via object location task is not a correct name. The objective is not self-supervised. Self-supervised implies that the learning objective is valid regardless of the dataset source (i.e. training data or data we'd see at evaluation). Ground-truth paths are certainly not part of the data we'd see at evaluation so this method doesn't fit. The task is possibly object localization with pseudo-labels (so weakly supervised), but those pseudo-labels are soft labels for predicting the ground-truth path. Since the motivation comes from object localization, that part of the name is probably fine, but the self-supervised part isn't.

Overall, I think this paper presents a meaningful contribution and am happy to increase my rating if these concerns are addressed.
questions: See weaknesses
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper presents an approach for vision-and-language navigation in continuous environments, centered around a semantic map which both aggregates and is used to predict object-centric information from the agent's observations. The approach uses fine-grained features extracted from a U-Net segmentation model applied to the agent's visual observations, as well as features for unseen parts of the environment predicted by a network supervised with ground-truth 3D info from the Matterport environment. This "multi-granularity" map is supervised using the ground-truth path of the agent through the environment, and used together with a recurrent, instruction-conditioned attention mechanism to predict waypoints for an off-the-shelf point-nav model to navigate to. The approach obtains the highest performance, by a substantial margin, among published VLN-CE leaderboard approaches that do not use panoramic image representations. Ablations show that each component of the approach helps pretty substantially.
strengths_and_weaknesses: *Strengths*

S1) The object representation and semantic map approach is well-motivated and well-executed, and the performance on the benchmark seems very strong.

S2) The paper performed a thorough set of experiments that helped me convince me all the decisions made in the approach (both levels of granularity in the semantic map; the localization task) were useful. 

*Weaknesses*

W1) One weakness of the approach is that the full version depends on 
ground-truth 3D semantic information from the Matterport3D simulator in training (although not in inference, I believe). If I understand Table 3 correctly, without this info (just using the fine-grained map with segmentation features) the approach performs underperforms most of the other non-panoramic approaches (CMA, LAW, CM2). Since this info is only used in training, I don't think it's a crucial weakness, but I feel the paper could be clearer about this limitation, in particular perhaps rephrasing or removing the motivating claim in the intro that contrasts its approach with "a naive solution that annotated a dataset ... with object classes and attributes for a segmentation model". (lines 40-41). 

_Edit after response_: The response presented some compelling experiments showing that the proposed approach also improves when the semantic information is predicted by a pre-trained Faster R-CNN model, so I now feel like this is even less of a weakness. 

W2) The approach, as I understand it, is quite similar to the approach of Blukis et al, "Mapping Navigation Instructions to Continuous Actions with Position-Visitation Prediction", CoRL 2018, which also uses object-centric semantic maps and waypoint prediction, to do language-conditioned navigation in a continuous environment. However, I don't think this is a crucial weakness as that approach was applied in a different environment with what seem to be much simpler visual observations (involving a discrete set of objects). It would strengthen this paper to contrast with that approach in the related work.

_Edit after response_: As the authors pointed out in their response, Blukis et al. does not use the multi-granular information that the proposed approach does. 

W3) The writing could be clearer, see suggestions below.

questions: Q1) Could the authors clarify what sources of ground-truth semantic information from the Matterport3D simulator (beyond RGB and depth imagery) are used, and when they are used -- in training, inference, or both?

Q2) What parts of the model are updated during each of the two stages of training: teacher-forcing, and DAgger training? (e.g. is DD-PPO model held fixed during both of these, and the map prediction and object localization modules fine-tuned)?

Q3) What are the off-the-shelf FGB and depth encoders (line 197)?

limitations: I found the discussion of limitations in the paper somewhat lacking, although it should be easy to address given more space. In addition to discussing limitations due to object and environmental supervision (see above), it would also improve the paper to discuss more about possible directions that could improve the language-to-object grounding.

*Suggestions*
- Figure 1: the "bed" and "door" layers were unclear.
- The motivation in the intro confused me a bit about what type of object supervision is used.
- The object supervision in coarse localization was confusing to me. The localizer doesn't really seem to get any explicit object supervision, just closeness to the path -- and there doesn't seem to be flexibility for the model to choose how to focus the region attention using e.g. some sort of weakly-supervised or MIL loss. I think it might help to present this in a different way, e.g. as learning general info about the path (which may or may not be object-relevant). 

*Minor suggestions*
- 112: "continued" -> "continuous"
- 149: "camera is with a" -> "the camera has a"
- 168: "reason over precise semantic information"
- 322: "carper" -> "carpet"
review_title: null

Meta-review: metareview: The paper received all positive reviews (3x accept ratings, 1x strong accept rating). The meta-reviewer agrees with the reviewers' assessment of the paper.
confidence: Certain
Decision: Accept