Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The manuscript describes a method by which rich robotic commands provided in natural language are turned into robotic actions.
strengths_and_weaknesses: The major weakness of this approach is that it does not handle language, despite this being the focus of the manuscript. It handles keywords, specifically noun phrases. It cannot handle verbs, adverbs, determiners, etc.

The sentences "After passing a white building go left" and "Instead of passing a white building go left" mean exactly the same thing!

The manuscript is written to give the impression that the model handles sentences like "After passing a white building take a right next to a white truck..." but it only handles 'white building && white truck'.

I understand what the authors are trying to do, but, this is misleading to readers and merits far more than a footnote on page 4!

This could be a good contribution, but, the title, abstract, manuscript and evaluation should be completely rewritten to remove language and to focus on what the algorithm actually does, landmarks.

Then, the algorithm should be compared against a baseline method that simply uses Named Entity Recognition to find the landmarks in text and then use those.

It is not clear to me what the value of a model that essentially does NER and then recognizes landmarks using a pretrained transformer is. This seems like a straightforward application that doesn't offer much. If the method were extended to handle linguistic commands in general, that would be exciting. But it's not clear that this is possible with the given approach.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The method promises to handle linguistic commands. But, it handles conjunctions of keywords. And it doesn't compare against baselines that do the same.
issues: I indicated these above. The method doesn't handle language, only keywords.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposed an instruction-following navigation system. The approach consists of three components: 1) A VNM (i.e. ViNG) to construct a topological graph of the environment from pre-collected images of the environment, 2) a LLM (i.e. GPT-3) to parse the instructions into an ordered list of “landmark” object names, and 3) a VLM (i.e. CLIP) to match nodes of the VNM’s topological graph (i.e. images of landmarks) to the landmark object names. Since VNM can perform navigation from its constructed graph, the key idea of the paper was to use two pretrained models without fine tuning to translate natural language instructions into a sequence of nodes in the topological graph. The key assumption of their approach is that instruction-following navigation reduces to landmark-sequence-following navigation, whose limitations were addressed in the discussion.
strengths_and_weaknesses: Strengths:
 - Informative qualitative results on video.
 - Code and data released in supplementary material and webpage.
 - By building on top of large internet-scale pretrained models, the approach is open-vocabulary. This is an important strength and distinction over many prior works. I recommend highlighting this strength more, either through writing or empirically by providing navigation instructions which references objects not in the MS-COCO list of classes (and therefore, closed-world object detectors such as Faster-RCNN-FPN will fail).

Weaknesses:
 - Claims of their language driven robotic navigation is "self-supervised". Meanwhile, their experiments are based on human-collected data, and cited autonomous data collection approaches do not trivially cover all relevant landmarks and areas. Given that the authors were manually involved in both creating the navigation instructions (containing references to landmarks) and in manually collecting the robotic experience (which presumably passes through all relevant landmarks), this raises concerns in the validity of experiments for backing up claim of a "self-supervised system" (L28-29). More details in issues section.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper proposed a simple but effective system for outdoor navigation from language instructions. It has good performance well for the 5 queries reported and provides sufficient ablations for understanding impact of each component of the system. The reviewer's main feedback is in claiming the system is "self-supervised".
issues: Issues:
 1. One of the key claims of the paper is "a self-supervised" system (L28-29, L46) and that the map is "constructed from prior exploration of the environment" (L297). While the approach does not require "human annotations of robot navigation data" (L55), it does depend on humans for "manually driving the robot and collecting image and GPS observations" (L214-215). This human collected offline data play key roles in the approach's performance (e.g. in coverage of all relevant waypoints) and the evaluation process (e.g. view angles for 100 images for VLM grounding evaluation, L268). What is the precise procedure of the initial data collection? What properties about the collected data affect the final approach's performance and evaluation procedure? How can this data collection process cover all the landmarks which might be used by downstream instructions?
       - I recommend making assumptions about this prior data collection explicit, either by explicitly stating some dependency on a human collected data or an exploration algorithm (along with experimental results). In addition, phrases like "constructed from prior exploration of the environment" (L297) implies that the robot is actively collecting its own data, and should be modified to be more accurate.
      - The authors mentioned that such data collection could be obtained from past traversals or with autonomous exploration methods like [Shah et al](https://proceedings.mlr.press/v164/shah22a/shah22a.pdf). It is not clear to me that these approaches will have enough coverage of the area and/or of landmark objects.
 2. CLIP's performance can be sensitive to not just the prompt but also the exact phrase used to describe the object category (e.g. "person" v.s. "pedistrian"). What are the 30 commonly occuring landmarks (L268) and how are they "manually curated" (L270), especially since instruction following experiments only contained 19 landmarks (L219)?

Minor Issues:
 1. I find it surprising there was such a big performance difference CLIP-derived object detector (i.e. ViLD) and directly using CLIP, and recommends that the author includes per-class detection tables in the appendix for the community to better compare open-vocabulary object detectors in different scenarios.
 2. What is the setup for quantitative experiments in Table 2? Especially for zero-shot open-vocabulary object detectors, whose performance depends on the amount and specific choice of classes considered, are all approaches given all of MS-COCO's classes?
 3. Figure 2’s text is way too small. Please make it bigger.
 4. \overline{l}’s repeated definition in L39 and L44.
 5. Grammar mistake in L249 “On independently evaluating the efficacy of our the VLM at retrieving landmarks, .."
 6. I feel that 5 queries is not enough to quantitatively evaluate the model's navigation performance. This is an extremely small sample size compared to instruction following works, but in the same range of ViNG's 6 pairs of start-goal image pairs.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper presents Large Model Navigation (LM-Nav), a system that combines 3 pre-trained models (visual navigation model, vision-and-language model, and large language model) for long-horizon instruction following in the real-world. The authors demonstrate that utilizing these pre-trained models without any fine-tuning in the target environment results in successful following of natural language environments for 100s of meters outdoors. The various models are trained on large-scale datasets using self-supervised objectives, obviating the need to build a photorealistic simulator, or trajectory-level annotations.
strengths_and_weaknesses: Strengths:

The paper is very well written, and is a strong systems paper in combining various modules for long-horizon instruction following in outdoor environments on a real robot. The results are impressive, and the motivation is clear. I also appreciated the  investigation into the different components of their approach, demonstrating high performance of the LLM and VLM .

Weaknesses:
- The VNM is trained using a large dataset from outdoor environments on the same robot embodiment used in this paper, and is not transferable to a different robot embodiment (i.e., a quadruped robot, or a robot with a different height). This dataset is also costly to collect. 
- The VNM requires the user to manually drive the robot around in the target environment to build the topological map and collect image/GPS observations which seems time-consuming. How long does this process take? What happens if obstacles change in the environment after this topological map is built (i.e., a car is parked on the road, blocking the path)? This seems to be the main drawback between this work, and other VLN approaches, which are more adaptive to the test environment, and require no previous exploration of the test environment to follow language instructions. 
- This approach is dependent on the VNM finding the same landmarks during the exploration phase as the natural language instruction. What happens if the VNM misses a landmark during the exploration phase? 
- How does the motion blur of the robot affect the performance of the VLM?
- The main paper should include more quantitative results (that are currently in the appendix). 


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper has strong results, but more details on the limitations of the VNM module and time to adapt to novel environments should be included. 

issues: Some limitations of the VNM module should be addressed and added to the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presented an approach to combining pretrained navigation, vision-language, and language models for vision-language suburban navigation.  The proposed method uses large language models (LLM) such as GPT-3 for landmark extraction, vision-language models (VLM) such as CLIP to match landmarks with text descriptions, and ViNG to navigate based on the constructed topological map. The experiment showed that the combined system can successfully carry out instructions in outdoor settings.
strengths_and_weaknesses: Strengths
- This paper demonstrates several important properties for embodied navigation
    - LM-Nav does not require photo-realistic simulation for training.
    - LM-Nav does not require environment-specific natural language annotations for training. 
- LM-Nav was deployed in a realistic outdoor environment and showed successful navigation.

Weakness
- The space of language instruction is quite limited.
    - It is understandable that LM-Nav only considers landmarks. But even for landmarks, this system requires the utterances of landmarks ordered in the sequence of visits. For example, LM-Nav cannot understand instructions such as “you’ll find a car after passing the stop sign” because the visit order is opposite from the order in the sentence. This is one of the main limitations if we want to use it as an interface. We cannot guarantee the order of landmark utterances.
    - Since the input sentences are parsed into landmarks, the system cannot understand “take right next to a white truck”. LM-Nav can perform this query possibly because it optimizes the reachability of the next landmark, if the instruction stops at this sentence, it will fail too. Spatial information is important for navigation. For two paths that end at similar landmarks, e.g. white truck, the system cannot disambiguate which path to take unless that are other landmarks on the path that can be added to the instruction.
- LM-Nav requires past observations of the environment to build topological maps. To deploy LM-Nav to a new environment, it needs to collect data on the new environment, this gives a performance boost to navigation. This also makes language only provides constraints for graph search.  
- It is hard to interpret the quantitative results for the combined system. The main quantitative result appeared in appendix B which should be in the main text. I currently have some questions about the descriptions here.
    - In the main text, it says the experiment is based on 5 queries but the appendix says 20 instructions. Are these 20 instructions from the 5 queries?
    - If the experiments include multiple language queries, are Table 3 and 4 in the appendix refer to the mean?
    - The language instructions in the experiments seem to be different, in this case, how can the expt. length to be similar? 
    - The ablation of LM-Nav only runs on “No VNM” version. We’ll need more to understand how each component affects the performance of the whole system rather than each component individually. For example, these ablations:
        - No LM, No VLM, provide visual landmarks instead. This gives us the upper bound of the system if we have correct visual landmarks.
        - No LM, but use VLM with ground truth landmark parses. This tells us how landmark localization impacts performance.  
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: This paper tackles an important problem in visual language navigation and shows some promising properties. However, the experiment section in the current form is hard to evaluate the effectiveness of the whole system and the contribution of the LM and VLM to navigation performance. 

issues: The following issues need to be addressed:
- Appendix B should be in the main text and include more ablations to understand the contribution of each component to the whole system. For the failure cases, it would be great to include the failure mode analysis to understand the limitation of the design.
- The space of text input is more restricted than currently stated in the paper. The author will need to describe the space of input command and its limitations.
review_title: null

Meta-review: metareview: 
All reviewers appreciate the qualitative results and writing and agree it demonstrates a robust system for visual language navigation. However, there are a few claims made by the author need to be further clarified. Two most important ones: 

1. Language understanding. Area Chair agrees with reviewer Mudn and 2UZt's concern about the limited understanding of the language instruction -- ignoring information beyond landmark, which makes it a partial solution for "instruction following." Authors could consider updating the paper writing to address this concern.  

2. Claim about "Self-supervised" raised by reviewer eeqY.

During rebuttal, authors are able to update the manuscript to include additional discussion on "limited language understanding", and remove the claim about self-supervision. 

AC think this paper presents an interesting and practical system and would recommend acceptance. Authors are encouraged to consider including comparisons to methods using "name-entity recognition" (reviewer Mudn and  2UZt)
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)