Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposes a technique that enables zero-shot object-goal navigation: given a text description of an object that is reachable in a 3D environment, an agent needs to find the object using visual input. The contribution of the paper mainly lies in that it utilizes a pre-trained visual-language model (i.e. CLIP) to project the image goal of the object and the textual description of the object into the same embedding space, such that a navigation agent trained on only image goals can perform zero-shot navigation when only given textual goal at test time. The proposed method is evaluated on three datasets and is shown to outperform prior methods in zero-shot setting. Because CLIP is pretrained on large-scale image-language pairs, this approach also enables the possibility of specifying navigation goal in free-form language at test time, such as “find a kitchen sink”.
strengths_and_weaknesses: Strengths:

1. The paper text is written with clarity and figures are well made.
2. The method appears to be novel for the reviewer.
3. The evaluations are done on several different datasets and show consistent improvement over zero-shot baseline, though there still exists certain gap to supervised baseline.
4. The most significant strength of the paper is that it investigates an important question in the field: how to leverage internet-scale data (or existing large models pre-trained on these data) for open-world embodied tasks? This is particularly important because any dataset from a single (or a few) domains would not be able to capture the complexity of the open world, so leveraging large-scale unstructured data is one of the few promising approach to achieve this. This paper investigates exactly this and shows a meaningful step forward.

Weaknesses:

1. Though requiring unannotated data, the ImageNav agent presented in this paper relies on large amount of data for training. And even by significantly increasing the dataset size, as shown in Table 2, only moderate improvement is shown on ObjectNav (along with decreased performance in ImageNav). It raises a question about how scalable the ImageNav agent is, i.e. how much more improvement in success rate can one expect by continuing scaling the training data.
questions: Please refer to the comments in the weakness section.

To summarize and re-iterate:

1. In Table 2, even though the dataset is scaled to 1000 envs from 72 envs, why is there decreased performance for ImageNav?
2. And why is there only 6.6% improvement in ObjectNav success rate, given the significant increase of training envs?
limitations: Yes, the authors have adequately addressed certain limitations of the method.
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The paper addresses ImageNav and ObjectNav tasks, using pre-trained CLIP representations.
strengths_and_weaknesses: Strengths:

The paper is well-written.

I do appreciate the discussion on open-world scenarios.

Weaknesses:

Section 1: The manuscript states "An important advantage of our approach is that it reduces the data labeling burden."  To be precise, the approach proposed in the manuscript, itself, does not reduce the data-labeling burden; instead, it enjoys the reduction in required data-labeling through the use of CLIP — particularly since CLIP was not even fine-tuned on the task.

Section 1: I remain somewhat unconvinced by the approach proposed by the manuscript. Arguably, the approach *has* observed general information about the task, through CLIP’s (pre-)training step and that the proposed approach is just utilising this background experience to perform somewhat above-random on the transfer task.

Section 4 / Figure 2: I have a strong concern with the novelty claims in this manuscript. CLIP is used off-the-shelf (with no effort to adapt CLIP to the tasks), visual encoder is pre-trained, policy network architecture and training paradigm is taken from the plethora of related approaches in embodied tasks.

Section 5 / Table 1: Experiments are incomplete? Partial experiments and different baselines on different tasks? Some consistency is required.
questions: See above
limitations: Transfer of existing generalised representations is only useful if those representations can be suitably adapted to specific downstream tasks (maximising performance of the approaches therein), while retaining their cross-task generalisability. I feel like this discussion is important, but it is missing from the present manuscript.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This work proposed the semantic-goal navigation task on the object goal navigation without the provided object-goal annotation. Instead of the conventional object-goal annotation, they converted the image-goal navigation task into semantic-goal navigation (SemanticNav) by aligning goal images from image-goal nav into a multimodal, semantic embedding space for object goal nav.

The key idea is that they use the CLIP representation to transform image-goals and object-goals into semantic-goals representing navigation targets. They trained a semantic-goal navigation agent using image-goals transformed by CLIP. This allows the agent to search for the goal position from both visual and textual supervision.

They compared their model of ZSON with two baselines for SemanticNav: Zero Experience Required (ZER) of mapping a goal object category into goal-image embedding space and CLIP on Wheels (CoW), which is free from the learned navigation policy. They also compared to the conventional supervision model of OVRL. Their experimental results surprisingly suggest that ZSON is competitive or better than the supervised model of OVRL in SPL.

Although their approach for SemanticNav is quite interesting, their approach reminds me of the data augmentation or multi-tasking for the conventional object-navigation task from a different task of the image-goal navigation. If the proposed model is effective, it will surely succeed in the conventional object-goal navigation settings compared to the existing object-goal models. It might be also interesting to compare the multi-tasked models with the state-of-the-art (supervised) object navigation models.
strengths_and_weaknesses: **Stregness**
- Proposal to a new task of SemanticNav driven by CLIP joint representation of the target object image and language. This allows the vocabulary target supervision.

**Weakness**
- The proposed model is evaluated in the limited condition. It seems that it is possible to apply the proposed model for the conventional (supervised) object-goal navigation
- Model comparisons are limited. I'd like to see more strong baselines in both the SemanticNav setting and the (supervised) object-goal navigation setting.
questions: - If my understanding is correct, it is possible to use your navigation dataset combined with the original object-goal navigation. As you didn’t use the object-goal annotations, you can finetune the semantic-nav agent in the conventional object-goal navigation. Did you experiment with your model in this setting?
limitations: - Lack of the comparisons with other state-of-the art models in the conventional object-goal navigation setting
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The problem of zero-shot object navigation has been addressed in the paper. The idea is to project images into a joint space of images and text using a CLIP encoder. Then, the agent is trained to navigate towards images sampled from the environments (which does not require any annotation). At test time, the target categories are specified using text embedded in that joint space. 
strengths_and_weaknesses: **Pros**

+ The problem that this paper investigates i.e. open-world object navigation is quite important.

+ The paper provides strong zero-shot results. 

**Cons**

- The idea is a small increment over the recent works such as [19] and [21]. The paper argues that [19] requires annotation during training (L85). However, their annotations are obtained automatically. So it is not different from the proposed approach in terms of annotation. 

- The comparisons with the baseline [20] is not fair. The baseline uses ResNet-9 while the proposed method uses ResNet-50. So it is not clear if the improvement comes from the higher capacity of the model or this particular type of training.

- It is not clear if scaling up can improve this approach (as claimed in L255). Mapping language to vision might not be simply improved by scale since the CLIP encoders are fixed.

- Only a few qualitative examples are shown for more complex instructions. It is not clear if those are cherry-picked examples. So, quantitative analysis should be provided for that claim. 



questions: - L151: "substantially improves training time (by ∼3.5x)" --> What is the baseline for this comparison?

- L188 mentions 3000 episodes for 3 levels of difficulty. L190 mentions 9M episodes. It seems they are not consistent.

- It is well known that CLIP text and image embeddings are quite different. Is there any additional trick applied to these embeddings to make them similar?
limitations: The limitations of the work are discussed, but not the societal impact.
review_title: null

Meta-review: metareview: This work introduced a method for object-goal navigation in open-world settings. The crux of the approach is to use a pre-trained vision-and-language model CLIP to associate embeddings of image goals and textual descriptions of the objects such that the model can perform zero-shot navigation given textual goals.

This paper received mixed reviews from four expert reviewers, ranging from Borderline Reject to Strong Accept. The reviewers appreciated the problem motivation, the novel method of using large-scale pre-trained models for embodied AI, and the demonstrated zero-shot navigation performances. Nonetheless, concerns have been raised by the reviewers, including the limited evaluations and unfair baseline comparisons.

The authors did a good job easing many of the concerns raised in the initial reviews. They successfully swayed Reviewer mStZ to update their rating to Borderline Accept and Reviewer tra9 to Borderline Reject. In addition, Reviewer J2xC mentioned that they will update their rating to Weak Accept if the authors include the results in the new revision, which the authors promised to do. The only reviewer who held a negative opinion after the discussion period is Reviewer tra9, who argued against the authors' claim that their approach reduces the data labeling burden (by using pre-trained CLIP) and pointed out that different baselines are used for different tasks. The AC read the paper, the reviews, and the reviewer-author discussions carefully and found that the authors made convincing arguments in response to Reviewer tra9's criticisms. Given that the majority of the reviewers held positive opinions on this work and no major flaw had been identified, the AC would recommend accepting this work at NeurIPS and letting the community judge the merit of this work. The AC would strongly suggest the authors incorporate all reviewers' comments in the final version.
confidence: Less certain
Decision: Accept