Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes a task of interactive language-based object search in a navigation environment. To that end, it defines an alternating dialogue where a human sends progressively more specific queries and an agent replies in candidate pictures. The paper introduces a dataset which includes scanned scenes with objects placed inside, chosen to include multiple objects from the same category with potentially distracting properties. It then proposes a method for addressing this task where a 3D cached set of object detections from an open-vocab vision-language detection model is processed with a large language model to rank the candidates based on parsed attributes.
review: Strengths
- I like the task and interactive human-centric setting proposed and the formulation provided for the FindThis task is sensible
- Prompting an LLM for distractor objects seems like a good way to calibrate similarity outputs from CLIP embeddings.
- The open-source is constructed to include diverse objects and valid distractor objects in each scene

Weaknesses
- Real experiments don't have as much complexity as simulated, for example one round only has 2 objects in it (laptop with/without sticker), so the agent will always succeed within 2 rounds. Similarly, there are only 3 mugs so a randomly performing agent would also achieve quite a high success rate.
- Success being counted as >10% of the cropped image containing the object seems like a weird metric, it seems to me the metric should also include the % of the target object included in the crop, with some high threshold required for success (say >80%). Otherwise, a crop could be clearly centered on an adjacent object and only include part of the target object, but still be counted as correct. Does this ever happen in the experiments provided? Why is such a low threshold for performance chosen for the success metric?
- The 3D representation used in GoFind includes lots of redundant views of objects and no way to associate them as the same object. 
- This work implicitly assumes that every query describes an object that actually exists in the scene, since the robot has no mechanism for responding "I can't find it", which is a natural response to a query. This should be discussed in the limitations and ideally mentioned in the problem statement, or if not evaluated in experiments. Additionally, having examples in the 3DOC dataset where queries do not exist would be helpful as a benchmark.

Comments
- In the 3 considerations presented, I feel "natural language" would be a better term than "open vocabulary", which sometimes has some connotations of fixing a set of supported terms before system deployment, such as how "open vocabulary" object datasets still have a restrictive set of object descriptions. On page 2 line 44, "open-vocabulary, natural language" is a bit redundant in my opinion, wouldn't any true natural language system be inherently open vocab?
- The conversation is described as a "multimodal dialogue" (page 4 line 112), which I feel is a misleading personification since the human agent is only able to respond with text, and the LLM only with an image. It is a dialogue which includes multimodal elements, but isn't a full 2-way multimodal dialogue.
- I feel there should be a short section in RW about using code generation from LLMs as an interface for an agent's actions, for example ViperGPT, Code as Policies, and ConceptFusion to name a few. I'm not sure of related work on human-in-the-loop interactive task completion with an LLM, but I'd be surprised if there wasn't (ChatGPT!)
- There are some additional citations on 3D semantic/language maps that could make the section more complete: ConceptFusion, CLIP-Fields, LERF, Distilled Feature Fields, Panoptic Lifting.
- The design decision of using CLIP delta vectors for object attributes is interesting, but more motivation for this method should be provided in the description. Also, the main text describes which lines in the table represent the delta ablation, but the table only refers to "basic parsing". I feel the table could be made clearer, possibly by changing "+ ours" to "+ delta"
- page 6 line 205, "an language query"-> "a language query"
- A discussion of failures should be included given some of the ~50% success rates in simulated scenes.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I think the method and dataset are thorough and contributes a useful dataset for a potentially fruitful direction of human-in-the-loop object localization. Despite some concerns and questions I've listed above, my overall rating is weak accept.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposed a method for robots to search for the location of specific objects using vision and language models.
The proposed method is divided into two phases: a phase in which objects in the environment are investigated by search using an open vocabulary object detector, and a phase in which the robot's execution command is output using LLM from human commands.
In the experiments, several open vocabulary object detectors were compared using the Habitat-Matterport3D dataset.

review: The research topic is a very interesting application of VLM and LLM to real-world tasks in robotics. A unique feature of this research is the disambiguation based on multiple dialogs with the user. Much of the content of this paper has been compressed, and some details seem to have been omitted. I present my questions and concerns below.

The specificity, open-vocabulary, and interactive mentioned by the author are all important for mobile service robots.
However, open-vocabulary can be solved by simply using LLM. It seems to me that this is a problem that has already been solved. The proposed method does not seem to solve open-vocabulary in a new way. Thus, there were no big surprises for me.

The proposed method is able to deal with instruction ambiguity and wrong requests by Multi-around interaction.
However, having multiple linguistic interactions in every instruction may be burdensome for the user.
Could the proposed method use the previous interactions if the same ambiguous instruction is repeated the next time as the first?
In other words, it does not seem to allow the memory and knowledge exchanged once to be used the next time.
Some action or consideration of the above may be needed.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper incorporates the latest machine learning techniques into an important robotics problem. The focus of the problem is very interesting.
The content seems appropriate for this conference.
However, perhaps due to the page limit of the conference, the richness of the content is compressed and some details are omitted.
In addition, more detailed and extensive experiments and comparisons are required to demonstrate the proposed method.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper proposes the GoFind algorithm, which allows a robot to be instructed with a fetching task. The algorithm pipeline detecting ROI of objects and classify them given the parsed input sentence. The authors also propose a dataset 3DOC on which their algorithm is validated.
review: The problem addressed in this paper is very interesting and relevant for service robots. Furthermore, the authors have shown the relevance of their approach in real world experiments. Nonetheless the reviewer has some concerns regarding:
- The clarity of the paper. The proposed method is somehow unclear (delta vectors) and no supporting figure is provided
- The use of templates/parsing. Although the authors provided comparisons to benefits of using a parser, the reviewer is not convinced by such an approach (cf. rebuttal questions)

robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Although the task addressed by the authors is highly relevant and some interesting ideas are proposed (use of delta vectors), the following
 limitations unfortunately hinders the paper:
 *template-based sentences, 
 *unfair comparison with SOA, 
 *lack of sentence diversity, 
Therefore reviewer cannot recommend yet this paper for CoRL.

Update:
Most of the comments have been convincingly addressed by the authors. Nonetheless, the reviewer is not fully convinced by the parsing approach given the simple template sentences. The authors implies that LLM parsers work with natural sentences but this is not shown nor evaluated in the paper. A quick review of the literature suggests that LLM are not 100% accurate:

[1] On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex

[2] ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models

[3] BenchCLAMP: A Benchmark for Evaluating Language Models on Semantic Parsing
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper focuses on finding objects in physical environments when provided with underspecified language commands. The authors first introduce a new environment for testing physical object identification from language commands as well as a dataset for evaluation. It is a household environment with objects that have realistic properties and collocation statistics. They then introduce a system that attempts to tackle the task. The proposed GoFind system is a pipeline system consisting of an object detector, an object localizer, and an instruction parser. 

The system is able to find objects with a greater success rate than the baseline systems. The authors also show the benefits of having an interactive setting, where a user can provide additional information to a model if it fails to find the correct object. 
review: ## Strengths 
1. **Novel task/dataset**: As the authors point out, natural language is often ambiguous. Many current frameworks ignore this fact, and the FindThis task seeks to correct the oversight. The construction of the dataset is solid, especially the attention to distractors and the distinction made between intrinsic and extrinsic attributes. 

2. **Modeling contributions**: As far as I can tell, the integration of delta vectors into the object ranker is novel. The factorization of the task into discrete steps is sensible and the way the steps (location, ranking, measurement) are implemented into an API makes the method flexible w.r.t. the parser. 

3. **Strong experimental results**: Their model shows strong improvements over relevant baselines. The baselines ablate the relevant parts of the model. 

4. **Real-world demonstration**: the authors include a convincing real-world demonstration of their system. 


## Weaknesses 
The first weakness is my only major issue with the paper; the second is more minor. 


1. **"Dialogue" is a stretch**: I have concerns both about the setting and the way it is pitched. Firstly, the fact that the dialogue is triggered by a localization mistake hurts the proposed method's generalizability. This is fine for simple actions like finding an object, but what if the agent were to also have state-changing actions. In this case, waiting for the agent to make a mistake before correcting it would be untenable. A more general system would detect underspecification *before* acting and ask for clarification. 

This ties into my second concern, which is with the pitch of the interaction as a dialogue. The dialogue here is really a scripted 2-stage interaction and doesn't involve language generation from the model side. A more realistic interaction might involve a multiturn dialogue that takes place over the course of the agent performing a series of actions. 

2. **Hard-coded/limited agent**: There are several components in the modeling pipeline that seem hardcoded. Firstly, the exploration strategy is fixed and it is assumed that the agent has explored the environment beforehand (this is mentioned in the Limitations section). Additionally, the script followed by the localization API seems pretty much fixed and has limited breadth. As far as I can tell, program diversity is limited. From the prompt in Appendix A, it seems like there are only 2 types of programs (find_objects -> sort_objects and find_objects -> find_objects -> is_close). This makes parsing task is quite simple and means that it could be handled by intent indentification and slot-filling rather than a full-blown API. Real language, on the other hand, is likely quite a bit more complex and requires a larger set of primitives. 

Another limitation that reduces the method's generalizability is the lack of complex spatial relationships and the lack of compositionality. It is assumed that there is only one relationship that objects can have (near). The authors leave other relationships for future work. Even focusing just on "near", their treatment is overly simplistic. Near is both gradable and context dependent, meaning that a) the meaning of near is vague and it isn't clear what is and isn't near; and b) "near" means different things depending on what the objects are. "Near" in "my house is near the airport" doesn't mean the same thing as "the mug is near the apple". 0.5m may be "near" on a kitchen table, but it is far if the objects are located inside a fridge. 

The intrinsic and extrinsic relations could also be composed, leading to a more realistic and challenging scenario. For example, the templates could be extended to include utterances like "Find the orange mug near the large sink", which would also introduce more complex programs. 


(The following is a suggestion rather than a weakness): The authors point out that certain objects have typical environments. Past work has found that this kind of commonsense information is often encoded in LLMs, so the system could include a planning stage in which the agent determines likely locations to explore. 

3. **Missing related work**: The authors have related works sections for referring expressions, language-conditioned navigation, and 3D semantic mapping, but none for ambiguity and underspecification in robotics, which has been studied in the past. Specific work that also has underspecified instructions (just from a cursory Google scholar search) includes:
- https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9197315
- https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8392364
- https://dl.acm.org/doi/pdf/10.1145/3382507.3418863
- https://journals.sagepub.com/doi/full/10.1177/0278364918760992


robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Based on the paper's strengths, I think it should be accepted. The proposed dataset and system address a gap in current research on interactions with language. Tackling ambiguity will be an important component to scaling up systems that interact with language and enabling use by real people. By framing the disambiguation process as a dialogue (at least conceptually) I think this paper is aiming in the right direction, and takes some important initial steps in that direction. My main concerns are described in the weaknesses section. With additional positioning w.r.t. past work on ambiguity the paper and more hedging around the framing of the interaction as a dialogue, I would likely have recommended a strong accept.
review_title: null

Meta-review: The reviewers appreciate the novelty and formulation of the approach and the interactive human-centric setting, but questioned the experimental setup and evaluation measures, and ask for further discussion of the limitations of the approach.
After the rebuttal and author discussion phase, the reviewers agree that this paper can be accepted for CoRL. The reviewer suggestions should be thoroughly addressed in the camera-ready version of the paper.
Decision: Accept (Poster)