FindThis: Language-Driven Object Disambiguation in Indoor Environments
Authors: Arjun Majumdar
Fei Xia
brian ichter
Dhruv Batra
Leonidas Guibas
Venue: CoRL 2023 Poster (2023)
Keywords: object disambiguation
instruction following
language interaction
visual navigation

Abstract: Natural language is naturally ambiguous. In this work, we consider interactions between a user and a mobile service robot tasked with locating a desired object, specified by a language utterance. We present a task FindThis, which addresses the problem of how to disambiguate and locate the particular object instance desired through a dialog with the user. To approach this problem we propose an algorithm, GoFind, which exploits visual attributes of the object that may be intrinsic (e.g., color, shape), or extrinsic (e.g., location, relationships to other entities), expressed in an open vocabulary. GoFind leverages the visual common sense learned by large language models to enable fine-grained object localization and attribute differentiation in a zero-shot manner. We also provide a new visio-linguistic dataset, 3D Objects in Context (3DOC), for evaluating agents on this task consisting of Google Scanned Objects placed in Habitat-Matterport 3D scenes. Finally, we validate our approach on a real robot operating in an unstructured physical office environment using complex fine-grained language instructions.

FindThis: Language-Driven
Object Disambiguation in Indoor Environments
Arjun Majumdar Fei Xia
Brian Ichter! Dhruv Batra²
Leonidas Guibas
'Google DeepMind Georgia Institute of Technology Stannford University
Work done while at Roboties at Google.
Natural language is naturally ambiguous. this work, consider interac- tions between user and a mobile service robot tasked with locating a desired object, specified by a language utterance. We present task FindThis, which addresses the prob- lem of how to disambiguate and locate the particular object instance through dialog with the user. To approach this problem we propose an algorithm, GoFind, which ex- ploits visual attributes of the object that may be intrinsic (e.g., color, shape), or extrinsic (e.g.. location, relationships to other entities), expressed in an open vocabulary. GoFind leverages the visual common sense learned by large language models enable fine- grained object localization and attribute differentiation in a zero-shot manner. We also provide a new visio-lingue linguiiic dataset, 3D Objects in Context (3DOC), for evaluating agents on this task, consisting of Google Scanned Objects placed in Habitat- Matitat-Matterport 3D scenes. Finally, we validate our approach real robot operating in an unstructured physical office environment using complex fine-grained language instructions.
Abstract:
Round
Find myy laptoo please
sticker on
Language-driven Object Disambiguation
Parsed alin butes Foregrounde Isptop'
Foregroundt
'aaptpp 'oomputer", beok', paper', Yolder]
Disrsatoos:: [Tacklet 'shonel", 'computerr, bbook;, 'paper: "Ialder.
'notebook; "aindar, Vdakk "table, "countarr) Attribtte None
Attribute:
Figure I: We propose FindThis: an object navigation task in which an agent engages in multiple rounds of open-vocabeln opea-vocabulary, natural language interaction to locate a specific object in a complex environment. We present GoFind, an approach to parse the instruction and robustly find objects by identifying attributes and removing distractors. this example, though the robot initially finds a laptop, the user able to provide additional attributes ('stickers on it') to identify the correct object.
Introduction
long-standing goal in robotics to develop agents that can follow natural language instructions to find objects in the real world. From the onset, researchers have envisioned systems capable of identifying specific objects through a free-flowing dialog - allowing
to specify and clarify instructions (e.g..
SHRDLU [1]). Recently, progress has been made by on sub-tasks within this grand challenge. For example, one line of research has centered on grounding referential language (e.g., 'Find the lamp next to the sofa." ') in ousey-bbbe 2D images [2] or 3D scenes [3, 4, 5). Complementary work has focused on developing agents that can visually navigate (i.e., take actions) through partially-observale
3D environments by following step-by step-by-step bt-step navigation instructions (e.g., 'Exit the bedroom. Walk down the hall. Stop next to the bathromm sink.) [6. 7, 8]. Furthermore, researchers have developed agents that explore 3D environments to find objects given concise instructions that specify objects by category labels (e.g., 'Find table", 'Find a bed', Find a chair", etc.) [9, 10].
7th Conference on Robot Leaming (CoRI. 2023), Atlanta, USA.

---

Given the tremendous progress in each of these domains, we believe it is time to ask: what type of user interaction should ultimately be supported by robots tasked with finding objects? Towards answering this question we identify three key desiderata:
1) Specificity, Users should be able request specific instance of an object. For example, when asking for a 'laptop' we often would like our 'laptop" and not one belonging to someone else. Thus, as illustrated in Figure I, users might specify distinguishing features such as 'the laptop with stickers on it' or 'the laptop on the bed'. This requires systems to differentiate intrinsic attributes (e.g.. color, patterns, shape) and/or extrinsic properties (e.g., spatial location, relationship to other entities).
2) Open-vocabulary. The set of objects should not be restricted - users should be able to request any object they require. Thus, systems must support an open vocabulary [11, 12, 13].
3) Interactive. When providing instructions users may make ambiguous or even mistaken requests. For example, a user might request an 'orange mug' when multiple mugs of that color exist. In such cases, the request underspecified and further interaction required to provide additional details or corrections.
To evaluate agents under these real-world conditions, we introduce a task and dataset. As shown in Figure I, our task (FindThis) requires following natural language instructions (e.g.. 'Find my laptop.) localize spe- cific object in an indoor 3D environment such as a home or office. If the agent fails by finding an incorrect object (as Figure 1), the task continues with the user providing further details about the desired object (e.g.. 'No, mine has sticker on it.). In other words, FindThis agents must engage with users in a multi-round interaction, interpreting open open-vocabulary, natural language instructions to localize a desired object.
In addition, we present the 3D Objects in Context (3DOC) dataset, which is designed for large-scale evaluation of FindThis agents in zero-shot manner i.e., the 3DOC dataset does not include training split. The dataset consists of language instructions generated from dialog templates, paired with scenes constructed using a diverse set of 3D objects from the Google Scanned Objects (GSO) dataset [14] and placed in 3D scanned indoor environments from the Wlaitat-Matterppa 3D (HM3D) dataset [15].
To address the FindThis task, we propose novel zero-shot approach (GoFind) to handle the challenges of multi-round interaction for fine grained object localization. In our approach, we build an veecbbocayy
3D scene representation using vision-am -osoon-nn--aaae -langgga models (e.g., ViLD [16], OWL- VIT [17]. CLIP [18).. Then, we query the representation with the help of a large language model (e.g.. GPT-3 [19] or PaI.M [20]) which identifies object attributes mentioned in the dialog and uses visual common sense (i.e., knowledge about visual similarity) to propose background objects that the embodied agent (i.e., robot) should ignore.
Finally, we validate our approach on both 3DOC and on a real robot operating in real-world office environment using complex fine-grained instructions such as Find a bowl full of cereal' or 'Find an upside down mug'. We find that our proposed approach generalizes well to the real-world setting, performing as well or better than the performance observed in simulation.
2 Related Work
2D and 3D Referring Expression Tasks. Aligning referential language (e.g., 'the mug on the table") with image or spatial regions has been studied in both 2D [2] and 3D [3, 4, 5] settings. A common framing of this task is a game between two players in which both players fully observe scene, then one describes an object that the second player must identify [2]. The 3D versions of this game (e.g., ReferIt3D [3], ScanRefer [4], Refer360 [5], REVERIE [21]. SOON [22]) require fine-grained disambiguation of objects based on intrinsic and extrinsic attributes. Our proposed task (FindThhis) expands on this prior work by lifting this problem into a real-world setting in which target objects are described through multi-round dialog with a user and an embodied agent must explore 3D environments, observing objects at different distances and viewing angles and ignoring distractor objects that share some but not all properties with the target.
Language-Conditione Conditioned Visual Navigation Tasks. Language-conditione navigation in 3D environments is often studied through instruction following [23, 24, 8, 25, 26, 27] or object-goal navigation (ObjectNav) task [9. 10]. This work falls into ObjectNav, which prior work has focused on a setting in which objects are only described by a category label (e.g., 'sink", 'table', etc.) drawn from a closed set of categories, and finding any instance of an object with that label is defined as success[ 10, 28, 29, 30, 31, 32, 33). By contrast, we study an alternative setting in which the user prefers a specific instance of an object (e.g., 'the orange mug") ") drawn from an open set of categories that are not predetermined. Recent work has studied similar vocauuacyy setting [11, 12, 13]. Closely related, [12] propose an open-vocabulary object
2

---

Example Object Collection
Agent Observations
Multi-Round Dialog
Target
Wilhi-Clese Ditrrctions
Intrinsic RI: stripe
Environment
Environmant
R2: No, my towel is also red.
Out-of-Clss Distrcttors
Estrinsic
RT: Flid mhe towal on the chali.
R2: No. towel amped
Diversity Dataset
Figure 2: Examples from the 3DOC dataset. 3DOC is designed benchmark the ability of agents to parse multi-round open-vocabulary dialog to identify objects in a 3D scene. Compared with other object-in-sccot datasets, 3DOC designed include more challenging within-class and out-of-class distractors and includes more diversity in objects and scenes.
navigation task coined language-driven zero-shot object navigation (L-ZSON). The primary difference between L-ZSON and the task proposed in this work (FindThis) that we consider a multi-round dialog. which is often required for disambiguating similar object instances.
3D Semantic Mapping. Numerous methods have been proposed for building semantic maps of 3D environments. One line of work adds semantic features to traditional SLAM methods. Alternatively, several works propose using object detectors or semantic segmentation techniques to build semantic maps. A key limitation of these methods is that the detectors and segmentors are pre-trained on fixed set of object categories. This makes such methods inapplicable the open-vocabulary setting studied in this work.
Recent methods have addressed this challenge using recently proposed open-vocabulary vision-and- language models (e.g., CLIP [18]. ALIGN [34], BASIC [35]) and object detectors and segmentors (e.g.. ViLD [16]. MDETR [36], OWL-VIT [17], LSeg [37]). Specifically, NLMap [38] uses ViLD detector and CLIP features to represent 3D environments. VLMaps [39] leverages LSeg [37], a model that projects 2D images into the CLIP representation space at the pixel-level, for mapping. And the best version of CoW [12] uses the OWL-ViT detector. In all of these methods, objects are found using queries generated by a pre-trained text encoder. By contrast, in this work we consider more complex query mechanisms required to address the unique challenges the FindThis task. Furthermore, we leverage a large language model (LLM) to provide common sense visual knowledge and decompose language instructions, improving object localization performance.
3 The FindThis Task and the 3DOC Dataset
Our goal is to design task in which user interacts with embodied agent using natural language with an open- vocabulaayy to describe a specific object in a 3D scene for the agent to find. In this section, define such task (Section 3.1), evaluation metrics (Section 3.2), and present dataset (Section 3.3) designed for large-scue evaluations of embodied agents on this task in a zero-shot manner (Figure 2).
3.1 Task Definition
In FindThis, for each episode an embodied agent (a mobile robot) must localize target object T, (e.g.. 'a mug') )) in 3D environment E, through a multi-round dialog D, in which a user describes the object. Specifically, the agent is initialized at starting location in the environment E,, and must explore to find the target object T,. other words, given descriptions an object, the agent must 'go find this'.
By nature, language instructions can be ambiguous and underspecify the task. For example, user might ask for their specific 'mug" when multiple mugs are present in the scene. In such cases, users must provide additional information to resolve ambiguities. Specifically, the additional information may describe

---

intrinsic attributes of the object (e.g. 'Find my orange mug') ") or extrinsic attributes such as the spatial relationship with another object in the scene (e.g. 'Find the mug next to the sink'). A FindThis dialog may include both types of disambiguating information.
As illustrated in Figure 1, we consider multimodal dialog in which users provide natural language instructions such as 'Find my laptop please' or 'No, mine has sticker on it', and the agent responds with an image of a candidate object Cj (e.g.. an image of laptop). Formally, sequence of these user instructions Ij and agent responses Cj compose a M-round dialog D; [i CI CI,...,IM,.CM In this work, we consider M [1.2}. An episode is considered successful if the candidate object matches the target (i.e., T, is in Cj).
3.2 Evaluation Metries
We use two variations of success rate (SR) measure performance: Top-1 SR and Top-5 SR. Top-1 SR (a standard variant) corresponds with the percentage of episodes in which the agent presents the user with the correct candidate object (i.e., T; is in Cj) within M rounds of dialog. Top-5 SR considers setting in which the agent can present five candidates in each round. If the target object is within the set of candidates, the episode is considered success in terms of Top-5 SR. A target T; is considered within the image crop Cj if the ground truth segmentation mask of T; covers at least 10% of the cropped image Cj.
3.3 Evaluation Dataset
This section describes the 3D Objects in Context (3DOC) dataset, designed for evaluating agents on the FindThis task. 3DOC is designed for zero-shot evaluation, training episodes are not provided. We instantiate the task in the Habitat [40, 41] simulator by placing 3D scanned objects from the Google Scanned Objects (GSO) dataset [14] into photorealistic 3D environments from the Habitat- -abttat-Matterportt. 3D (HM3D) dataset [15]. Examples are shown in Figure 2 and statistics summarized in Table 2 in Appendix B.
Scene Construction. We create scenes by procedurally dropping the 3D objects surfaces within the HM3D [15] environments. Specifically, for each object we annotate if the object typically appears on elevated surfaces (e-g., table, desk, counter, etc.), on the floor, or on both surface types. Then, we place the object above an appropriate surface location, drop the object using the simulator's physics engine, and verify that the object stably landed in close proximity (i.e., within 0.05m) of the desired surface location. This procedure ensures that objects are placed in semantically reasonable locations that might represent a messy house (e.g., with toys randomly placed on the floor). A similar approach for determining reasonable object placements was used to construct an evaluation dataset for an open-vocabulary object navigation task in [42].
A key challenge in FindThis is disambiguating distractor objects that are similar to the target object in some way. To include such scenarios in 3DOC, we annotate the object category and intrinsic attributes for 72 objects from the GSO dataset [14]. We use these annotations to curate 50 object collections composed of two different types of distractors. Specifically, we sample up to objects from a target object category with different intrinsic attributes and up 4 objects from other categories that may share intrinsic attributes with the target objects. Thus, for a given target object, agents must disambiguate both within-class and outside-of-claat distractors to solve a FindThis episode. An example object collection is shown in Figure 2. Finally, we place objects from each of the 50 object collections into 10 different HM3D [15] validation environments, which results in 500 unique scene layouts in 100 different HM3D environments.
Intrinsic Attribute Instructions. Given a target object from an object collection, we create an initial intrinsic attribute instruction using the template: Find the <attr 1> <object>, where <attr 1> corresponds to the most prominent intrinsic attribute of an object (e.g., its color or pattem) and <object> is name for the object (e.g., 'mug' or 'coffee mug'). In subsequent rounds of interaction, a user response is synthesized using secondary intrinsic attributes (<attr 2>) with the template: No, my <object> is also <attr 2>. In total, generate 1,713 unique episodes, for 10 different object categories and 53 unique attribute mentions to form the intrinsic attribute split of the 3DOC dataset.
Extrinsic Attribute Instructions. Initial extrinsic attribute instructions are generated with the template: Find the <ob jebt> trelations <ref>, where <object> is the target, <ref> is an object in proximity of the target, and <relation> is drawn from the set: fon the, in the, next to the, near the). An example extrinsic attribute instruction is shown in Figure 2. In multi-round dialog for extrinsic attribute episodes, subsequent rounds use intrinsic attributes for further differentiation with the template: No, my <object> is <attr>, where <attr> is an attribute of the target. In total, the extrinsic attribute split of 3DOC contains 50 unique episodes.

---

Vocabulary 3D Scene Representation
Fine-Grained Object Localization
nortee
my orange
mi_tid_iig namee way
distcactorany "plate bowt.
LLM Parsing
locabalery Object Detector
Locatiaatio Eind ()
maga sogs sejectar decestarmuga
abjacnn)) (ngeloset)
eutucn eaga(0)
Figure 3: Overview of the GoFind algorithm. Fisst, we frontier exploration and class-agnostic object detection obtain Rols. CLIP [18] and ViLD [16] features are extracted from each Rol to establish an open- vocabvoca scene representation. Given a natural language query, we use an LLM to parse the target ("mug') ) propose distractors (e.g., 'plate', "bowl", etc.), and call functions from the localization API (operating on the open-vocabulary 3D scene representation) locate the object.
Note, instruction templates are only used for generating a large-scale evaluation dataset, while our approach can handle open-vocabulary user interactions as demonstrated in real-world experiments (Section 5.3).
4 Approach
This section describes our approach (Figure 3) for multi-round, fine-grained object localization. In our approach, an agent first explores new environments to build a 3D semantic representation of the scene (Section 4.1). Then, an LLM with access to a fine-gratne tiee-graied object localization API (Section 4.2) parses user interactions to propose objects to the user in a multi-round dialog (Section 4.3).
4.1 Been-Vocabulary 3D Scene Representation
As illustrated in Figure 3, our agent first explores the environment using frontier exploration [43]. At each timestep r, RGB observations are processed with an opea-vocabulary object detector (ViLD [16] or OWL-ViT [17]) produce regions- offinteresttttttt (Rols). For each Rol. calculate 3D bounding box by projecting pixel locations corresponding to the object's predicted 2D segmentation mask (ViLD) or 2D bounding box (OWL--VIT) into 3D space using the depth observation and camera matrices W. Then, we use the minimum and maximum projected 3D locations to define an Rol's 3D bounding box. For each Rol, compute CLIP representation using the CLIP visual encoder CLIPs, which is used in addition to the Rol features produced by the object detector. As a result, the 3D environment is represented as list of Rols, where each Rol is represented by 3D bounding box and types of semantic features (CLIP plus ViLD or OWL-ViT), Note: because the object detector is run independently at each timestep, multiple Rols might correspond different views of the same object.
4.2 Fine-Grained Object Localization API
To perform fine-graing fine-grained object localization on 3D semantic scene representation, we need a mechanism to answer three basic questions: (I) Does object X exist? (2) Does object X have the intrinsic attribute Y? and (3) Does object X have an extrinsic relationship with another object z? In this work, we design simple API calls that operate on the open- vocabulary scene representation (Section 4.1) to answer each question. Specifically, we frame the questions as a ranking problem where the goal is to sort the list of Rols. Then, we design algorithms to answer each question that assign each Rol score between 0 and 1. In Section 4.3, we demonstrate how LLM can compose these API calls in response to multi-round FindThis user queries.
1. find_ob jojtects (obj, di distratorrs) Given an object name obj (e.g.. 'mug'), ) our goal is to filter RoIs that do not contain an instance of the requested object, and then sort the remaining RoIs based on confidence score. Open-vocabulary object detectors (c.g., ViLD [16] and OWL-ViT [17I), score RoIs by encoding object names obj with a text encoder (CLIP,) )) and calculating a similarity score with the visual features for each RoL. These scores can be used to sort the RoIs and filter ones with scores below a threshold r. In this work, we dynamically calculate for each Rol based on the similarity score with set of background categories. If an Rol is more similar to background category than the desired object obj it is discarded. We generate background categories on-the-fly by asking an LLM to answer the question, what objects look similar to desired foreground object obj?
sort port_objects (attribute) When an intrinsic attribute attr is provided, the list of Rol must be sorted according to both the object category obj and attribute attr to return object that
5

---

M I rounds of interaction
M= 2 rounds interaction
intrinsic intrinsic
extrinsic intrinsic
$ method
intrinsic
extrinsic
average
uveraae
kop-1
1
lup-1
1up
lop-1
top-5
top-1
upp
lop-1
upp-5
top-1
lep-5
1 ViLD (no parsing) [38]
16.7
28.8
10.5
26.3
13.6
27.6
23.9
37.6
28.9
44.7
26.4
41.2
2 ViLD + basic parsing
23.8
37.3
18.4
26.3
21.1
31.8
29.6
43.9
23.7
44.7
26.6
44,3
3
ViLDD ours
27.8
40.7
31.6
36.2
35.0
31.6
39.5
33.3
43.4
ViLD+CLIP (uo parsing) [38]
33.4
44.3
21.1
20.6
32.7
41.2
50.4
36.8
52.6
39.0
51.5
ViLD+CLIP + basic parsing
38.7
48.5
13.2
28.9
26.0
38.7
45.0
53.3
31,6
42.1
38.3
47.7
ViLD&CLIP + curs
40.3
48.6
47.4
55.3
43.8
$2.0
46.4
52.7
71.1
73.7
588
63.2
OWL-VIT (no parsing) [12]*
27.6
38.3
21.1
34.2
24.4
36.3
32.2
44.2
26.3
44.7
29.2
44.5
OWL-VIT basic parsing
29.2
41.3
18.4
28.9
23.8
35.1
35.0
46.2
36.8
60.5
35.9
53,4
9
OWL-NIT ours
30.2
42.2
36.8
44.7
33.5
43.5
34.8
46.4
57,9
63.2
46.4
54,8
Table I: Main Results. We show results on the 3DOC benchmark by applying our proposed approach on top of three different semantic scene representations: ViLD, ViL.+CLIIP, and OWL.-VIT. We find our method with ViLD + CLIP performs the best. Both parsing and delta vector contributes to improvement of the performance. ("our implementation of CoW [12] in a pre-explored setting instead of object navigation in novel environment)
meets both criteria. To facilitate this, we propose using attribute delta vectors to generate scores for intrinsic attribute queries. Specifically, we define the delta vector as the difference in the CLIP text embedding space between the attribute conditioned object category (attroobj) and the generie object category (obj):
(ab
(I)
Then, we use the magnitude of the projection of the visual features for each Rol onto the delta vector as an attribute score. Finally, we add the attribute score to the original object score produced by the findlob jects () method to calculate score for each Rol.
3. is close (obil, obj2) Given an language query expressing an extrinsic relationship between objects (e.g.. 'Find my mug next to the sink') minimum requirement is the two objects ('mug' and "sink") are close proximity. We mechanize this requirement by comparing the minimum distance between a pair of 3D bounding boxes with a fixed threshold (0.5m in our experiments). Future work may explore additional API calls that capture more refined spatial relationships such as 'left of or 'behind'.
4.3 Instruction Parsing
key challenge in the FindT'his task is parsing the multi-round dialog with a user. While the 3DOC uses templated language, FindThis system should be able to handle arbitrary inputs, reflecting the open-vocabulary motivation for the task. As shown in Figure 3, in this work, we use a LLM with few shot prompting to translate multi-round user interactions into go-find_it () function that uses methods from the localization API (Section 4.2) to return a candidate object Cj to the user. Specifically, we use a small number of example dialog-to- to-function translations in a few-shot prompt enable this capability. Further details including the prompt used in our experiments is presented in Appendix A.
5 Experiments
In this section, evaluate our approach in both simulation using our 3DOC dataset (Section 5.2) and on a robot in real-world kitchen in an office environment (Section 5.3).
5.1 Experimental Setup
Occupancy Map. Our experiments assume that agents are provided with a top-down 2D occupancy map that indicates navigable regions in the environment (Fig. 3). In simulation, the occupancy map is directly generated by the Habitat simulator. However, in our real-world experiments only coarse occupancy map (similar to an empty floor plan) is provided. Specifically, the real-world occupancy map only delineates walls and other immovable structures (e.g., kitchen island) but does not include movable objects such as tables or chairs. The robot uses RGB-D and lidar sensors to update the occupancy map in real-time, which allows navigating to waypoints without collisions.
Exploration Strategy, For all experiments, all agents (including baselines) use the occupancy map to pre-explore the environment using frontier exploration [43]. Specifically, at each timestep, visible regions

---

Fongroun mag
Ponepeurd bow!
Gamadere leat bowt yaaks glees botel tader tounter]
Daearoo [oup Yrugt your bawe lourss
Parsed Desmatore (oop trug. pare! tourte] Ansouss Sall clnwss
Ponepcurd truaf
Image. 640x612
Disracion: Toup: bowd Haane ylass bodal satee, (ount)
Robot
Instruction: Find mug-
mag
Instructions Find bowl Find bowi cansel
Figure 4: Real robot qualitative results. We implemented our approach on a mobile robot and show qualitative results in which our algorithm guides the robot through the environment to find objects specified by complex fine-grained language, c.g, 'an upside down mug' or 'a bowl full of cereal'.
within a fixed radius of the agent are marked as 'explored.' Then, the agent selects the nearest waypoint along the boundary (or frontier) of the explored area to navigate to next. The agent updates the explored regions along the way and stops exploring when all of the navigable regions have been marked 'explored.'
Baselines. We implement variations of our method to compare with two state-of-the-e methods from prior work: MLMap [38] and CoW [12]. Both methods use frontier exploration [43], an open-vocabulary object detector (e.g.. ViLD [16] or OWL-VIT (17)) to build scene representation, and a frozen text encoder to query the representation.
NLMap [38] represents the scene list of Rols produced by ViLD's class-agnostic region proposal network. Each Rol represented with positional information (i.e., object location and size) and visual features from ViLD [16] and CLIP [18]. A frozen CLIP text encoder is used to query the representation.
CoW [12] uses the OWL-ViT [17] detector to process images each timestep during exploration. If the similarity score between a detected region and the query text exceeds a threshold, the CoW agent navigates the 3D location corresponding with that Rol.
These methods as well as other prior works (e.g., VLMaps [39]) use simple mechanism to query scene representations: cosine similarity between image and text features from vision-and-languaar Specifically, language queries (e.g.. FFnd the orange mug.) are feed directly to a CLIP text encoder potentially, after some basic parsing such removing the phrase 'Find the". By contrast, our approach uses an LLM to parse instructions into parts (e.g., 'orange' and 'mug') ') that are used to query for objects.
model.
To compare with these works, we implement two variations of our method: 1) no parsing where the full multi-round dialog used as text query and 2) basic parsing where the dialog is converted into a concise attribute-ojjec phrase (e.g., 'orange mug') ") or referring expression (e.g.. 'mug next the sink'), '). Furthermore, we experiment with creating scene representations using three different feature sets: ViLD [16] features, ViLD [16] and CLIP [18] features (à la NLMap [38]), and OWL-ViT features (a la CoW [12].
Implementation Details. We conduct experiments simulation on the 1,763 total (1,713 intrinsic and 50 extrinsic) evaluation episodes in the 3DOC dataset. We use the 3 [19] for experiments simulation and 540B parameter model from PaL. for real-world experiments.
000 version GPT-
Agent Embodiment. We simulate an agent with an embodiment similar to a mobile manipulation robot produced by Everyday Robots (shown in Figure 1). Specifically, the agent has height of 1.25m with an RGB-D camera with 90' horizontal ofeld-of-view (FOV) and a sensor resolution of 640 x 480. The camera placed at the top of the agent. We assume the agent has access areas in the environment. In real-world settings, such occupancy maps can be pre-collected and then updated on-the-fly fly using depth and/or lidar sensors. We simulate a discrete action space in which the agent can move forward 0.25m and turn left or right by 10,, where the turn actions are in-place rotations of the agent's 's base.
occupancy map indicating navigable
5.2 Results on 3DOC
Table shows results on the 3DOC benchmark by applying our proposed approach on top of three different semantic scene representations: ViLD, ViLD+CLIP, and OWL-VIT. We find that in all cases approach
7

---

(rows 3, 6, and 9) improves top-1 success rate (SR) for both a single round (k = 1) and two rounds of interaction (k=2), Specifically, with ViLD+CLIP our approach improves average top-1 success rate (SR) by +17.8 points (26.0 - 43.8) for 1= over basic parsing (row 5 vs. 6). With two rounds of interaction (k=2), we observe similar gains in average top-1 SR of +20.5 points (38.3 - 58.8 from row 5 to 6).
For intrinsic attribute differeeniation, we find that basic instruction parsing (e.g.. converting the multi-round dialog 'Find my wicker basket. No, my basket also white' to 'white and wicker basker") consistently leads to improvements (row vs. 2, 4 vs. 5. and 7 vs. 8). For example, with the ViLD+CLIP representation basic parsing improves intrinsic top-1 SR by +5.3 points (33.4 + 38.7) for k= and +3.8 points (41.2 - 45.0) for 2 (row vs. 6). These results indicate that these scene representations are sensitive to spurious words unrelated to attributes or objects, and highlight the need for instruction parsing for multi- round interaction.
Furthermore, we find that our approach of using CLIP delta vectors for intrinsic attribute differentiation consistently improves top-1 1 SR (row 2 3, 5 6, and 8 vs. 9). For example, with the Vil ViLD+CLIIP representation intrinsic top-1 SR improves by +1.6 points for k= 1 (38.7 + 40.3) and +1.4 points 2=2 (45.0 46.4) (row 5 vs. 6). With ViLD representations, we observe larger improvements in top-1 SR of +4.0 points (23.8 - 27.8) for k= and +5.4 points (29.6 + 35.0) for k k= 2 (row 2 vs. 3). We emphasize that these scene representations are built on CLIP [18], which was trained to differentiate intrinsic attribute mentions that are likely contained in the WIT dataset [18]. Despite such training, our proposed approach substantially improves over this strong baseline.
For extrinsic differentiation, we find that our proposed approach of separating instructions (c.g.. 'Find the mug on the table.) into its component parts and enforcing spatial constraints (i.e., proximity) significantly improves performance. For instance, with ViLD+CLIP representations, extrinsic top-1 SR improves by +34.2 points (13.2 47.4) for k= (row 5 vs. 6). We observe further gains for multi-round dialogs 2k when an intrinsic attribute is provided as an additional clue (e.g.. 'Find the mug on the table, No, my mug white and yellow:) of +39.5 points (31.6 71.1). These results highlight the benefits of composing multiple capabilities (intrinsic and extrinsic) through the API introduce in Section 4.2 that build on top of open-vocabulary scene representations for fine-grained object localization.
5.3 Qualitative Results on Real Robot
As a last step, we deployed our method on a mobile robot to qualitatively show generalization to a real world setting. As seen in Fig 4, the robot is mobile manipulator, deployed in an office kitchen environment, with a 140x511 512 RGB camera image observation. In each of our runs, the robot successful identifies a 'black upside-down mug' and 'bowl full of cereal' through two rounds of interaction. Specifically, in the first round the robot find different 'mug" or 'bowl', but correctly identifies the target after a second round of interaction. Owing to the scale and generality of the CLIP-based vision backbone, we find this real world application is high performing with no adaptions to the algorithm. See additional details in Appendix C.
6 Discussion
We proposed a novel algorithm called GoFind for fine-grained object localization in complex indoor environments using natural language queries and visual attributes. We addressed the challenge of disambiguating and locating the particular object instance desired through dialog with the user, and exploited visual attributes of the object that may be intrinsic or extrinsic, expressed in an open vocabulary. We demonstrated that the visual common sense learned by large language models enables fine-grained object localization and attribute differentiation in a zero-shot manner. We also provided a new
dataset, 3D Objects in Context (3DOC), for evaluating agents on the FindThis task. Finally, we validated our approach on a real robot operating in an unstructured physical office environment using complex fine-grained language instructions.
Limitations. Our proposed method showed improvements in performance over existing methods on benchmark datasets and demonstrated qualitative results on a mobile robot. However, there are still limitations to our approach. A key limitation is that we use a pipeline of region proposal detection and feature extraction, which might miss some objects, causing inreversible misdetections. Another limitation that we use frontier exploration to do exhaustive search all our experiments, while some dynamic stopping algorithms could potentially provide a shorter path length. We hope that our work provide solid baseline and a dataset for further research in this area and lead to the development of more robust and efficient algorithms for natural language object disambiguation in complex indoor environments.
8

---

Acknowledgments
The Georgia Tech effort was supported in part by ONR YIP and ARO PECASE. The views and conclusions contained herein are those of the authors and should not be interpreted necessarily representing the official policies or endorsements, either expressed or implied, of the U.S. Govermment, or any sponsor.
References
[1] T. Winograd. Procedures representation for data in computer program for understanding natural language. Technical report, MASSACHUSETTS INST OF TECH CAMBRIDGE PROJECT MAC, 1971.
[2] S. Kazemzadeh. V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 787-798, 2014.
[3] P. Achlioptas, A. Abdelreheem, Xia, M. Elhoseiny, and L. J. Guibas. Refertt3D: Neural listeners for fine-grained 3d object identification in real-world scenes. In 16th European Conference Computer Vision (ECCV), 2020.
[4] Z. Chen, A. X. Chang, and M. Nießner. Scanrefer: 3d object localization rgb-d scans using natural language. In Computer Ecion-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Purt XX, pages 202-221. Springer, 2020.
[5] V. Cirik, T. Kin-KKirkpatrick, and L.-P. Morency. Refer360": A referring expression recognition dataset in 360 images. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7189-7202, Online, July 2020. Association for Computational Linguistics. 220g02000..mm acl-main. 644.
URL
https: https://aclatt ftapss//aclaatthllogy
[6] P. Anderson, Q Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. van den Hengel. rision-anddlanguage navigation: Interpreting visually-grundded navigation instructions in real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
[7] A. Ku, P. Anderson, R. Patel, E. Ie, and Baldridge. Room-across-room Multilingual vision-and- language navigation with dense spatiotemporal grounding. arXiv preprint arXhv:2010.07954, 2020.
[8] H. Chen, A. Suhr. D. Misra, N. Snavely, and Y. Artzi Touchdown: Natural language navigation and spatial reasoning visual street environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2558-12247, 2019.
[9] P. Anderson, A. Chang, D. Chaplot, A. Dosovitskiy, Gupta, V. Koltun, Kosecka, Malik, R. Mottaghi, M. Savva, et al. On evaluation of embodied navigation agents. arXiv preprint 1807 2018.
[10] D. Batra, A. Gokaslan, A. Kembhavi, O. Maksymets, R. Mottaghi, M. Savva, A. Toshev, and Wijmans. Objectnav revisited: On evaluation of embodied agents navigating to objects. arXiv preprint 2xXiv:2006.33774 2020.
[11] Z. Al-Halah, S. K. Ramakrishnan, and K. Grauman. Zero experience required: Plug & play modular transfer learning for semantic visual navigation. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages I 17031-17041, 2022.
[12] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Baselines and benchmarks for language-driven zero-shot object navigation. arXiv preprim arXiv:2203. 2et.:222........1... 2022.
[13] A. Majumdar, G. Aggarwal, B. S. Devnani, J. Hoffman, and D. Batra. 7son: Zero-shot object-goal navigation using multimodal goal embeddings. In Advances in Neural Information Processing Systems, 2022.

---

[14] L. Downs, A. Francis, N. Koenig, B. Kinman, R. Hickman, K. Reymann, T. B. McHugh, and V. Vanhoucke. Google scanned objects: high-quality dataset of 3d scanned household items. In ICRA, 2022.
[15] S. K. Ramakrishnan, A. Gokaslan, E. Wijmans, O. Maksymets, A. Clegg. J. M. Turner, E. Under- sander, W. Galuba, A. Westbury, A. X. Chang, M. Savva, Y. Zhao, and D. Batra. Habitat-matteroot 3d dataset (HM3d); 1000 large-scale 3d environments for embodied Al. In Thirty-fifih Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021. URL https:/ http:///ppereeeiieee net
[16] Gu, T.-Y, Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104. 2021.
[17] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenbom, A. Dosovitskiy, A. Mahendran, A. Arnab, M. Dehghani, Z. Shen, al. Simple open-vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.062300 2022.
[18] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PMLR, 2021.
[19] T. Brown, B. Mann, N. Ryder. M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert- G. Knueger, T. Henighan, R. Child, A. Ramesh, D. Ziegler, Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Bemer, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Language models are few-shot leamers. In Advances in Neural Information Processing Systems, 2020. URL https: h/tooe/d////ocednggss. wwwrroceeiinnnn.uiissss....... nwwripe. wwwpurpss.ccppppe///00/ /ttp://ppppp//////pppppp/ /0tp://///////eee/eee/e/ /ile// 1457c0d6bfcb49e
www.B//aacaaaa4aa.....e 2ewee144444aaaappeee pdf.
[20] A. Chowdhery, S. Narang, J. Devlin, M. Bosma, G. Mishra, A. Roberts, P. Barham, H. W. Chung, C. Sutton, S. Gehrmann, P. Schuh, K. Shi, S. Tsvyashchenko, J. Maynez, A. Rao, P. Barnes, Y. Tay, N. Shazeer, V. Prabhakaran, E. Reif, N. Du, B. Hutchinson, R. Pope, J. Bradbury, J. Austin, M. Isard, G. Gur-Ari, P. Yin, T. Duke, A. Levskaya, S. Ghemawat, S. Dev, H. Michalewski, X. Garcia, V. Misra, K. Robinson, L. Fedus, D. Zhou, D. Ippolito, D. Luan, H. Lim, B. Zoph, A. Spiridonov, R. Sepassi, D. Dohan, S. Agrawal, M. Omernick, A. M. Dai, T. S. Pillai, M. Pellat, A. Lewkowycz, E. Moreira, R. Child, O. Polozov, K. Lee, Z. Zhou, X. Wang, B. Saeta, M. Diaz, O. Firat, M. Catasta, J. Wei, D. Eck, J. Dean, S. Petrov, and N. Fiedel. Palm: Scaling language modeling K. with pathways. arXiv preprint 2rXiv:2004.02311, 2022.
[21] Y. Qi. Q. Wu, P. Anderson, X. Wang. W. Y. Wang, C. Shen, and A. d. Hengel. Reverie: Remote embodied visual referring expression in real indoor environments. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9982 1991,9991, 2020.
[22] F. Zhu, X. Liang, Y. Zhu, Q Yu, X. Chang, and X. Liang. Soon: Scenario oriented object navigation with graph-based exploration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12689-12260 12699, 2021.
[23] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. iision-ande languaae navigation: Interpreting visually- navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3674-3683, 2018
[24] J. Thomason, M. Murray, M. Cakmak, and L. Zettlemoyer. Vision-and-dian navigation. In arXiv, 2019.
[25] A. Majumdar, A. Shrivastava, S. Lee, P. Anderson, D. Parikh, and D. Batra. Improving vision-and- language navigation with image-text pairs from the web. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part VI 16, pages 274. Springer, 2020.
[26] J. Krantz, E. Wijmans, A. Majumdar, D. Batra, and S. Lee. Beyond the nav-graph: Vision-and- language navigation in continuous environments. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXVIII 16, pages 104-120. Springer, 2020.
10

---

[27] D. Shah, B. Osinski, B. Ichter, and S. Levine. Lm-nav: Robotic navigation with large pre- vre-trained models of language, vision, and action. arXiv preprint arXiv2200..04429, 2022.
[28] D. S. Chaplot, D. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247 4258, 2020.
[29] A. Wahid, A. Stone, K. Chen, B. Ichter, and A. Toshev. Learning object-conditioned exploration using distributed soft actor critic. In Conference on Robot Learning, pages 1684 1695. PMLR, 2021.
[30] J. Ye, D. Batra, A. Das, and E. Wijmans. Auxiliary tasks and exploration enable objectgoal navigation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16117-16126, 2021.
[31] Mayo, T. Hazan, and A. Tal. Visual navigation with spatial attention. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 16898-16907, 2021.
[32] R. Ramrakhya, E. Undersander, D. Batra, and A. Das. Habitat-web: Learning embodied object-search strategies from human demonstrations at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5173-5183 2022.
[33] S. K. Ramakrishnan, D. S. Chaplot, Z. Al-Halah, J. Malik, and K. Grauman. Poni: Potential functions for objectgoal navigation with interaction-frre learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18890-18900, 2022.
[34] C. Jia, Y. Yang. Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904 4916. PMLR, 2021.
[35] H. Pham, Z. Dai, G. Ghiasi, K. Kawaguchi, H. Liu, A. W. Yu, J. Yu, Y-TT. Chen, M.-T. Luong, Y. Wu, et al. Combined scaling for open-vocabulary image classification. arXiv e-prints, pages arXiv-2111, 2021.
[36] A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1890-1990, 2021.
[37] B. Li, K. Q Weinberger, S. Belongie, V. Koltun, and R. Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03544 2022.
[38] Chen, Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. Ryoo, A. Stone, and D. Kappler. Open-vocabulary queryable scene representations for real world planning. arXiv preprint a7Xiv:2209.09874, 2022.
[39] C. Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. arXiv preprint arXhv:2210.05714 2022.
[40] M. Savva, A. Kadian, O. Maksymets, Y. Zhao, E. Wijmans, B. Jain, J. Straub, J. Liu, V. Koltun, J. Malik, D. Parikh, and D. Batra. Habitat: A Platform for Embodied AI Research. In ICCV, 2019.
[41] A. Szot, A. Clegg, E. Undersander, E. Wijmans, Y. Zhao, J. Turner, N. Maestre, M. Mukadam, D. Chaplot, O. Maksymets, A. Gokaslan, V. Vondrus, S. Dharur, F. Meier, W. Galuba, A. Chang, Z. Kira, V. Koltun, J. Malik, M. Savva, and D. Batra. Habitat 2.0: Training home assistants to rearrange their habitat. In NeurlPS, 2021.
[42] M. Deitke, D. Schwenk, Salvador, I.. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051, 2022.
[43] B. Yamauchi. A frontier- approach for autonomous exploration. In Proceedings 1997 IEEE In- ternational Symposium on Computational Intelligence in Robotics and Automation CIRA 197. Towards New Computational Principles for Robotics and Automation', pages 146-151. IEEE, 1997.
11

---

A Few-Shot Prompt
We use the few-shot prompt shown in Listing for the 3DOC experiments Section 5.2. The prompt uses Python syntax, and begins with "import" statement that indicates functions available from the fine-grained object localization API (Section 4.2). Users queries are converted to lower case, the ending punctuation removed, and prepended with a # (which represents comment Python). Then, the preprocessed query is added to the few-shot prompt after an empty line. The modified prompt is then processed with an LLM. For multi-round dialog, additional interactions or corrections added after the response from the LLM without an empty line separator (as demonstrated in the second and third few-shot examples).
from utils import object,, sort_objects, is_close
find the wooden chair
def go_find_it(): chairs
find_objects ( ('chair', distractors=("ssfaa,, bench'' 'stool', 'desk') chairs = sort_abjects (chairs, attributes- I' wooden return chairs]0]
find backpack with white polka dots on it def go_find_it()
backpacks
find_objects (' backpack', distractore=(pplllor 'jacket', 'shirt']) backpacks sort_ sort_objects (backpacks, attributes- white polka dots'l) return backpacks [0]
no, my backpack is also yellow
def go ()
backpacks
= find_objects (' (' backpack', distractors=I'ple p'plloo'' "jacket,, 'shirt') ))
backpacks
sort_objects (backpacks, attributes- (rrreess[[whhiii ['white polka dots', 'yellow"] 1)
return backpacks [0]
find the pants on the dresser def go_fto__ooooooo () f)
pants find_objects
('pants', distractore=(' shirt', 'socks', 'shocs', 'dress') dressers find_objeets
('dresser', pants is_close (pants, dressers) return pants[0]
(otcractoos="bbookttt
'bed', 'dek'' 'chair']) ])
no, my pants are also red
def go go_ find find_it(): (t
pants
Lind_objects
('pants', distractors- (hirtt', 'socks', 'shoes', 'dress'))
pants sort_objects (pants, attributes ('reee red']) dressers find_objects
('dresser', distractors=("bookthe 'bed', 'desk', 'chair') ])
pants = is_close (pants, dressers) return [0]ts[0]
find the apple next to the microwave def find_it():
apples = find_abjects
('apple', distractors- p'a''', "tomato', 'orange', (bowl'])
microwaves find_objects
('microwave', distractors= dishwasher', apples = is close (apples, micrewaves) return apples[0]
"diaattors 'sink', 'zeirigerator", 1)
Listing 1: Few-shot prompt used for experiments in simulation.
12

---

Table 2: 3DOC Statistics
Num of Object Categories
10
Num of Unique Objects
72
Num of Unique Intrinsic Attributes
53
Num of Unique Scenes
100
Num of Unique Scene Layouts
500
Num of Extrinsic Attribute Episodes
50
Num of Intrinsic Autribute Episodes
1,713
B 3DOC Statistics
Table provides additional details the composition of the 3DOC dataset.
C Qualitative Examples
video presenting qualitative examples of our GoFind agent executing the FindThis task in both simulation and in the real-world is provided in the supplemental material.
D Dataset and Code Release
The 3DOC dataset and source code to reproduce the results presented in Section 5.2 will be publicly released.
13


Reviews:
[{"review_id": "t-j7HB5cMv", "reviewer": "Reviewer_rPQj", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes the GoFind algorithm, which allows a robot to be instructed with a fetching task. The algorithm pipeline detecting ROI of objects and classify them given the parsed input sentence. The authors also propose a dataset 3DOC on which their algorithm is validated.", "summary_of_recommendation": "Although the task addressed by the authors is highly relevant and some interesting ideas are proposed (use of delta vectors), the following\n limitations unfortunately hinders the paper:\n *template-based sentences, \n *unfair comparison with SOA, \n *lack of sentence diversity, \nTherefore reviewer cannot recommend yet this paper for CoRL.\n\nUpdate:\nMost of the comments have been convincingly addressed by the authors. Nonetheless, the reviewer is not fully convinced by the parsing approach given the simple template sentences. The authors implies that LLM parsers work with natural sentences but this is not shown nor evaluated in the paper. A quick review of the literature suggests that LLM are not 100% accurate:\n\n[1] On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codex\n\n[2] ZEROTOP: Zero-Shot Task-Oriented Semantic Parsing using Large Language Models\n\n[3] BenchCLAMP: A Benchmark for Evaluating Language Models on Semantic Parsing", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1) The authors show in Table 1 results showing the superiority of parsing-based method over non-parsing ones. The authors find this comparison unfair. From the reviewer understanding, the authors only use sentences  \"Find X\", for which a parser would most likely be 100% accurate. With natural language, parsers would not be 100% accurate and error would accumulating in the proposed pipeline.\n\n2) The accompanying video shows that the proposed method is able to handle free-from natural language. So the reviewer does not understand why the authors created a benchmark with only template-based sentences. Such a benchmark could be solved with rule-based approaches. There is not enough linguistic diversity.\n\n3) Please provide either in the paper or in the annex, samples of sentences of the Go_find task which were used in experiments to demonstrate the ability to handle open-vocabulary. So far, the reviewer only find sentences \"Find X  \" which support the comments 1) and \n2).\n\n4) Lines 194-196, the authors mention background categories for calculating $\\tau$.  This part is unclear for the reviewer.  if an LLM is used to specify similar objects to the foreground object (Line 196), why are they classified as background objects? \n\n5) The authors proposed the delta vector vector to score each of object ROI. The reviewer finds this approach interesting instead of the usual cosine similarity. Unfortunately the description of the method is unclear and does not appear in any of the figures. E.g. Why  is the difference make only in the CLIP text embedding space? What does this difference represent? \n\n6) Figure 3 is too small and barely readable. As such it does not provide any added value to the paper. The same comment is valid for Figure 4.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The problem addressed in this paper is very interesting and relevant for service robots. Furthermore, the authors have shown the relevance of their approach in real world experiments. Nonetheless the reviewer has some concerns regarding:\n- The clarity of the paper. The proposed method is somehow unclear (delta vectors) and no supporting figure is provided\n- The use of templates/parsing. Although the authors provided comparisons to benefits of using a parser, the reviewer is not convinced by such an approach (cf. rebuttal questions)\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "dHsZEUL9sB", "reviewer": "Reviewer_qZX4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a task of interactive language-based object search in a navigation environment. To that end, it defines an alternating dialogue where a human sends progressively more specific queries and an agent replies in candidate pictures. The paper introduces a dataset which includes scanned scenes with objects placed inside, chosen to include multiple objects from the same category with potentially distracting properties. It then proposes a method for addressing this task where a 3D cached set of object detections from an open-vocab vision-language detection model is processed with a large language model to rank the candidates based on parsed attributes.", "summary_of_recommendation": "I think the method and dataset are thorough and contributes a useful dataset for a potentially fruitful direction of human-in-the-loop object localization. Despite some concerns and questions I've listed above, my overall rating is weak accept.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Questions\n- Why is there such an imbalance between intrinsic and extrinsic properties in the dataset? \n- What are the specific 10 categories, and 72 objects? images in the appendix or an exhaustive list would be very helpful\n- Only 2 rounds are considered, why not more? ", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n- I like the task and interactive human-centric setting proposed and the formulation provided for the FindThis task is sensible\n- Prompting an LLM for distractor objects seems like a good way to calibrate similarity outputs from CLIP embeddings.\n- The open-source is constructed to include diverse objects and valid distractor objects in each scene\n\nWeaknesses\n- Real experiments don't have as much complexity as simulated, for example one round only has 2 objects in it (laptop with/without sticker), so the agent will always succeed within 2 rounds. Similarly, there are only 3 mugs so a randomly performing agent would also achieve quite a high success rate.\n- Success being counted as >10% of the cropped image containing the object seems like a weird metric, it seems to me the metric should also include the % of the target object included in the crop, with some high threshold required for success (say >80%). Otherwise, a crop could be clearly centered on an adjacent object and only include part of the target object, but still be counted as correct. Does this ever happen in the experiments provided? Why is such a low threshold for performance chosen for the success metric?\n- The 3D representation used in GoFind includes lots of redundant views of objects and no way to associate them as the same object. \n- This work implicitly assumes that every query describes an object that actually exists in the scene, since the robot has no mechanism for responding \"I can't find it\", which is a natural response to a query. This should be discussed in the limitations and ideally mentioned in the problem statement, or if not evaluated in experiments. Additionally, having examples in the 3DOC dataset where queries do not exist would be helpful as a benchmark.\n\nComments\n- In the 3 considerations presented, I feel \"natural language\" would be a better term than \"open vocabulary\", which sometimes has some connotations of fixing a set of supported terms before system deployment, such as how \"open vocabulary\" object datasets still have a restrictive set of object descriptions. On page 2 line 44, \"open-vocabulary, natural language\" is a bit redundant in my opinion, wouldn't any true natural language system be inherently open vocab?\n- The conversation is described as a \"multimodal dialogue\" (page 4 line 112), which I feel is a misleading personification since the human agent is only able to respond with text, and the LLM only with an image. It is a dialogue which includes multimodal elements, but isn't a full 2-way multimodal dialogue.\n- I feel there should be a short section in RW about using code generation from LLMs as an interface for an agent's actions, for example ViperGPT, Code as Policies, and ConceptFusion to name a few. I'm not sure of related work on human-in-the-loop interactive task completion with an LLM, but I'd be surprised if there wasn't (ChatGPT!)\n- There are some additional citations on 3D semantic/language maps that could make the section more complete: ConceptFusion, CLIP-Fields, LERF, Distilled Feature Fields, Panoptic Lifting.\n- The design decision of using CLIP delta vectors for object attributes is interesting, but more motivation for this method should be provided in the description. Also, the main text describes which lines in the table represent the delta ablation, but the table only refers to \"basic parsing\". I feel the table could be made clearer, possibly by changing \"+ ours\" to \"+ delta\"\n- page 6 line 205, \"an language query\"-> \"a language query\"\n- A discussion of failures should be included given some of the ~50% success rates in simulated scenes.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "8vP5juhJD0", "reviewer": "Reviewer_r5uR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper focuses on finding objects in physical environments when provided with underspecified language commands. The authors first introduce a new environment for testing physical object identification from language commands as well as a dataset for evaluation. It is a household environment with objects that have realistic properties and collocation statistics. They then introduce a system that attempts to tackle the task. The proposed GoFind system is a pipeline system consisting of an object detector, an object localizer, and an instruction parser. \n\nThe system is able to find objects with a greater success rate than the baseline systems. The authors also show the benefits of having an interactive setting, where a user can provide additional information to a model if it fails to find the correct object. ", "summary_of_recommendation": "Based on the paper's strengths, I think it should be accepted. The proposed dataset and system address a gap in current research on interactions with language. Tackling ambiguity will be an important component to scaling up systems that interact with language and enabling use by real people. By framing the disambiguation process as a dialogue (at least conceptually) I think this paper is aiming in the right direction, and takes some important initial steps in that direction. My main concerns are described in the weaknesses section. With additional positioning w.r.t. past work on ambiguity the paper and more hedging around the framing of the interaction as a dialogue, I would likely have recommended a strong accept. ", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "(In addition to the comments in the weaknesses)\n\n1. I would argue that there is a distinction to be drawn between ambiguity and underspecification, and that ambiguity is a particular type of underspecification, but not all underspecification is ambiguity. How do you define ambiguity, and does your definition distinguish between ambiguity and underspecification? \n", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths \n1. **Novel task/dataset**: As the authors point out, natural language is often ambiguous. Many current frameworks ignore this fact, and the FindThis task seeks to correct the oversight. The construction of the dataset is solid, especially the attention to distractors and the distinction made between intrinsic and extrinsic attributes. \n\n2. **Modeling contributions**: As far as I can tell, the integration of delta vectors into the object ranker is novel. The factorization of the task into discrete steps is sensible and the way the steps (location, ranking, measurement) are implemented into an API makes the method flexible w.r.t. the parser. \n\n3. **Strong experimental results**: Their model shows strong improvements over relevant baselines. The baselines ablate the relevant parts of the model. \n\n4. **Real-world demonstration**: the authors include a convincing real-world demonstration of their system. \n\n\n## Weaknesses \nThe first weakness is my only major issue with the paper; the second is more minor. \n\n\n1. **\"Dialogue\" is a stretch**: I have concerns both about the setting and the way it is pitched. Firstly, the fact that the dialogue is triggered by a localization mistake hurts the proposed method's generalizability. This is fine for simple actions like finding an object, but what if the agent were to also have state-changing actions. In this case, waiting for the agent to make a mistake before correcting it would be untenable. A more general system would detect underspecification *before* acting and ask for clarification. \n\nThis ties into my second concern, which is with the pitch of the interaction as a dialogue. The dialogue here is really a scripted 2-stage interaction and doesn't involve language generation from the model side. A more realistic interaction might involve a multiturn dialogue that takes place over the course of the agent performing a series of actions. \n\n2. **Hard-coded/limited agent**: There are several components in the modeling pipeline that seem hardcoded. Firstly, the exploration strategy is fixed and it is assumed that the agent has explored the environment beforehand (this is mentioned in the Limitations section). Additionally, the script followed by the localization API seems pretty much fixed and has limited breadth. As far as I can tell, program diversity is limited. From the prompt in Appendix A, it seems like there are only 2 types of programs (find_objects -> sort_objects and find_objects -> find_objects -> is_close). This makes parsing task is quite simple and means that it could be handled by intent indentification and slot-filling rather than a full-blown API. Real language, on the other hand, is likely quite a bit more complex and requires a larger set of primitives. \n\nAnother limitation that reduces the method's generalizability is the lack of complex spatial relationships and the lack of compositionality. It is assumed that there is only one relationship that objects can have (near). The authors leave other relationships for future work. Even focusing just on \"near\", their treatment is overly simplistic. Near is both gradable and context dependent, meaning that a) the meaning of near is vague and it isn't clear what is and isn't near; and b) \"near\" means different things depending on what the objects are. \"Near\" in \"my house is near the airport\" doesn't mean the same thing as \"the mug is near the apple\". 0.5m may be \"near\" on a kitchen table, but it is far if the objects are located inside a fridge. \n\nThe intrinsic and extrinsic relations could also be composed, leading to a more realistic and challenging scenario. For example, the templates could be extended to include utterances like \"Find the orange mug near the large sink\", which would also introduce more complex programs. \n\n\n(The following is a suggestion rather than a weakness): The authors point out that certain objects have typical environments. Past work has found that this kind of commonsense information is often encoded in LLMs, so the system could include a planning stage in which the agent determines likely locations to explore. \n\n3. **Missing related work**: The authors have related works sections for referring expressions, language-conditioned navigation, and 3D semantic mapping, but none for ambiguity and underspecification in robotics, which has been studied in the past. Specific work that also has underspecified instructions (just from a cursory Google scholar search) includes:\n- https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9197315\n- https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8392364\n- https://dl.acm.org/doi/pdf/10.1145/3382507.3418863\n- https://journals.sagepub.com/doi/full/10.1177/0278364918760992\n\n", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning."}, "extra_text": {}}, {"review_id": "1sART7lamF-", "reviewer": "Reviewer_pqhb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposed a method for robots to search for the location of specific objects using vision and language models.\nThe proposed method is divided into two phases: a phase in which objects in the environment are investigated by search using an open vocabulary object detector, and a phase in which the robot's execution command is output using LLM from human commands.\nIn the experiments, several open vocabulary object detectors were compared using the Habitat-Matterport3D dataset.\n", "summary_of_recommendation": "This paper incorporates the latest machine learning techniques into an important robotics problem. The focus of the problem is very interesting.\nThe content seems appropriate for this conference.\nHowever, perhaps due to the page limit of the conference, the richness of the content is compressed and some details are omitted.\nIn addition, more detailed and extensive experiments and comparisons are required to demonstrate the proposed method.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "How did the robot acquire a map of its environment? In the proposed method, the robot seems to search for unexplored areas of the environment using the Frontier approach. However, I did not find any description of how the map was created in the experimental section. Were the maps created by SLAM? The Frontier approach should be applied to unexplored areas of the map. Therefore, it was my understanding that the map was constructed during exploration. The author may be able to clarify on the above.\n\nThe author wrote as if he had compared NLMap and CoW in an experiment. However, I could not find any results for these methods. In fact, it seems that some open-vocabulary object detectors and with/without parsing were compared, not with these methods.\nTo avoid this confusion, the authors could clarify their description of Baselines.\n\nThe format of the citation of the paper is not consistent.\nAuthors need to closely check and correct all citations with respect to their descriptions.\nAlso, there are too many arXiv preprints in the citations. To some extent, this is unavoidable due to the nature of this industry. However, arXiv is not peer-reviewed and can undermine the credibility of a paper. Therefore, I encourage authors to make an effort to reduce arxiv preprint citations as much as possible.\n\n\n\nWhat is \"background categories\"? Details on this are unclear.\n\nFigure 1.\nIt is difficult to tell what the image sequence and arrows indicate.\nSome of what the figure shows is unclear.\n\nThe success rate is defined as SR in section 3.2 and is also redefined in section 5.2.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The research topic is a very interesting application of VLM and LLM to real-world tasks in robotics. A unique feature of this research is the disambiguation based on multiple dialogs with the user. Much of the content of this paper has been compressed, and some details seem to have been omitted. I present my questions and concerns below.\n\nThe specificity, open-vocabulary, and interactive mentioned by the author are all important for mobile service robots.\nHowever, open-vocabulary can be solved by simply using LLM. It seems to me that this is a problem that has already been solved. The proposed method does not seem to solve open-vocabulary in a new way. Thus, there were no big surprises for me.\n\nThe proposed method is able to deal with instruction ambiguity and wrong requests by Multi-around interaction.\nHowever, having multiple linguistic interactions in every instruction may be burdensome for the user.\nCould the proposed method use the previous interactions if the same ambiguous instruction is repeated the next time as the first?\nIn other words, it does not seem to allow the memory and knowledge exchanged once to be used the next time.\nSome action or consideration of the above may be needed.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers appreciate the novelty and formulation of the approach and the interactive human-centric setting, but questioned the experimental setup and evaluation measures, and ask for further discussion of the limitations of the approach.
After the rebuttal and author discussion phase, the reviewers agree that this paper can be accepted for CoRL. The reviewer suggestions should be thoroughly addressed in the camera-ready version of the paper.
Decision: Accept (Poster)