Semantic Mechanical Search with Large Vision and Language Models
Authors: Satvik Sharma
Huang Huang
Kaushik Shivakumar
Lawrence Yunliang Chen
Ryan Hoque
brian ichter
Ken Goldberg
Venue: CoRL 2023 Poster (2023)
Keywords: Vision and Language Models in Robotics
Mechanical Search
Object search

Abstract: Moving objects to find a fully-occluded target object, known as mechanical search, is a challenging problem in robotics. As objects are often organized semantically, we conjecture that semantic information about object relationships can facilitate mechanical search and reduce search time. Large pretrained vision and language models (VLMs and LLMs) have shown promise in generalizing to uncommon objects and previously unseen real-world environments. In this work, we propose a novel framework called Semantic Mechanical Search (SMS). SMS conducts scene understanding and generates a semantic occupancy distribution explicitly using LLMs. Compared to methods that rely on visual similarities offered by CLIP embeddings, SMS leverages the deep reasoning capabilities of LLMs. Unlike prior work that uses VLMs and LLMs as end-to-end planners, which may not integrate well with specialized geometric planners, SMS can serve as a plug-in semantic module for downstream manipulation or navigation policies. For mechanical search in closed-world settings such as shelves, we compare with a geometric-based planner and show that SMS improves mechanical search performance by 24% across the pharmacy, kitchen, and office domains in simulation and 47.1% in physical experiments. For open-world real environments, SMS can produce better semantic distributions compared to CLIP-based methods, with the potential to be integrated with downstream navigation policies to improve object navigation tasks. Code, data, videos, and Appendix are available here.

Semantic Mechanical Search with Large Vision and Language Models
Satvik Sharma" Huang Huang",, Kaushik Shivakumar' Lawrence Yunliang Chen', Ryan Hoque', Brian Ichter², Ken Goldberg'
Abstract: Moving objects to find fully-occluded target object, known me- chanical search, is a challenging problem in robotics. As objects are often orga- nized semantically, we conjecture that semantic information about object relation- ships can facilitate mechanical search and reduce search time. Large pretrained vision and language models (VLMs and LLMs) have shown promise in generaliz- ing uncommon objects and previously unseen real-world environments. In this work, we propose novel framework called Semantic Mechanical Search (SMS). SMS conducts scene understanding and generates semantic occupancy distribu- tion explicitly using LLMs. Compared to methods that rely on visual similarities offered by CLIP embeddings, SMS leverages the deep reasoning capabilities LLMs. Unlike prior work that uses VLMs and LLMs as end-to-end planners, which may not integrate well with specialized geometrie planners, SMS can serve plug-in semantic module for downstream manipulation navigation policies. For mechanical search in closed-world settings such shelves, compare with geometric-baset planner and show that SMS improves mechanical search per- formance by 24% across the pharmacy, kitchen, and office domains in simulation and 47.1% 1% in physical experiments. For open-world real environments, SMS can produce better semantic distributions compared to CLIP-based methods, with the potential be integrated with downstream navigation policies to improve object navigation tasks. Code, data, videos, and Appendix are available here.
Keywords: Vision and Language Models, Mechanical Search, Object Search
Introduction
Mechanical search, where a robot manipulates objects and/or navigates to find fully occluded target object [1, 2], is challenging robotics problem. Prior work has shown success in revealing the desired object by manipulating the occluding objects [3, 4, 5], obtaining new observations after rotating the camera [6), or navigating to new locations [7, 8). However, generalization to unseen environments remains challenging due to the numerous long-tail objects present in the real world.
Environments are often organized semantically, for example, toothpaste often stored in home bathroom near toothbrushes. In this paper, we explore how LLMs can provide such semantic lationships facilitate mechanical search. Large vision and language models (VLMs and LLMs)) show promise for such relationships they are pretrained on internet-scale data which empirically captures knowledge of semantics. A large body of prior work has shown that these models can pro- vide good visual representations [9, 10, 11, 12, 13], ground language instructions [14, 15, 16, 17, 18, 19, 20], and serve as planners out of the box [21, 22, 23, 24, 25, 26). CLIP [27] is a commonly-used interface to associate vision and language, and many works [7, 8, 28, 29] use it to build semantic scene representations and show improved performance on object query and navigation tasks. How- ever, while informative, the dot product of CLIP text and image embeddings lacks deep reasoning
"Equal Contribution
AUTOLabb at University of California, Berkeley Google Brain
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

Inputs& Query
Scone Understanding
Generating Semantic Distribution
Mechanical Search Policies
Find the Irottbrush
Known Object Liet
Iarond steast goes nest to
Shelf Searching Algoriths
Object Detection
Mase Labela
Model
1D Distribution
OR
Ofjectt Liet
Navigation Algoritim
Smart Oiaping
20 Distribution
RGB Image
Maak Labala
Alfilly Monria
jolliow,
Figure 1: Overview of Semantic Mechanical Search (SMS), SMS accepts as input a scene image and a desired target object. It applies an object detection, or segmentation algorithm combined with captioning as necessary when object lists unavailable. SMS then uses LLM to compute affinities between detected objects the target object, and it uses these affinities output a semantic occupancy distribution which can be used for downstream mechanical search policies.
capabilities and sometimes behaves as bag-of-words [30]. As such, CLIP is most useful for local- izing objects that are already visible somewhere in the scene or map [13, 8], a property that many open-vocabulary object detectors build on [31, 32, 33]. When the target object is fully occluded, CLIP alone may not provide enough clues about potential target object locations.
LLMs demonstrate advanced reasoning and planning capabilities [34]. Many prior works [25, 35, 7] use VLMs and LLMs as end-to-end planners for both perception and planning. While such paradigms benefit from the semantic reasoning abilities of LLMs, they do not handle additional information that cannot be easily expressed through language and may not integrate well with other domain-speciffe policies. For example, for mechanical search on shelves, the geometric properties of objects provide valuable cues for identifying potential target object positions, and various algorithms have been proposed for handling uncertainty and planning ahead [3, 4, 5J. Likewise, for object navigation, prior research has explored exploration and navigation strategies that are independent of semantic understanding [36, 37, 38, 39]. As such, decoupling semantic reasoning and geometric planning may allow flexible integration with task-specific modules for various downstream settings.
We propose Semantic Mechanical Search (SMS), which generates an explicit intermediate repre- sentation, semantic occupancy distribution, as plug-in semantic module for existing mechanical search algorithms. This distinguishes it from prior work where VLMs and LLMs serve end-to- end planners doing both semantic reasoning and action planning. With the goal of adding semantic reasoning to existing search policies, we study two questions: (1) Can semantic distribution facil- itate mechanical search? (2) What is the best way to generate this semantic distribution? For the second question, we hypothesize that translating image features into language features (with VLMs) first and then extracting semantic distributions from only language features (with LLMs) can outper- form VL.M-only methods that most current works use. We show that, rather than burdening VLMs (e.g. CLIP) with both object detection and reasoning, decoupling these two tasks leads to better results the LLM language feature space is better at capturing semantic relations. For the first question, we show SMS can be easily integrated with a geometric shelf searching algorithm [3] to improve performance for closed-wond world environments such as pharmacy shelves with known object lists. In cloled-world settings where object lists are available, SMS uses an open vocabulary ob- ject detection model [40] refined with Optical Character Recognition (OCR) to identify objects. In open-world settings where object lists are unavailable, SMS combines segmentation [41] and image captioning [42] to generate object mask descriptions.
This paper makes three contributions:
1. SMS, a novel framework that uses pretrained VLMs and LLMs to synthesize semantic occupancy distributions that can be easily integrated to enhance mechanical search policies;
2. A way of using LLMs augment the reasoning capabilities of VLMs for generating better semantic distributions, with evaluations of semantic distribution quality in both closed- world and world settings.
2

---

3. Closed-world experiments for mechanical search on shelves showing that SMS improves a geometric-based planner by 24% across the pharmacy, kitchen, and office domains in simulation and 47% in real, and preliminary study of SMS in open-world settings.
Related Work and Preliminaries
2.1 Mechanical Search
Mechanical search [1. 2] refers to broad class of robotics problems searching for occluded and out-of-view objects via manipulation and navigation. In the former case, bin [1] and shelf en- vironments [43, 44, 45, 46, 47] are widely studied, where intelligent estimation and manipulation planning based on possible locations of the hidden target object significantly affects the search effi ciency. Many prior work uses only geometric priors [1, 4, 5, 3, 48]. A number of authors have also explored using semantic context object information [49]. Kollar and Roy [50] obtain co-occurrence statistics from web-based ontologies and Wong al. [51] extend the approaches occluded target objects. Kurenkov et al. [2] propose hierarchical model to integrate semantic and geometric infor- mation and learn in simulation. However, they manually craft semantic categories, which are also sparse and can not accurately and scalably reflect real-world distributions. Instead, we harness large pretrained models to extract open-vocabulary semantic information zero-shot.
There are many types of navigation tasks, such as point goals [52, 53, 54], image goals [55, 56], and object goals [57, 58). Finding out-of-view objects is object goal navigation task, and the problem is also known active visual search [59, 60]. Classical geometry-basee methods typically first build map [61, 62] and then perform planning [63, 64). Learning-base based methods typically use reinforcement learning trained in simulation [65, 57, 66, 53, 67, 68, 69, 70, 71]. through YouTube videos [58], or by querying the Internet [72] to learn semantics and efficient exploration strategies. Recently, many works have explored using LLMs and VLMs out-of-the-box for semantic scene un- derstanding [73] and zero-shot object navigation, which this work belongs to. The most common strategy is to CLIP features [27] obtained from pretrained open-vocab detectors [74, 75] in VL-Maps [7] and OpenScene [76] or from region proposals models as in CLIP-Fields [77], Con- ceptFusion [78), and NLMapssSayCan [79] and fuse them into 3D point clouds or implicit represen- tations [30]. The constructed representations can then be used for open- vocebubcayl target queries to locate the object and perform navigation. Gadre et al. [8] propose a family of methods to adapt CLIP and open-vocabulary models to localize target objects. Through a systematic comparison, they find OWL-VVT detector [80] works best, followed by patchifying images to obtain separate CLIP em- beddings and compute similarity with text embeddings. In this work, instead of using the similarity of CLIP embeddings to construct relevancy maps [81],
the LLM feature space to explicitly reason about the object's 'ss semantic relationships and show that SMS outperforms these two methods.
2.2 Natural Language in Robotics
Grounding natural language instructions is widely-studied problem in robot navigation [82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93], human-robot interaction [94, 95], and is increasingly studied in the manipulation literature [96. 97, 98, 99]. While classical methods commonly rely on semantic parsing and factor graphs [87, 91, 94], end-to-end learning and leveraging pretrained models are now the most popular paradigms thanks to advances deep learning and LLMs. Examples include imitation learning [100, 19, 16, 18, 20, 101]. caaguage-conditionn. reinforce- ment learning [102, 103, 14], and online correction of robot policies through language feedback [104, 105]. In particular, pretrained image encoders and open-vocabulat object detectors have en- abled generalization to novel object queries at test time [25, 24, 13]. In this work, we also take in novel object targets specified using natural language, but since the target objects are not visible in the scene, the robot instead needs to detect and localize other objects and reason about their rela- tionships. This is particularly challenging in an open-world environment when the set of possible objects unknown, making object detectors significantly less accurate. Our method shares similar- ity with HOLM [6], which uses an LLM to hallucinate nearby objects in partially observable scenes based semantics computed from affinity scores. However, it relies object list and only considers camera adjustment actions in simulation. We relax this assumption of accessing object
language-condition
3

---

lists 17, 76, 6], propose pipeline for generating object labels without access to any object lists, and generate semantic distributions for open-world environments.
Many studies have also used LLMs as a planner by letting them break down tasks through step- by-step reasoning [22, 23, 24, 25, 35, 106] or directly write code [21, 26). While these end-to-end planning paradigms benefit from the deep reasoning abilitics of LLMs, it's not straightforward to incorporate additional non-language information and integration with domain-specifff policies. The latter is particularly valuable when the task is more complex and a flexible generalist LLM can ben- efit from specialized searching and planning algorithms developed by the robotics community. We propose decoupling semantic reasoning and geometric planning: rather than directly output prim- itive instructions from image observations, SMS uses LLMs' semantic reasoning from its feature space into a semantic distribution that specialized planning and manipulation policies can use.
2.3 Occupancy Distribution
An occupancy distribution indicates the probability of each pixel in the image containing the target object's 's amodal segmentation mask [1]. Prior works [1, 3, 4, 5] have utilized geometric information to generate spatial occupancy distributions by considering object geometries and camera perspective (e.g., tall target objects cannot be occluded by short objects and objects in the center of an image occlude more areas) facilitate the search. Huang al. [4] propose the LAX-RAY system, which uses neural network predict the spatial occupancy distribution. A greedy policy called Distribu- tion Area Reduction (DAR) uses this distribution to greedily reduce the overlap between objects and the distribution the most. SMS generates the occupancy distribution using semantic information, which can then be combined with the LAX-RAY spatial distribution for downstream search.
3 Problem Statement
We consider a partially observable environment that contains target Or and N other objects {O,.... ON}. We assume the scenes in the environment are semantically organized, meaning that the starting state of the environment is sampled proportionally to their approximate likelihood of occurrence in the real world. With this assumption of semantically organized scenes, the tar- get object location probability is proportional to object pair affinities. States st S consist of the full geometries, poses, and names of the objects in the scene at timestep f, and observations Ve y = R xxx3 are RGB images from robot-mounteds RGB camera at timestep t. Given the name of the target object and the observation yt, the goal is to generate a useful dense occupancy distribution that encodes semantic affinities (with respect to the target object).
4 Semantic Mechanical Search
We propose SMS, a framework using VL.Ms and LLMs to create a dense semantic distribution be- tween a scene and the target object to used for downstream tasks. Fig. visualizes the pipeline. SMS first uses VLMs to perform scene understanding by creating mask-label pairs to densely de- scribe all image portions. It then uses an LLM to generate affinity scores between the labels and the target object. We spatially ground these affinities using the labels' corresponding masks. In this way, we densely represent the affinities between target object and all parts of a scene using an LLM. SMS can be applied to two common situations: 1) closed world where all objects in the scene are subset of known list and 2) an open world where some objects in the scene are previously unseen.
4.1 Scene Understanding
The goal of scene understanding is to generate mask-label pairs that characterize the scene.
Object Detection OCR When an object list is available, tion model, specifically ViLD [40], to obtain object segmentation masks and labels from an RGB image. We also find using Keras Optical Character Recognition (OCR) [107] can improve the qual- ity of the ViLD object detection, by increasing the mean average precision (mAP) on 100 pharmacy scene images from 2.4 to 28.9 (Table 5). Details are in the Appendix.
open vocabulary object detec-
Crop Generation Image Captioning When object list is not available, many open vocabulary detectors such ViLD cannot be used. We instead create image crops and use VLM for crop

---

captioning, specifically BLIP-2 [42], to convert object crops to their text descriptions. We ask for the dominant objects in each crop for less noisy captions. We generate crops that are both object-centric (using Segment-Anything (SAM) [41]) for better object boundaries in the semantic distribution and general multiscale, overlapping crops that help encode large-scale semantic information.
4.2 Creating the Semantic Distribution
We consider two ways to use language model to generate affinity scores for the semantic distri- bution. (1) SMS-LLM: We iterate over all the mask-label pairs and query the LLM with a specific prompt: "I see the following in a room: {label). This is likely to be the closest object to {targer object)". This prompt directly represents the probability the target object given we see the label. Since object labels are contained within the prompt, we do not need to normalize to account for the prior. A similar prompt with the label and the target object switched would also provide affinity scores between objects but would then have to be normalized to account for that object's s prior. The affinity score for the target object with each label is the completion probability for the tokens that represent the target object. We generate semantic distribution from these affinity scores and de- tected objects. The semantic distribution models the probability of the target object occupying each location, which we approximate to be proportional to the affinity score between the target and the object closest to that location. To account for noise, we apply spatial smoothing using a Gaussian kernel with std σ. More details are in the Appendix. (2) SMS-E: An alternative method we explore is to use a language embedding model (e.g. OpenAl Embedding Model (108)) to get embeddings for all labels and the target object, and obtain affinity score between each label and the target object through the dot product between these vectors.
When there are no object lists, the Crop Generation + Image Captioning pipeline described Sec- tion 4.1 can contain many incorrect or hallucinated labels, making the distribution noisy. To mitigate this, we use CLIP to verify the captions and not for any semantic reasoning. Specifically, we com- pute the CLIP dot products between the image crops and the generated labels and weight the affinity scores by these relevance scores. To produce the final semantic distribution, each pixel receives the average of the weighted affinity scores of all the masks it belongs to. We find that averaging across multiple overlapping masks also helps reduce noises in the absence of object lists.
4.3 Combining with Mechanical Search Policies
Environments We consider semantically organized shelves with objects from a known list. For mechanical search on shelves, the robot needs to manipulate objects in the shelves to reveal the occluded target object using pushing and pick-and ank--aee actions. The goal is to minimize the number of actions taken to reveal the target object. Additional details are in the appendix. We use SMS as plug-in semantic module for existing search algorithm, LAX-RAY [4], by multiply- ing the semantic occupancy distribution with learned spatial distribution that LAX-RAY generates based on geometry. We then use the DAR policy [4] to perform mechanical search. Since the search in cluttered environments requires manipulating other objects, once the search begins, the shelf may become semantically disorganized. As such, at each step a rollout, SMS computes the semantic distribution using the object locations where each object was first discovered.
Open-World World Environments We consider large room spaces, with semantic diversity (rooms of an office, home, aisles in a grocery store, etc.). We do not perform any manipulation in this setting and explore a downstream heuristic navigation policy that terminates when the object is within view. Given a starting position, the policy moves a fixed distance towards the highest affinity region in the image. Afterward, it takes four new images by rotating in place. We first select the desired view direction amongst the four by choosing the one that has the highest 90-percentile affinity score to ensure we more robust to outlier affinities that may result from not having an object list. Then, after selecting the view, again select the highest affinity point and move to that location.
5 Experiments
We investigate two questions: with given downstream search policy, (1) can a semantic distribution improve search performance? and (2) what is the best way to generate semantic distribution?
5

---

Query
Rel
CLI SMS-E
Query
ROB
CLIP SWB
food
Telorsponn
Figure 2: Generating semantic distributions in open-world environments. Four examples from the eval- uation dataset with the 2D probability distributions generated for SMS-E and CLIP. These heatmaps are red for high-probability regions of finding the target object and blue for low probability. Top Left: An example of grocery store, where the target object is "incense sticks." CLIP highlights both near the candles and near the flowers as they are somewhat visually similar sticks, while SMS-E only highlights the candles. Bottom Left: An example of an office kitchen, where the target object is "cat food." CLIP gets distracted by the re- frigerator and only slightly highlights the cat sign. Top Right: An example of a house, where the target object is "paddle." CLIP incorrectly highlights the wooden panels along the walls, while SMS-E highlights the ping pong table. Bottom Right: For the target word "microphone," SMS E highlights the box with the speaker but CLIP struggles as objects are not visually similar.
5.1 Evaluation of Semantic Distributions Quality
We investigate the second question first to obtain a semantic distribution for the downstream policy. We evaluate semantic distribution generation both in closed-world and open-world environments. In close-world environments, we evaluate the affinity matrix quality where the semantic distribu- tion generated, for the given object list. In open-world environments, we evaluate the semantic distribution quality dataset of real-life scenes.
5.1.1 Cloled-World Environments
Experimental Setup: With an object list, the semantic distributions are directly generated from se- mantic affinity matrices, with rows and columns as objects from the list and the entries representing the affinities scores between objects calculated in Section 4.2. We use an object list of 27 objects in the pharmacy domaín (list in Appendix). We directly compare the affinity matrix quality. We approximate ground truth affinity matrix with Google Taxonomy in Section B.I. The Google taxonomy provides semantic information for evaluation purposes to avoid human bias. Since it has limited categories, it cannot be used directly for objects that do not appear in the taxonomy.
We compare the quality of the affinity matrix for the given object list generated by SMS and a CLIP-based baseline proposed by CoW [8], which uses the dot products CLIP text embedding as the affinity scores. We compute the reduction of Jensen-Shannon Distance (JSD) [109] between the generated affinity matrix and the ground truth affinity matrix compared the JSD between uniform matrix and the ground truth. This quantifies the benefit SMS provides distribution. From Table 1, we see that SMS significantly outperforms the CI.IP-based method, while the SMS-LLM variant slightly outperforms the SMS-E. This suggests the reasoning capability of LLM models is more valuable for capturing semantics than CLIP embeddings. We compare other methods in Table 6 in the Appendix.
Results:
uniform
Metrie CLIP SMS-E
SMS-LLM
Method IoU
OWL-VET
CLIPP
SMS-LLM
SMS-E
A% 20.0%
33.8%
44.6%
0.138 1 0.031
0.221 1 0.034
0.345 1 0.039
0.391 0.039
Table I: Closed-world semantics eval-
Table 2: Open-world semantics evaluation. IoU results for dif- ferent methods. The object patector-bassee method, OWL-ViT, per- forms poorly because even though the target objects are semanti- cally related, many have very little visual similarity. CLIP per- forms worse than SMS because SMS is getting semantic similarity in a language-only latent space which can capture more nuance than the iisual-languee embedding space.
uation. Percentage improvement of the generated semantic distributions in the
pharmacy domain compared a uni-
form prior, measured based on the Jensen- Shannon Distance (ISD) from the ground truth distribution.
5.1.2 Open-World Environments
Experimental Setup: For the open-world environment, we evaluate the semantic distribution gen- eration static image dataset consisting of 30 real scenes taken from 4 houses, 4 office buildings, and 3 local grocery stores. We sampled 90 objects across the three domains and chose those scenes

---

based on our accessibility to those places. In all scenes, the objects' numbers and placements are set by their management. All scenes and the target object list are in Appendix E. Since these scenes are large, we are interested in quantifying the accuracy of the semantic distribution along both the a-- and y-axes. We annotate the ground truth search area based on the real scene and use Intersection over Union (IoU) to quantitatively evaluate the accuracy of each method.
Results: We evaluate the following VLM-only baselines: CLIP and OWL-ViT, the two best- performing methods found by Gadre et al. [8]. For CLIP, it uses the same crop-label pairs as SMS to generate a semantic distribution as described in Section 5.1 but with further augmentations (jittrring and horizontal flipping) on those crops for better performance [73]. We threshold this distribution and create a mask to calculate IoU with the ground truth. OWL-ViT gives bounding boxes for its la- bels and directly use them to calculate the IoU. We find that OWL-ViT performs better if the best bounding box is selected rather than weighting all bounding boxes by their score and thresholding that distribution. Table 2 shows the results. SMS generates semantic distributions within 35 to 45 seconds. We see that SMS outperforms the VLM-only methods including both CLIP and OWL-ViT. More examples are in the Appendix. We hypothesize that this is because CLIP focuses more on the visual appearance of the objects rather than semantic relations. This would be less of problem for searching visible objects but not ideal for searching objects that are outside the field of view or occluded. For example, CLIP would associate incense sticks with sticks used for gardening while LLMs would associate the incense sticks with the candles. In addition, CLIP has a "bag of words" behaviour [30], causing it to incorrectly relate "cat food" with fridge instead of a cat sign. In con- trast, LL have better semantic reasoning as shown in Figure 8, where "cat food" highlights the cat sign as the highest region but also highlights the gray bag because cat food could be occluded inside of a bag. Since LLMs are trained on large corpora of human language, we hypothesize that they effectively encode the semantics of both common and rare objects and are also capable of semantic reasoning (e.g. cat food can be inside the bag) beyond just creating class categories and thus are better suited for searching fully-oceluded objects. SMS-E slightly outperforms SMS-LLM they are both bottlenecked by the quality of labels from BLIP-2.
We also conduct an ablation study for each module of SMS on semantic distribution quality with results in Table 11 in the Appendix, indicating the effectiveness of cropping with SAM and CLIP weighting, and the impact of image captioning model choice.
5.2 Semantic Distribution Effect on Mechanical Search Performance
Given a semantic distribution, we investigate the first question by conducting simulation and real experiments in close-world environments to evaluate search performance improvement brought by the semantic distribution. We combine the semantic distribution with an existing mechanical search policy LAX-RAY in Section 4.3.
Experimental Setup: For simulation, we consider pharmacy, a kitchen, and an office domain. In addition to the 27 objects for the pharmacy domain, we consider 24 and 40 representative objects for the kitchen and office domains from the Google Product Taxonomy [110]. We generate semantically organized scenes with the procedure detailed in Appendix D.1 and examples in Figure 5. The simulation and real experiments take place within a 0.8 0.35 m 0.57 m shelf environment.
5.2.1 Simulation Experiments
We run extensive experiments using the First Order Shelf Simulator (FOSS) from [3]. In simulation experiments, we assume perfect object detection but consider geometry for occlusion. For each domain, we generate semantically organized scenes (details in Appendix D. with various numbers of objects N=12, 15, 18, 21 with 200 scenes for each. Termination occurs when the target object becomes visible or reaches maximum action number 2NN.
For each scene, we evaluate whether SMS improves the performance of LAX-RAY [3], which only uses geometric models. We consider both SMS-E and SMS-LLM for augmenting the geometric distribution from LAX-RAY. We report two metries: Success rates: The ratio of trials where the target object is found within the maximum action limit the total number of trials. Number of actions: The mean and standard error of the number of actions required to reveal the target object.
7

---

Pharmacy Domain
Kitchen Domain
Successes
* *Actions
2%
Successes
# Acctions
2%
Successes
N AAttoons
2%
LAX-RAV
676/741
6.56
N/A
703/770
3.32 0.14
VIA
076/
4.14 = 0.19
N/A
SMS-E
591/741
1.18 0.11
21.8
725/770
2.43 0.10
26.8
580/ 733
4.10 = 0.18
0.9
SMS-LLM
608/741
32.4
710/770
2.42 : 0.10
27.1
598/753
3.63 :: 0.16
12.3
Table 3: Simulation experiment results for three domains averaged over 12, 15, 18, 21 number of objects, also reported with A %. the percentage reduction in the number of actions compared to LAX-RAY.
We report results for all numbers of objects N in the Appendix and the results averaged across all values of N in Table 3. In all domains, SMS-LLM and SMS-E improve LAX-RAY performances with higher success rates and fewer search actions. In the pharmacy and office domain, SMS- L.M outperforms SMS-E, while in the kitchen domain, they perform comparably. For the office experiments, the performance improvement relatively small. We hypothesize that this is due to majority of the office environment consisting of generic office supplies that do not have clear semantic categorization, making semantic prior less effective. Overall, the results suggest that SMS- LLM can serve as semantic blug-in module and improve LAX-RAY performance in semantically arranged environments by 32.4%, 27.1%, and 12.3% in the pharmacy, kitchen, and office domains respectively while improving success rates. SMS-LLM outperforms SMS-E, indicating the quality of the affinity matrix is directly correlated with the task performance.
We also show strong positive correlation between object detection accuracy and task performance with results and details in Appendix D.3, indicating the benefits of SMS using OCR. In addition, we show SMS are effective on different downstream policies by using SMS as the plug-in module for Distribution Entropy Reduction (DER) from [3]. The details are in Appendix D.3.1.
5.2.2 Physical Experiments
Method
Actions
A
Method
Actions A
LAX-RAY 4.25 = 0.64
N/A
SMS-LLM
2.25 = 0.46
47.1
Table 4: Physical experiment results (12 trials each). We report the average number of actions taken reveal the target object as well as the percentage reduction in the number of actions over the spatial neural network.
We conduct experiments on a physical pharmacy shelf. use the Kinova Gen2 robot with a 3D- printed blade and suction tool [4] (see Figure 1). An Intel RealSense depth camera mounted on the tool provides RGBD observations. We use 3 scenes each of N 7,8, 8, 9, and 10 objects for total of 12 scenes and a threshold visibility of 50% for determining success. More details are in the Appendix. As simulation results from Table 3 suggest SMS-LLM outperforms SMS-E, we evaluate SMS-LLM in physical experiments, An identical set of 12 semantically arranged scenes (starting configurations) is used for each method.
Results are shown in Table 4. We observe that SMS significantly accelerates mechanical search on shelves, reducing the average number of actions by 47.1%. In physical experiments, the noises in the depth images result in worse spatial distribution than in simulation, making the semantic distribution more critical in identifying where target object may lic. We also conduct preliminary navigation experiment in open-world environments as Section 4.3 with details in Appendix D.5.
6 Limitations and Future Work
We present Semantic Mechanical Search (SMS), an algorithm for semantic distribution generation for a fully-occluded target object. SMS facilitates mechanical search in closed-world environments and improves semantic distribution quality for open-world environments. SMS has several limita- tions, which open up possibilities for future work: (I) We only evaluate one open-world navigation task with heuristic navigation policy without large-scale evaluations of other open-world envi- ronments; (2) While SMS, which operates in the L.LM feature space, generates better semantic distributions than CLIP-based method, we have not compared to other VLMs such as GPT-4 [111] or LLaVa [112] due to their inaccessibility to obtain affinity scores. VLMs with strong reasoning abilitics, such as GPT-4V [113], have the potential to directly generate high-quality semantic dis- tributions. Further applying GPT-4V in object search would be an interesting future direction. (3) SMS for cloled-world relies on creating an offline affinity matrix which can take few minutes with large object lists, while SMS for open- world takes 35 to 45 seconds for each semantic distribution (4) The performance of SMS is sensitive to the quality of each module in the framework.
8

---

Acknowledgments
This research was performed at the AUTOLAB at UC Berkeley in affiliation with the Berkeley AI Research (BAIR) Lab. The authors are supported in part by donations from Toyota Research Institute, Bosch, Google, Siemens, and Autodesk. I. Y. Chen is supported by the National Science Foundation Graduate Research Fellowship Program under Grant No. DGE 2146752. Any opinions, findings, conclusions, or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the sponsors. We thank our colleagues who helped with the project and provided helpful feedback and suggestions, in particular Chung Min Kim and Alishba Imran.
References
M. Danielczuk, A. Kurenkov, A. Balakrishna, M. Matl, D. Wang, R. Martin-Martin, A. Garg, S. Savarese, and K. Goldberg. Mechanical search: Multi-step retrieval of target object occluded by clutter. In 2019 International Conference on Robotics and Automation (ICRA), 2019.
[2] A. Kurenkov, R. Martín-Martín, Ichnowski, Goldberg, and Savarese. Semantic and geometric modeling with neural message passing in scene graphs for hierarchical mechan- ical search. International Conference on Robotics and Automation (ICRA), 2021.
[3] H. Huang, M. Jominguez-KKunee V. Satish, M. Danielczuk, K. Sanders, J. Ichnowski, A. Lee, A. Angelova, V. Vanhoucke, and K. Goldberg. Mechanical search on shelves us- ing lateral access x-ray. In 202/ NEEE/SS International Conference on Intelligent Robots and Systems (IROS), pages 2045-2052, 2021.
[4] H. Huang, M. Danielezuk, C. M. Kim, Fu, Z. Tam, J. Ichnowski, A. Angelova, B. Ichter, and K. Goldberg. Mechanical search shelves using novel "bluction" tool. IEEE Interna- tional Conference Robotics and Automation (ICRA), 2022.
[5] H. Huang, I. Fu, M. Danielczuk, C. M. Kim, Z. Tam, J. Ichnowski, A. Angelova, B. Ichter, and K. Goldberg. Mechanical search on shelves with efficient stacking and destacking of objects. International Symposium on Roboties Research (ISRR), 2022.
[6] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. Holm: Halluci- nating objects with language models for referring expression recognition in partially-observed scenes. In Proceedings of the 60th Annual Meeting of the Association for Computational Lin- guisties (ACL), 2020.
[7] Huang, O. Mees, A. Zeng, and W. Burgard. Visual language maps for robot navigation. In Proceedings of the IEEE International Conference Robotics and Automation (ICRA), London, UK, 2023.
[8] S. Y. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Basclines and benchmarks for language-drive zero-shot object navigation, 2022.
[9] S. Parisi, Rajeswaran, S. Purushwalkam, and A. Gupta. The unsurprising effectiveness of pre-trained vision models for control. In International Conference Machine Learning, pages 17359-17371. PMLR, 2022.
[10] S. Nair, A. Rajeswaran, V. Kumar, C. Finn, and A. Gupta. R3m: universal visual represen- tation for robot manipulation. arXiv preprint arXiv:2203. /2601, 2022.
[11] I. Radosavovic, T. Xiao, S. James, Abbeel, J. Malik, and T. Darrell. Real-world robot learning with masked visual pre-training. In Conference Robot Learning, pages 416-426. PMLR, 2023.

---

[12] Y. J. Ma, S. Sodhani, D. Jayaraman, O. Bastani, V. Kumar, and A. Zhang. Vip: Towards universal visual reward and representation via value-implicit pre-training. arXiv preprint arXiv:2210.00030, 2022.
[13] A. Stone, T. Xiao, Y. Lu, K. Gopalakrishnan, K.-H. Lee, Q- Vuong, P. Wohlhart, B. Zitkovich, F. Xia, C. Finn, et al. Open-world object manipulation using pre-trained vision-language models. arXiv preprint MrXiv:2303.00005 2023.
[14] S. Nair, E. Mitchell, K. Chen, B. Ichter, S. Savarese, and C. Finn. Learning language- conditioned robot behavior from offline data and crowd-sourced annotation. In Conference Robot Learning (CoRL), 2021.
[15] E. Jang, A. Irpan, M. Khansari, D. Kappler, F. Ebert, C. Lynch, S. Levine, and C. Finn. Be-z: Zero-shot task generalization with robotic imitation learning. Conference on Robot Learning, pages 991-1002. PMLR, 2022.
[16] M. Shridhar, L. Manuelli, and D. Fox. Perceiver-actor: A multi-task transformer for robotic manipulation. Conference on Robot Learning (CoRL), 2022.
[17] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. arXiv preprint arXiv:2005.07644 2020.
[18] C. Lynch and Sermanet. Language conditioned imitation learning over unstructured data. Robotics: Science and Systems (RSS), 2021.
[19] C. Lynch, A. Wahid, J. Tompson, T. Ding. J. Betker, R. K. Baruch, T. Armstrong, and P. R. Florence. Interactive language: Talking to robots in real time. ArXiv preprint 1rXiv22200066400 2022.
[20] Y. Jiang, A. Gupta, Z. Zhang, G. Wang, Dou, Y. Chen, L. Fei-Fei, A. Anandkumar, Y. Zhu, and L. Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint 2022.
arXiv:2210.084
[21] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. R. Florence, and A. Zeng. Code as policies: Language model programs for embodied control. ArXiv preprint urXiv:2209007553, 2022.
[22] B. et al. Do as i can, not as i say: Grounding language in robotic affordances. In 6th Annual Conference on Robot Learning, 2022.
[23] W. Huang, P. Abbeel, D. Pathak, and I. Mordatch. Language models zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference Ma- chine Learning, pages 9118-9147. PMLR, 2022.
[24] Zeng, A. Wong, Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sind- hwani, Lee, Vanhoucke, et al. Socratic models: Composing zero-shot multimodal rea- soning with language. arXiv preprint arXiv:2204.000 00598, 2022.
[25] W. H. et al. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint 2.Xi::2277006003 2022.
[26] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209. /302, 2022.
[27] Radford, W. Kim, C. Hallacy, A. Ramesh, Goh, S. Agarwal, G. Sastry, Askell, P. Mishkin, Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In International Conference Machine Learning (ICML),
1021.
10

---

[28] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vocabulary queryable scene representations for real world planning, 2023.
[29] D. Shah, B. Osiński, brian ichter, and S. Levine. LM-nav: Robotic navigation with large pre- trained models of language, vision, and action. In 6th Annual Conference on Robot Learning, 2022. URL https: / openrevieweeeeeeeeeeeeeeeeee net forum? Ld=
[30] J. Kerr, C. M. Kim, K. Goldberg, A. Kanazawa, and M. Tancik. Lerf: Language embedded radiance fields. arXiv preprint arXiv:2303.09553, 2023.
[31] X. Gu, T.Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation, 2022.
[32] F. Liang, B. Wu, X. Dai, K. Li, Zhao, H. Zhang, P. Zhang. P. Vajda, and D. Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip, 2023.
[33] Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision, 2022.
[34] J. Wei, X. Wang, D. Schuurmans, M. Bosma, E. Chi, Q. Le, and D. Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201 1990, 2022.
[35] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tomp- son, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, Sermanet, Duckworth, S. Levine, V. Van- houcke, Hausman, M. Toussaint, K. Greff, Zeng, I. Mordatch, and Florence. Palm-e: An embodied multimodal language model. In arXiv preprint arXiv:2303.03378. 2023.
[36] B. Yamauchi. A frontier-based approach for autonomous exploration. In Proceedings 1997 IEEE International Symposium on Computational Intelligence in Robotics and Automation CIRA '97. "so'assssss New Computational Principles for Robotics and Automation' pages 146 151. IEEE, 1997.
[37] J. Maja. Integration of representation into goal-driven Schavior-based robots. IEEE transac- tions on robotics and automation, 8(3):304-31,, 1992.
[38] S. Thrun and A. Bücken. Integrating prid-based and topological maps for mobile robot igation. In Proceedings of the national conference artificial intelligence, pages 944-951, 1996.
[39] B. Yamauchi and R. Beer. Spatial learning for navigation in dynamic environments. IEEE Transactions on Systems, Man, and Cybernetics, Part (Cybernetics)) 26(3):496-50) 1996.
[40] Gu, T.-Y. Lin, W. Kuo, and Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. International Conference on Learning Representations (ICLR), 2021.
[41] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y, Lo, P. Dollár, and R. Girshick. Segment anything. arXiv.2304.02643, 2023.
[42] J. Li, D. Li, S. Savarese, and S. Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. ArXiv, abs/2301.12597, 2023.
[43] M. Gupta, Rühr M. Beetz, and Sukhatme. Interactive environment exploration in clutter. pages 5265-5272, 2013.
[44] M. R. Dogar, M. C. Koval, A. Tallavajhula, and S. S. Srinivasa. Object search by manipula- tion. Autonomous Robots, 36(1):153-167, 2014.

---

[45] J. K. Li, D. Hsu, and W. S. Lee. Act to see and see to act: Pomdp planning for objects search in clutter. pages 5701-5707, 2016.
[46] Y. Xiao, S. Katt, A. ten Pas, S. Chen, and C. Amato. Online planning for target object search in clutter under partial observability. pages 8241-8244 2019.
[47] W. Bejjani, W. C. Agboh, M. R. Dogar, and M. Leonetti. Occlusion-awar search for object retrieval in clutter. pages 4678-4685, 2021.
[48] L. Y. Chen, H. Huang, M. Danielczuk, Ichnowski, and K. Goldberg. Optimal shelf arrange- ment minimize robot retrieval time. IEEE International Conference on Automation Science and Engineering (CASE), 2022.
[49] L. E. Wixson and D. H. Ballard. Using intermediate objects to improve the efficiency of visual search, International Journal of Computer Vision, 12(2-3):20--230, 1994.
[50] T. Kollar and N. Roy. Utilizing object-object and object-scene context when planning to find things. In 2009 IEEE International Conference on Robotics and Automation, pages 2168- 2173. IEEE, 2009.
[51] I. L.. Wong, L. P. Kaelbling, and T. Lozano-Pérez. Manipulation- based active search for occluded objects. In 2013 IEEE International Conference on Robotics and Automation, pages 2814-2819. IEEE, 2013.
[52] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural slam. arXiv preprint arXiv:2004001555, 2020.
[53] P. Chattopadhyay, J. Hoffman, R. Mottaghi, and A. Kembhavi. Robustnav: Towards bench- marking robustness in embodied navigation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 15691-15700, 2021.
[54] Gordon, A. Kadian, Parikh, J. Hoffman, and D. Batra. Splitnet: Sim2sim and task2task transfer for embodied visual navigation. In Proceedings of the IEEE/CVF International Con- ference Computer Vision, pages 1022-1031, 2019.
[55] L. Mezghan, S. Sukhbaatar, Lavril, O. Maksymets, D. Batra, P. Bojanowski, and K. Ala- hari. Memory--ugmenneed reinforcement learning for image-goal navigation. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3316- 3323. IEEE, 2022.
[56] Y. Zhu, R. Mottaghi, E. Kolve, J. Lim, A. Gupta, L. Fei-Fei, and Farhadi. Target- driven visual navigation in indoor scenes using deep reinforcement learning. In 2017 IEEE international conference on robotics and automation (ICRA), pages 3357-3364. IEEE, 2017.
[57] Z. Al-Halah, Ramakrishnan, and K. Grauman. Zero experience required: Plug play modular transfer learning for semantic visual navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17031-17041, 2022.
[58] M. Chang, A. Gupta, and S. Gupta. Semantic visual navigation by watching youtube videos. Advances Neural Information Processing Systems, 33:4283 4294, 2020.
[59] J. K. Tsotsos. On the relative complexity of active vs. passive visual search. International journal of computer vision, 1(2):127-14, 1992.
[60] A. Aydemir, A. Pronobis, M. Göbelbecker, and P. Jensfelt. Active visual object search in unknown environments using uncertain semantics. IEEE Transactions on Robotics, 29(4): 986-1002, 2013.
[61] S. Thrun. Intelligence, 99(1):11-7,, 1998.
Learning metric-topolese toplloiaolll maps for indoor mobile robot navigation. Artificial
12

---

[62] J. S. Feder, J. Leonard, and C. M. Smith. Adaptive mobile robot navigation and mapping. The International Journal of Robotics Research, 18(7):650-668, 1999.
[63] Kuipers and Y.-T. Byun. A robot exploration and mapping strategy based on semantic hierarchy of spatial representations. Robotics and autonomous systems, 811-2):77-63 1991.
[64] B. H. Wilcox. Robotic vehicles for planetary exploration. Applied Intelligence, 2:181-193, 1992.
[65] P. Anderson, Q Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. Reid, S. Gould, and A. Van Den Hengel. Vision-ande -nd--aandddaaa navigation: Interpreting grounddd naviga- tion instructions in real environments. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3674-3683, 2018.
[66] D. S. Chaplot, D. P. Gandhi, A. Gupta, and R. R. Salakhutdinov. Object goal navigation using goal-oriented semantic exploration. Advances in Neural Information Processing Systems, 33: 4247-4258, 2020.
[67] M. Deitke, E. VanderBilt, A. Herrasti, L. Weihs, K. Ehsani, J. Salvador, W. Han, E. Kolve, A. Kembhavi, and R. Mottaghi. Procthor: Large-scale embodied ai using procedural genera- tion. Advances in Neural Information Processing Systems, 35:5982-5994, 2022.
[68] Y. Liang, B. Chen, and S. Song. Ssenav: Confidence-awann semantic scene completion for vi- sual semantic navigation. In 202/ IEEE international conference robotics and automation (ICRA), pages 13194-13200. IEEE, 2021.
[69] S. Wani, S. Patel, U. Jain, A. Chang, and M. Savva. Multion: Benchmarking semantic map memory using multi-object navigation. Advances Neural Information Processing Systems, 33:9700-9712, 2020.
[70] M. Wortsman, K. Ehsani, M. Rastegari, A. Farhadi, and R. Mottaghi. Learning to learn how to learn: Self-adaptive visual navigation using meta-learning. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 6750-6759, 2019.
[71] W. Yang, X. Wang, A. Farhadi, A. Gupta, and R. Mottaghi. Visual semantic navigation using scene priors. arXiv preprint 1rXiv:1810.06543 2018.
[72] M. Samadi, T. Kollar, and M. Veloso. Using the web to interactively learn to find objects. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 26, pages 2074-2080, 2012.
[73] H. Ha and S. Song. Semantic abstraction: Open-world 3d scene understanding from 2d vision-language models. In 6th Annual Conference on Robot Learning, 2022.
[74] Ghiasi, X. Gu, Y. Cui, and T.-Y. Lin. Open- Open- vocabulary image segmentation. arXiv preprint arXiv:2112.12143. 2021.
[75] B. Li, K. Q. Weinberger, S. Belongie, V. Koltun, and R. Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03542 2022.
[76] S. Peng, K. Genova, C. Jiang, A. Tagliasacchi. M. Pollefeys. T. Funkhouser, al. Openscene: 3d scene understanding with open vocabularies. arXiv preprint arXiv:2211./5654, 2022.
[77] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.05663, 2022.
[78] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li. G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.077 2023.
13

---

[79] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, A. Stone, and D. Kappler. Open-vocabulary queryable scene representations for real world planning. arXiv preprint arXiv:2209.09874, 2022.
[80] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arna,, M. Dehghani, Z. Shen, et al. Simple vpenbvocayulary object detection with vision transformers. arXiv preprint arXiv:2205.062350 2022.
[81] H. Chefer, S. Gur, and L. Wolf. Generic attention-model explainability for interpreting bi- modal and encoder-decoder transformers. Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 397-406, October 2021.
[82] D. Chen and R. Mooney. Learning to interpret natural language navigation instructions from observations. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 25, pages 859-865, 2011.
[83] F. Duvallet, Kollar, and A. Stentz. Imitation learning for natural language direction fol- lowing through unknown environments. In 2013 IEEE International Conference on Robotics and Automation, pages 1047-1053. IEEE, 2013.
[84] F. Duvallet, M. R. Walter, Howard, Hemachandra, Oh, S. Teller, N. Roy, and A. Stentz. Inferring maps and behaviors from natural language instructions. In Experimental Roboties: The 14th International Symposium Experimental Robotics, pages 373-388. Springer, 2016.
[85] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg- Kirkpatrick, K. Saenko, D. Klein, and T. Darrell. Speaker-follower models for vision-and-language navi- gation. Advances in Neural Information Processing Systems, 31, 2018.
[86] S. Hemachandra, F. Duvallet, T. M. Howard, N. Roy, A. Stentz, and M. R. Walter. Learning models for following natural language directions in unknown environments. In 2015 IEEE In- ternational Conference on Robotics and Automation (ICRA), pages 5608-5615 IEEE, 2015.
[87] T. M. Howard, S. Tellex, and N. Roy. A natural language planner interface for mobile manip- ulators. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 6652-6659. IEEE, 2014.
[88] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In 2010 5th ACM/IEEE International Conference on Human-Robo Interaction (HRI), pages 259-266. IEEE, 2010.
[89] M. MacMahon, B. Stankiewicz, and B. Kuipers. Walk the talk: Connecting language, knowl- edge, and action in route instructions. Def. 2(6):4, 2006.
[90] C. Matuszek, D. Fox, and K. Koscher. Following directions using statistical machine transla- tion. In 2010 5th ACM/IEEE International Conference on Human-Roke Interaction (HRI), pages 251-258. IEEE, 2010.
[91] C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to parse natural language com- mands to a robot control system. In Experimental robotics: the 1 13th international symposium on experimental robotics, pages 403-415. Springer, 2013.
[92] Mei, M. Bansal, and M. Walter. Listen, attend, and walk: Neural mapping of naviga- tional instructions action sequences. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
[93] S. Tellex, T. Kollar, Dickerson, M. Walter, A. Banerjee, Teller, and Roy. Under- standing natural language commands for robotic navigation and mobile manipulation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 25, pages 1507-1514, 2011.
14

---

[94] J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In Twenty-Fourth International Joint Conference on Artificial Intelligence, 2015.
[95] M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human- robot interaction. arXiv preprint arXiv:1806.03383/ 2018.
[96] D. K. Misra, J. Sung, K. Lee, and A. Saxena. Tell me dave: Context-sensitive grounding natural language to manipulation instructions. The International Journal of Robotics Re- search, 35(1-3)288-300 2016.
[97] R. Paul, J. Arkin, D. Aksaray, N. Roy, and T. M. Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms. The International Journal of Robotics Research, 37(10))1299-1299, 2018.
[98] S. Patki, E. Fahnestock, T. M. Howard, and M. R. Walter. Language-gun -uiigge--uuedddd semantic map- ping and mobile manipulation in partially observable environments. In Conference Robot Learning, pages 1201-1210. PMLR, 2020.
[99] O. Mees and W. Burgard. Composing pick-and-place tasks by grounding language, 2021.
[100] M. Shridhar, L. Manuelli, and D. Fox. Cliport: What and where pathways for robotic manip- ulation. Conference on Robot Learning (CoRL), 2021.
[101] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Hausman, A. Herzog, J. Hsu, et al. Rt-1: Robotics transformer for real-world control scale. arXiv preprint arXiv:22 11..222222.08111 2022.
[102] D. K. Misra, J. Langford, and Y. Artzi Mapping instructions and visual observations to tions with reinforcement learning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017.
[103] Y. Jiang, S. S. Gu, K. Murphy, and Finn. Language as an abstraction for hierarchical deep reinforcement learning. Conference on Neural Information Processing Systems (NeurIPS), 2019.
[104] P. Sharma, B. Sundaralingam, V. Blukis, C. Paxton, T. Hermans, A. Torralba, J. Andreas, and D. Fox. Correcting robot plans with natural language feedback. Robotics: Science and Systems (RSS), 2022.
[105] Y. Cui, S. Karamcheti, R. Palleti, N. Shivakumar, P. Liang, and D. Sadigh. No, to the right: Online language corrections for robotic manipulation via shared autonomy. ACM/IEEE In- ternational Conference on Human-Robot Interaction (HRI), 2023.
[106] W. Huang, F. Xia, D. Shah, Driess, Zeng, Y. Lu, P. Florence, I. Mordatch, Levine, K. Hausman, al. Grounded decoding: Guiding text generation with grounded models for robot control. arXiv preprint arXhv:2303.00855, 2023.
[107] Keras ocr. https:/ https://support. hl-en.
e. wcw.///rrccaaatt/ssseee//22222/
[108] A. N. al. Text and code embeddings by contrastive pre-training. arXiv preprint arXiv:2201.10005. 2022.
[109] J. Lin. Divergence measures based on the shannon entropy. JEEE Transactions on Information Theory, 34511,4 1991. 100110.
[110] G. M. Center.
Google product category.
com/
merchants
Accessed: 2023-01-31.
15

---

[111] OpenAL. Gpt-4 technical report, 2023.
[112] Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning, 2023.
[113] OpenAI. Gpt-4v(ision) system card, 2023.
[114] S. Karaoglu, J. Gemert, and T. Gevers. Object reading: Text recognition for object recogni- tion. volume 7585, 10 2012. ISBN 978-3-642-3388477, 46:110110019978---223333--------
[115] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. ArXiv preprint arXiv:1810.04805, 2019.
[116] S. Z. et al. Opt: Open pre-trained transformer language models. ArXiv preprint 2205.01064 2022.
16

---

Appendix
A Scene Understanding
A.1 Object detection + OCR
Because ViLD general-purpose detector, it cannot easily distinguish between objects belong- ing to the same domain (e.g., Advil versus Ibuprofen). Because this, we use OCR with Keras OCR[107] to improve the quality of the object detections. While OCR has been used prior work aid object detection [114], we use text embedding combined with OCR for better performance. For each object, concatenate the text observed it and compute the text embedding using Ope- nAI Embeddings.. We compute the dot product between the embeddings of the concatenated text and every class label. We normalize this probability vector by subtracting the minimum value and then adjusting the vector with some temperature. We finally multiply this by the object detection probability vector.
Let Cj denote the class label of object O₂ (e.g., "Tylenol" opposed the broader category "medi- cation"); I3 represent the general shape, size, and color-related features of O; and T; be the detected text on O, Recall that all objects belong to some class Ci. We calculate
P(C) 4,T
P(A,T P(C..)
P(I,T:)
P(T), C.) P(LCC))
P(I,T.)
P(T.CC)) P(IC) -(C())))(())
P(ITT))
P(CIT))((T) P(CIL)P()) P(C.) P(C.)
P(C,) P(I,T.)
P(C[E))((C())
T; is independent of when conditioned Ct, and P(C,) is uniform. This illustrates that the multiplication of the OCR probabilities and the object detection probabilities can give refined estimate of the category probabilities.
We test object detection performance on scenes generated through isolated perception experiments. We take RGB images of 100 scenes of the Pharmacy domain using a high-resolution camera and study the effect of having OCR. Results for this experiment are Table 5. As is standard in the computer vision literature, over-union (IOU) thresholds from 0.50 to 0.95 with step size of 0.05, as well as top-k classification accuracy (i.e., if the ground truth label appears in the lo labels with the highest probabilities). The results show that OCR leads to significant improvement across all metrics, with mAP improving by a factor of 12 and top-1 accuracy improving by a factor of 3.
report mAP (mean Average Precision) averaged over intersection-
Table 5: Object Detection Refinement Results. We study the effect OCR and report the mean average precision (mAP) of the predicted bounding boxes and top- accuracy of the predicted labels.
Method
mAP (1)
Top-K Accuracy % (1)
k=1
k=3
k=5
ViLD ViLD + OCR
2.4
14.7
32.3
41.6
28.9
45.0
62.0
69.5

---

B Creating the Semantic Distribution
B.1 Affinity Matrix Generation
We compare two ways to generate affinity matrices. First, we generate the affinity matrices using large language models using the following procedure: 1) We replace both the {obj} and the {target object] in the prompt: "I see the following in a room: {obj}. This is likely to be the closest object to (target object]." 2) We find the log probability of the target object instead the {obj} since it will be represented by the same number of tokens regardless of the {obj} in this prompt and use this instead for representing affinity values. Second, we use an embedding model to encode {obs} and {target object} and use the cosine similarity to find the affinity values. We normalize when appropriate. For the pharmacy domain, in Table 6, we generate affinity matrices with different L.MSs and embedding models and then compare the quality of affinity matrices quantitatively by comparing them to the open-source Google Product Taxonomy [110] as the "ground truth" matrix. Visualizations of the affinity matrices generated by the ground truth, the best embedding model (OpenAI Embeddings), and the best LLM are shown in Figure for the pharmacy domain. In the pharmacy domain, we have the following 6 categories and items from the taxonomy:
1. Supplements: vitamins, fish oil, omega-3, calcium, probiotics, protein powder, COQ10, anthocyanin
2. Hair Care:
shampoo, conditioner
3. Oral Care:
toothpaste, toothbrush, dental floss
4. Cosmetics: face wash, sunscreen, lotion, hand cream, body wash
5. Medication:
aspirin, tylenol, ibuprofen, advil, pain relief
6. Outliers: shaving cream, eye drops, deodorant, band-aid
For the ground-truth matrix, all elements a category are given uniform affinities to each other, and each row normalized to (e.g., eye drops) does not belong to any of the other 5 categories and is treated as its own category. We use the Google taxonomy to categorize the objects within each category. With the categories listed order along both axes the matrix, the ground truth affinity matrix has a block-diagonal structure with a uniform block for each category (Figure 3A). We evaluate the following LLMss and embedding models off off-t, without finetuning: BERT [1 [1],, CLIP [27], embeddings from OpenAI API [108], OPT-13B [116], and PaLM. JSD measures the similarity between two proba- bility distributions,
1.0 probability. Note that each item in the "outliers" category
of the affinity matrix and the corresponding row of the ground truth. Then, we average across the rows to get the average distance from each object's probability distribution to that object's ground truth. We observe that the choice of LLM has a significant impact on the affinity matrix (Table 6), and that the LLMs can approxi- mately recover the block diagonal structure the ground truth matrix (Figure 3). PaLM attains the highest accuracy, with 44.6% improvement over a uniform affinity matrix.
measure the similarity between each
Table 6: Affinity matrix results. We report the average Kinsen-Shannon Distance (JSD) between each row of the affinity matrix and the ground truth matrix, as well as the percentage improvement over the uniform JSD (i.c., (uniform JSD method JSD) / uniform JSD).
Method
JSD (L)
% Improvement (1)
Uniform
0.65
N/A
BERT Embedding
0.64
1.5
CLIP Embedding
0.52
20.0
OpenAl Embedding
0.43
33.8
OPT-13B
0.38
41.5
PaLM
0.36
44.6
18

---

(A) Ground Truth (Google Taxonomy)
OpenAl Embeddings
(C) PaLM
Figure 3: Three different affinity matrices for the pharmacy domain. The left shows the affinity matrix generated from the Google Taxonomy. The center shows the affinity matrix generated by the OpenAl Embeddings model and the right shows matrix generated by PaLM.
B.2 Offline Semantic Distribution Generation with Object List
We now generate semantic distribution based on the affinity matrix and detected objects. The se- mantic occupancy distribution models the probability that the target object occupies a given location, given the classes of observed objects in the scene, i.e. P(LT where Lr is the location of the target object, Lı.... are the positions of the vistble objects, and C1.... are the inferred classes of the visible objects. We abbreviate this quantity P(I. = IT, C).
Li... .l.... C1... CI...),,
We interpret affinity values Mjj to be the probability of object j being the closest to object in expectation across scenes. However, given the current scene, there may be more or less space that nearest to a particular object, so we interpret these affinity values as being normalized per unit area. Thus, formally, given (() representing the index of the object closest to location (.z, yr), P(L = L,, C`) x Mtarget.N(())
In simulation experiments for constrained environments, N()) is computed using the 3D coordinates of the visible objects obtained from the depth image. We compute the 2D semantic occupancy distribution (in the horizontal plane of the shelf) and reduce it to ID by summing along camera rays. In physical experiments, to avoid errors due to noisy depth readings we compute the distribution directly in 2D, using pixel distance for N-). instead of world coordinates.
C Clored-World Downstream Mechanical Search Policies
C.1 Problem Statement
We consider the problem of robotic mechanical search for a target object Or in cluttered, se- mantically organized shelf containing the target and N other rigid objects {O), ..., ON 3 of cuboidal shapes in stable poses. We build the problem statement and assumptions in Huang et al. [4]. We model the setup S consist of the full geometries and poses of the objects in the shelf at timestep t and obser- vations y/. E y = xxxX are RGBD images from robot-mounted depth camera at timestep I. Actions 0, E A = A, U A. are either pushing or suction actions, where the former are horizontal linear translations of an object along the shelf and the latter pick up an object with suction gripper and translate it to an empty location on the shelf with no other objects in front of it. We make the following assumptions:
finite-horizon Partially Observable Markov Decision Process (POMDP). States
The dimensions of the shelf are known.
Each dimension of each object is between size Smin 5 cm and size Smax 25 cm.
19

---

Offline
Target Object: Toothbrush
Lunguage Modal
guea
Semantic Occupancy Dist.
Semantic Spatial
RGBD Image
Occupancy Dist.
Affinity Matrix
VILD Object
Detection
Mechanical Search Pollcy
Object class labels Quiiing 1o-1-
Gacmetric Occupancy
Action R
Figure 4: Overview of SMS for Constrained Environments with Downstream Mechanical Search Policy. SMS receives scene image and desired target object. Since the object list is known, then applies object detection and OCR to identify objects within the scene. SMS then uses a large language model to compute affinities between detected objects to the target object, and it uses these affinities to output semantic occupancy distribution of the appropriate dimension for the downstream problem. This distribution indicates the likelihood the physical presence of objects which is used to determine the next action by the downstream mechanical search policy.
The shelf is semantically organized.
The names of all objects in the shelf are a subset of known list of object names.
Actions cannot inadvertently topple objects or move multiple objects simultaneously.
C.2 LAX-RAY
LAX-RAY[4] a mechanical search policy for shelf environments. LAX-RAY have utilized geo- metric information by considering object geometries and camera perspective (e-g., tall target objects cannot be occluded by short objects and objects in the center of an image occlude more areas) to facilitate the search. It consists of perception module and greedy action selection module. The perception module takes the depth observation and predicts the geometric/spatial occupancy distri- bution to encode the geometric information. LAX-RAY learns this module on a simulation dataset, with the ground-trut occupancy distribution calculated using Minkowski sum. greedy action se- lection module called Distribution Area Reduction (DAR) selects robot actions to greedily reduce the overlap between objects and the distribution. Another search policy has been proposed in [3], namaed Distribution Entropy Reduction (DER). DER selects the action that would reduce the tropy of the distribution the most after taking the action. We denote the searching pipeline with DER to be LAX-RAY (DER).
We show the SMS pipeline specifically for constrained environments with LAX-RAY in Figure 4. This pipeline was used conduct the simulation mechanical search experiments in Section D.2.1 and the physical mechanical search experiments in Section D.4.
D Experiments
D.1 Scene Generation
The taxonomy defines where each category is a node and each object name is leaf node. To create scene with N objects in given domain, we begin by uniformly sampling N objects without replacement from the total objects available in that domain. We then generate scenes in a top-down recursive manner using the taxonomy tree. At the root, we start with the whole shelf available to us. At each node, we split the shelf in half either horizontally or vertically with 50% probability each and recursively continue scene generation in these sub-shelves. a node has more than descendants, however, we always split the scene horizontally to avoid overerowding resulting from the aspect ratio of the shelf. At each level of recursion, we accumulate random noise to the eventual placement of each object in the current branch, uniformly sampled from -2 cm to 2 cm. At the last lon- node, we place all leaves in random positions within the current level's sub-shelf.
20

---

Table 7: Simulation Experiment Results.
Pharmacy Domain
12 objects
15 objects
18 objects
21 objects
Successes
# AAttoons
Successes
P.Actions
Successes
# #Actions
Successes
AAtioons 8:24
LAX-RAY
168/190
0.28
160/186
5.17
144/188
0.44
104,177
SMS-E
176/190
2.90 0.18
159/186
3.77 0.26
146/188
5.05 0.44
110/177
5.69 4 0.54
SMS-LLM
176/190
2.66 = 0.14
162/186 186
3.26 = 0.19
150/188
4.25 4 0.34
118/177
5.47 1 0.43
Kitchen Domain
12 objects
objects # Actions
18 objects
objects
Successes
# #Actions
Successes
Sucvesses
# Actions
Successes
Actions
LAX-RAY
185/192
2.15
182/191
2.97 = 0.23
177/I93
3.39 0.29
1597191
1.36 0.38
SMS-E
186/192
188/194
2.15
184/193
167/191
SMS-LLM
184/192
1.00 + 0.10
184/191
2.04 0.13
179/193
2.97 + 0.26
163/191
3.17 +028
Office Domain
12 objects
objects
18 objects
objects
Successes
#AAtioons
Successes
#Actions
Successes
# #Actions
Successes
AAtioons
LAX-RAY
1727194
2.18
152/188
0.335
136/190
1.641
115/181
5.80 1.00
SMS-E
173/194
3.01 = 0.22
132/188
3.50 = 0.31
140/190
1.88004
115/181
0.33
SMS-LLM
172/194
2.33 = 0.13
161 /188
3.50 = 0.31
142/190
3.75 0.32
123/181
5.50 0.49
Fanel
Yiund
00016
Poosas
Plax
Cocro
Rucenerr
destrnj Soog
Pick
Figure 5: Two examples of layouts for the simulated scenes from the birds-eye view, where the top is the rear of the shelf and the bottom is the front. The layouts are generated from the procedure Section D.1. The left scene is an example a layout for scene with objects and right scene is layout for a scene with objects. Green rectangles represents objects in personal care categories and pink rectangles represents objects in supplements.
We resolve collisions by iteratively moving objects along the displacement vector between colliding objects and discard scenes where such procedure takes longer than 1 second to run. We also discard scenes where there is no potential target object that is invisible from the camera's perspective at the start of the rollout. We reiterate that the taxonomy is independent of the language models used to generate affinities. The LLMs are applicable beyond manual semantic categorizations like the Google Taxonomy, but we use this resource for evaluation purposes. The scenes for all simulation, physical, and object detection experiments are generated by this procedure.
We use approximate sizes of these items to generate collision-free scenes. In simulation, we also scale these objects down in order to be able to run experiments on the same-sized shelf, which has an effect similar to running experiments in a larger shelf where more items could originally fit. The scaling factors for the pharmacy and kitchen domains are 0.7, but 0.4 in the office domain due to overall larger objects unable to easily fit and move within a small shelf.
D.2
Simulation Experiments
D.2.1
Simulation Experiments with LAX-RAY
We run extensive suite of experiments using the same simulator prior work in mechanical search on shelves [3] and study the benefit brought by SMS. We use grid search on the average number of actions required in the pharmacy domain with 15 objects to tune the Gaussian smoothing a to be 50 pixels. We use the same parameters for the other two domains.
We generate scenes with various numbers of objects: N 12, 15, 18, and 21. We generate 200 scenes for each value of N. In Figure 5, we show example layouts of the scenes created by the procedure in Section D.I. We discard scenes where the target object starts out visible, resulting in
21

---

just under 200 scenes for each value of N. Termination occurs when at least X = 1% of the target object becomes visible or reaching maximum action number 2.V. The reason for the low threshold is that the DAR policy has trouble making progress on a partially revealed target object [3], which may dilute the comparison between different methods for generating semantic distributions.
We report results for all numbers of objects N in Table 7. SMS-LLM outperforms both SMS-E, while also beating LAX-RAY across various values of N in terms of success rate (by an additional 30/741 scenes) and average number of actions required (by 32.4%). A point of note is that the action differential percentage grows as the number of objects increases. At 21 objects, LAX-RAY requires 8.24 actions on average, whereas SMS requires just 5.47. This trend agrees with intuition that is unscalable to search large environments with no semantic intuition.
D.3 Simulation Experiments with Object Detection Noise
Method
No noise
10% Noise
50% Noise
90% Noise
LAX-RAY
# of Actions
3.81 0.31
4.20 = 0.38
4.44 1 0.41
4.83 + 0.47
5.12. 0.43
Table 8: Experiment to determine the impact of object detection noise on task performance (# of actions). For SMS- LMM.LLMM we randomly perturb the object detection (i.e. randomly select a label from the object list) with probability P. We do 400 rollouts over the categories of 12, 15, 18, 21 objects in the scene for the pharmacy domain. We report the average number of actions taken to reveal the target object and standard error. We the general trend as object detection noise increases the task performance decreases.
We study the impact of the object detection accuracy on the task performance. We randomly change the object labels with a probability P. The results are shown in Table 8, where P = 0.1, 0.5, 0.9. The number of actions needed to find the occluded object increases as P increases. This is because random perturbations can cause the semantic distribution to approach a uniform distribution thus not modifying the existing action of the downstream policy. Therefore, Table 8 indicates there is also a strong positive correlation between object detection accuracy and task performance.
D.3.1
Simulation Experiments with DER
Policy
12 objects
15 objects
18 objects
21 objects
Success
# Actions
Success
# Actions
Success
# Actions
Success
# Actions
LAX-RAY (DER)
84%
5.79++0.38
74%
7.69-0.54
62%
8.08+0.64
42%
9.52++0.72 6.87100.7
SMS-LLM
90%
4.42 0.39
81%
5.06=0.43
71%
7.11110.60
45%
Table 9: Simulation experiments results of SMS-LLM with DER for the Pharmacy domain. We ablate the
downstream policy and see that SMS-LLM outperforms LAX-RAY with DER. We report the number of rollouts that were successful and the mean actions to retrieve the occluded object and the standard error.
We integrate SMS with different downstream policy Distribution Entropy Reduction (DER) from [3]. We multiply the semantic distribution with the geometric distribution as the input to DER. DER selects the action minimizing the distribution entropy after taking the action. We use the same setup and scenes as in Section 1.2. We report the results for 100 scenes with 12 objects in Pharmacy domain in Table 9.
D.4 Physical Experiments
We generate the physical scenes with the scene generation procedure outlined in Section D.1 to ensure the scenes were not biased. Examples of physical environment layouts are shown in Figure 6.
Because the RealSense camera is not able to capture the fine details of the text on the objects when observing the entire scene at resolution 640 x 480 pixels, we perform three-stage scan of the scene by moving the end-effector to adjacent positions, all of which are closer to the shelf, where the text is more casily readable. At each of these poses, we take a picture of the scene, project the known world position of the objects to the new camera frame, identify text with OCR, and assign each text detection to the object it is contained in. If there are detections on the same object from multiple scan locations, we use the OCR that has the lowest entropy for its distribution, a measure of confidence. During the physical experiments rollouts, when the action given by the policy causes
22

---

Figure 6: Here 3 example from the physical mechanical search experiments in the constrained environment setting.
Spatial Distribution
Semantic Distribution
Semantic Spatial Distribution
Step
Action
Step
Target Revected
Figure 7: Physical rollout example with the target object being the probiotics. Top: the spatial distribution,
semantic distribution and semantic spatial distribution step Bottom: RGB observations at step0, action given by DAR
RGB observation after executing action.
unintentional toppling or missed grasp due depth sensor noise, we reset the object to undo the action and run the policy again.
We show a physical experiment rollout with the target object being the probiotics in Figure 7. In this rollout, the spatial distribution generated based on geometric information by LAX-RAY indi- cates the left side of the shelf occludes more area. However, the semantic distribution generated by SMS indicates the target object more likely be on the right. This is because other objects from the supplements category where the target object probiotics belongs to are visible on the right. Com- bining the spatial distribution and semantic distribution into the semantic spatial distribution takes into account both the geometry and semantic information and results in more accurate distribution.
D.4.1 Ablating Object Lists for Semantic Distribution for Physical Experiments
Metric
Uniform Dist.
SMS-LLM w/o Object List
SMS-LLM w/ Object List
JSD 4
0.554 +: 0.006
0.421 +: 0.032
0.382 + 0.036
Table We measure the deviation the semantic distribution generated by these methods and the ground truth using JSD. SMS with the object list, which uses Object Detection+OCR, outperforms SMS without the object list, which uses the Crop Generation + Image Captioning pipeline.
To evaluate the benefit of object lists, we compare the performance of our method on shelves with and without access to the object list by computing the Jensen-Shannon Distance (ISD) [109] be- tween the generated distribution and the ground truth distribution on the 12 physical shelves as in Section 5.2.2. From Table 10, we see that SMS achieves better semantic distribution compared uniform prior in both cases. The SMS-LLM w/o Object List uses the same Crop Generation Image Captioning pipeline the open-world experiments which is equivalent using VLM for scene understanding. The SMS-LLM w/Object List uses the object list for object detection and to refine the labels using OCR. We see that knowing the object list improves results, which is ex- pected as it reduces the noise in the scene understanding and leads to a higher quality of the semantic distribution.
D.5 Experiments for Open-Workd Environments
We also ablate the modules SMS-E E with results shown in Table 11, where we study the impact of using CLIP weighting, different image captioning models and SAM crops. As mentioned previously that image captioning be noisy, we CLIP to verify the captions. We refer this CLIP weighting. Without this, the performance drops by 21%. When we use BLIP-IC instead of BLIP-2
23

---

Ablations loU
w/o CLIP Weighting
BLIP-IC 0.310 + 0.043
w/o SAM
SMS-E 0.391 0.039
0.307 0.038
0.286 0.038
Table 11: IoU results for ablated SMS-E. w/o CLIP Weighting doesn't using CLIP to refine the generated captions as described Section 4.2. BLIP-IC use BLIP-IC get the descriptions for each crops instead of BLIP-2. BLIP-IC (C's is linked in the Appendix Section D.5. w/o SAM doesn't I use crops given by SAM and crops generated by multi-scalc sliding windows are used.
Navigation
Query
stack of books
Pontion
Poctonn
Pectinn
RGB
CLIP
SMS-E
Figure Object navigation experiment in BAIR Office Kitchen. short horizon navigation example where we start at position and end at position 2. SMS is able to correctly maneuver to the stack of books while CLIP fails because its bag-of-words nature susceptible to incorrectly assigning high prohahility to the pillars in the scene as they are semantically related to "stack."
for image captioning. the performance drops by 21%. Finally, without cropping using SAM to get object-centered crops, the performance drops by 27%.
More examples of the semantic distribution comparison between SMS and CLIP-based models are shown below. BLIP-IC is the large image captioning model of BLIP.
We also conduct a preliminary navigation experiment where a mobile robot follows the downstream navigation policy described in Section 4.3 and selects the physical location to move based on the semantic distribution from CLIP and SMS-E. As shown in Figure 8, CLIP makes incorrect turn (at position it continues in the direction of the view with the red box) because of its bag-of-words behavior and attributes "stack of books" to having higher semantic similarity to concrete pillars in the scene rather than the area with office desks and chairs. SMS-E continues towards the office (green box in position 1) and finds a stack of books on a desk successfully.
D.6 Object Lists in Closed-Wone World Environments
Pharmacy Domain vitamins, fish oil, omega-3. calcium, probiotics protein powder, COQIO anthocyanin, shampoo, conditioner. toothpaste toothbrush. dental floss, face wash sunscreen lotion hand cream body wash aspirin, tylenol, ibuprofen, advil, pain relief,, shaving cream, eye drops, deodorant, band-aid
Kitchen Domain spoon ladle spatula, tongs. whisk fork, peeler, grater, saucepan frying pan, salt, pepper, cumin, coriander, basil, turmeric, parsley oregano, sugar, flour cornstarch oats quinoa rice
pen, , pencil, highlighter, sticky note binder paper, , printer paper, index card, paper clip, rubber band, stapler, staples, tape dispenser 3-hole punch, dry crase marker, sharpic,
Office Domain

---

label maker, notebook eraser white-out, calculator, thumbtack pencil sharpener, bubble wrap, styrofoam, packing tape, shipping boxes, ethernet cable, modem, router, network card, network bridge. headphones, speakers, aux cable, microphone keyboard, mouse USB adapter, hard drive flash drive
E
Object Lists and Examples for Open- World Environments
In this section, we show more examples of the semantic distributions generated by different methods from the static dataset and all the scenes.
Grocery Object list: 'blender', 'juicer', 'spatula', 'spray tan', 'sunglasses', 'gardening gloves', grass seeds'. 'headphones', 'pruning shears', 'SD card', 'crayons', 'paper towel', 'plunger'. 'Pow- erade", 'router', 'bottle opener', 'corkscrew', 'Danimals', 'Paneer', 'yogurt', "hagel', 'baguette', "daisy", 'danish pastry', 'red rose', 'toaster strudel', 'cocoa powder', 'incense sticks', 'succulents', "condensed milk', 'kale", 'lotion', 'scotch tape', 'garlic bread', 'moisturizing masks'
Office Object list 'box of paper', 'cat food', 'ice', 'leftover meatloaf", 'three ring binder', 'Bud- weiser beer', "coke can', 'lion figurine', 'tequila', 'panda soft toy'. 'acetone", 'facial cotton pad", 'HDMI cable', 'lipstick', 'pillow', 'beef patties', 'expo marker', 'mayo', 'relish', 'USB flash drive', 'thumb tacks', 'chain', 'Pringles', "trail mix', 'iPhone charger', 'throw blanket', 'shears', 'emer- gency whistle", "office party notice', 'yogurt', 'napkin holder', "Academy Award'
Home Object list 'bar soap'. 'boarding pass', 'empty bottle', 'pajamas', 'toothpaste', 'mi- crophone', 'mail", 'laundry brush', 'paint roller', 'hand wraps', 'paddle', 'alarm clock', 'duvet', "medal", 'pool balls', 'pool rack", 'soap', 'tissues", 'Neosporin', 'HP scanner', 'resistance bands', jump rope'
25

---

Figure 9: Office environments.
26

---

ech
Figure 10: Grocery stores environments.
27

---

Figure 11: Home environments.
28

---

Target Object
Image
SMS-E
SMS-LLM
CLIP
"blender"
"spray tan"
"spatula"
"sunglasses"
"juicer"
"cocoa powder"
"toaster strudel"
Figure 12: Example set 1 of the semantic distributions generated by different methods for grocery stores.
29

---

Target Object
Image
SMS-E
SMS-LLM
CLIP
Cm
C
"gardening gloves"
cn
con
"headphones"
cc
seeds"
cc
CU
"SD card"
"crayons"
"paper towel"
"plunger"
Figure 13: Example set 2 of the semantic distributions generated by different methods for grocery stores.
30

---

Target Object
Image
SMS-E
SMS-LLM
CLIP
"router"
"bottle opener"
"yogurt"
"incense sticks"
"kale"
ABNA
"scotch tape"
"bagel"
Figure 14: Example set 3 of the semantic distributions generated by different methods for grocery stores.
31

---

Target Object
Image
SMS-E
SMS-LLM
CLIP
"box of paper
"ice"
coka can"
"lion figurine"
"aadda soft toy"
"tequila"
Figure 15; Example the semantic distributions generated by different methods for offices.
32

---

Target Object
Image
SMS-E
SMS-LLM
CLIP
"acetone"
"acial
"cat food"
"trail mixe"
"Piingee"
"shears"
whistle"
"nakinn holder"
Figure Example the semantic distributions generated by different methods for offices.
33

---

Target Object
Image
SMS-E
SMS-LLM
CLIP
"boarding pass"
'empty
"loothaasee"
"microphone"
"mail"
"laundry
"aannt roller"
hand
Figure 17: Example the semantic distributions generated by different methods for houses.
34

---

Target Object
Image
SMS-E
SMS-LLM
CLIP
"paddle"
"Neosporin"
"medal"
"HP scanner"
"jump rope"
Tresistance bands"
Figure 18: Example set of the semantic distributions generated by different methods for houses.
35


Reviews:
[{"review_id": "tIEOfrqm8y8", "reviewer": "Reviewer_kaAs", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper, an Open-World Semantic Mechanical Search (OWSMS) framework which leverages the LLMs to generate a semantic occupancy distribution for the target object. And the physical experiments to search object on shelves are conducted. ", "summary_of_recommendation": "This paper provide a reasonable direction to leverage LLMs to practical robotic tasks, while the scenarios are somewhat simple and the impact of the semantic occupancy distribution should be further explained. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "As stated in the Weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths:\n- Generally, the structure of the paper is clear and it is reasonable to take advantage of the proposed OWSMS as a plug-in semantic module instead of an end-to-end planners. It is a good application of LLMs in robotic tasks. \n\nWeaknesses:\n- It is said that the scenarios used in this task are pharmacy shelves and kitchen cabinets, while as seen from the paper and the accompanied video, the scenarios are actually not very challenging and somewhat artificial. In these circumstances, I have some doubt about how important is the role of the semantic occupancy distribution here. And the paper did not present a specific affinity matrix. I think it would be helpful for me to evaluate the role of the semantic occupancy distribution if a detailed affinity matrix with objects detected from the scene for a certain scenario is presented. \n\n- In the experiments, the proposed methods are compared with the LAX-RAY method and some improvement is achieved, while the improvement is actually not very obvious. I think it would be helpful to have a Random baseline in the experiments, as it would reflect how difficult the task is to some extend. And it can also be regarded as an ablation study which evaluates the  impact of the semantic occupancy distribution in the framework. \n\n- Some minor problems such how the state $s_t$ encodes the full geometries, poses, and names of the objects in the scene, some visualization of the scenarios and  semantic occupancy distribution. \n\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "meuix-x9c_", "reviewer": "Reviewer_jFnz", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper considers the problem of active visual search in which a robot is tasked with searching for a named object that may be occluded or is otherwise not immediately visible to the robot and for which its pose is unknown. The paper specifically focuses on learning an occupancy map that models the distribution over the object's location that can be used by a downstream method to plan and execute the search. To that end, the paper proposes Open-World Semantic Mechanical Search (OWSMS), a framework that builds and maintains a pixel-level occupancy distribution over the target object's location based on its semantic similarities to visible objects in the scene, where semantic similarity is modeled in terms of object-object co-occurrence determined either using an LLM or textual embeddings.", "summary_of_recommendation": "The idea of using language models to facilitate active search is interesting and sensible. This work has potential, but the lack of qualitative comparisons to missing related work together with the inconclusive nature of the experimental results make it difficult to judge the significance of the contributions.\n\n\nSEE ABOVE FOR UPDATES AFTER AUTHOR RESPONSE/DISCUSSION", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "1) Section 3: Can the authors clarify what they mean by the assumption that \"the scenes in the environment are semantically organized, meaning that the starting state of the environment is sampled proportionally to their approximate likelihood of occurrence in the real world\"?\n\n2) Section 3: Does the method assume that the geometry of the objects is known exactly? If not, how are the geometries inferred from observations?\n\n3) Section 4.2: How many candidate completions from the LLM does OWSMS-LLM consider?\n\n4) Is it correct that the comparisons to the LAX-RAY baseline provide an ablation of the spatial distribution (i.e., running a version of OWSMS w/o the semantic distribution)?\n\n5) Can the authors justify the use of the Google Taxonomy-based distribution as ground-truth?\n\n6) Section 5.2: How is the experimental setup chosen, namely the number and placement of objects and the choice of target objects? One questions whether the scenarios may have been cherry picked.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Following up on the summary above:\n\nWhen the objects are known a priori, OWSMS identifies their location in the environment using an existing object detection model (ViLD) and OCR method. Without an object list, OWSMS identifies object crops in RGB and generates their captions using BLIP-2. Given the location and label of objects in the environment, the paper proposes two approaches to generating the target object distribution $P(\\textrm{target} \\vert \\text{label})P(\\text{label})$ (I recognize that this expression is an over-simplification): OWSMS-LLM uses an LLM to approximate the likelihood of a candidate location of the target object based on the nearest labeled object $P(\\textrm{target} \\vert \\text{label})P(\\text{label})$; OWSMS-E estimates $P(\\textrm{target} \\vert \\text{label})P(\\text{label})$ using the dot product between textual embeddings of the target and label object. When the objects (labels) are unknown, $P(\\textrm{label})$ is determined by taking the dot product between the image crops and the generated labels embedding, otherwise $P(\\textrm{label})=1$ (i.e., when the objects are known a priori, the detections are treated as truth). The resulting semantic distribution is combined with a spatial distribution (from LAX-RAY) to get the final distribution over the target object's location. This distribution is used by a downstream policy (LAX-RAY in the context of constrained environments and a heuristic in unconstrained environments) to then search for the object.\n\nThe paper evaluates the OWSMS variants in simulated and real-world domains that involve searching for an object on a shelf (constrained environment). The paper presents additional results that involve searching a collection of images taken in an open environment (unconstrained environment). In both cases, paper shows that OWSMS outperforms CLIP and other baselines in terms of the accuracy of the resulting semantic distribution and the number of actions required to find the target object.\n\n\n\nSTRENGTHS\n+ The problem of \"mechanical search\" is important and the idea of using LLMs to facilitate this search, notably to provide semantic information that can be used to hypothesize the location of a target object relative to a known object is sensible and interesting.\n+ The experimental results demonstrate that the manner by which the method incorporates information from the LLM and the word embeddings results in semantic distributions that are more accurate and that, at least in conjunction with spatial information, improve the performance of search.\n+ The paper is largely well written and easy to follow.\n\nWEAKESSES\n- The paper is missing a **large** body of work on language understanding in robotics [1-20]. The list of references in the paper is extensive, but with a significant bias towards papers from the last few years, which makes it difficult to put this work in the context. Particularly relevant are methods that similarly consider searching for a named object in a partially known environment, some using learned models of object-object interaction [3, 4, 6, 12, 1, 14] to similarly maintain distributions over the location of the target (and other unobserved objects). I don't expect the authors to cite all of these papers, but do feel that it's important to reference relevant existing work, even if it is more than a few years old :-)\n- Similarly, the paper omits relevant work in object (active visual) search [21-24], which includes methods that exploit learned semantics and object-object cooccurrence facilitate search. That said, it is worth noting that the authors have cited papers that are more than just very recent papers.\n- The scale of the environments considered during evaluation are particularly small in both size (e.g., 0.8m x 0.35m x 0.57m for the shelf experiments) and the number of objects.\n- The OWSMS-E variant is sensible, but also rather obvious, namely using textual embeddings as a measure of co-occurence. Thus, the significance of this contribution is limited.\n- It's not obvious that OWSMS-LLM is demonstrably better than OWSMS-E, the novelty of which is limited, in terms of search. In some cases, OWSMS-LLM outperforms OWSMS-E, in others OWSMS-E performs comparably or even better.\n- Indeed, the title and algorithm name emphasize \"open-world\" however the arguably straightforward embedding-based version performs better than the LLM variant in the open-world domain, while the latter performs best in the closed-world (known object list) environment.\n- The paper ablates access to an object list only with regards to the resulting distribution relative to a reference distribution based on the Google Taxonomy (the relevance of which as ground-truth is unclear). However, it does not evaluate the effect of task performance (e.g., success rate or number of actions) when the object list is not known. Meanwhile, it is not obvious that the results in Table 4 contribute much. As the paper notes, one would expect the distribution to be better when the list of objects is known a priori.\n- Related, my understanding is that the LAX-RAY baseline does not require the other objects to be known a priori (i.e., no object list). If this is the case, then the results in Tables 2 and 3 don't provide an apples-to-apples comparison.\n- It isn't obvious how the high-level problem considered here differs from active visual search. Assuming that the differences are minimal, why does the problem need to be rebranded as \"mechanical search\"?\n\n\n\nMINOR\n* Line 208: \"the manipulating\" --> \"manipulating\"\n* Table 2: The OWSMS-LLM numbers for Kitchen should not be bold as the differences relative to OWSMS-E are not statistically significant.\n\nREFERENCES\n\n[1] P. Anderson, Q. Wu, D. Teney, J. Bruce, M. Johnson, N. Sünderhauf, I. D. Reid, S. Gould, and A. van den Hengel. Vision-and-language navigation: Interpreting visually-grounded navigation instructions in real environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.\n\n[2] D. L. Chen and R. J. Mooney. Learning to interpret natural language navigation instructions from observations. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2011.\n\n[3] F. Duvallet, T. Kollar, and A. Stentz. Imitation learning for natural language direction following through unknown environments. In Proceedings of the IEEE International Conf. on Robotics and Automation (ICRA), 2013.\n\n[4] F. Duvallet, M. R. Walter, T. Howard, S. Hemachandra, J. Oh, S. Teller, N. Roy, and A. Stentz. Inferring Maps and Behaviors from Natural Language Instructions. In Proceedings of the International Symposium on Experimental Robotics (ISER) June 2014\n\n[5] D. Fried, R. Hu, V. Cirik, A. Rohrbach, J. Andreas, L.-P. Morency, T. Berg-Kirkpatrick, K. Saenko, D. Klein, and T. Darrell. Speaker-follower models for vision-and-language naviga- tion. In Advances in Neural Information Processing Systems (NeurIPS), Dec. 2018.\n\n[6] S. Hemachandra, F. Duvallet, T. M. Howard, N. Roy, A. Stentz, and M. R. Walter. Learning models for following natural language directions in unknown environments. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2015.\n\n[7] T. M. Howard, S. Tellex, and N. Roy. A natural language planner interface for mobile manipula- tors. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2014.\n\n[8] T. Kollar, S. Tellex, D. Roy, and N. Roy. Toward understanding natural language directions. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2010.\n\n[9] M.MacMahon, B. Stankiewicz, and B. Kuipers. Walk the talk: Connecting language, knowledge, and action in route instructions. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2006.\n\n[10] C. Matuszek, D. Fox, and K. Koscher. Following directions using statistical machine translation. In Proceedings of the ACM/IEEE International Conference on Human-Robot Interaction (HRI), 2010.\n\n[11] C. Matuszek, E. Herbst, L. Zettlemoyer, and D. Fox. Learning to parse natural language commands to a robot control system. In Proceedings of the International Symposium on Experimental Robotics (ISER), 2012.\n\n[12] H. Mei, M. Bansal, and M. Walter. Listen, attend, and walk: Neural mapping of navigational instructions to action sequences. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2016.\n\n[13] D. K. Misra, J. Sung, K. Lee, and A. Saxena. Tell me Dave: Context-sensitive grounding of natural language to manipulation instructions. International Journal of Robotics Research, 35 (1-3):281–300, January 2016.\n\n[14] S. Patki, E. Fahnestock, T. M. Howard, and M. R. Walter. Language-guided Semantic Mapping and Mobile Manipulation in Partially Observable Environments. In Proceedings of the Conference on Robot Learning (CoRL) October 2019\n\n[15] R. Paul, J. Arkin, D. Aksaray, N. Roy, and T. M. Howard. Efficient grounding of abstract spatial concepts for natural language interaction with robot platforms. International Journal of Robotics Research, 37(10):1269–1299, June 2018.\n\n[16] M. Shridhar and D. Hsu. Interactive visual grounding of referring expressions for human-robot interaction. In Proceedings of Robotics: Science and Systems (RSS), 2018.\n\n[17] S. Tellex, T. Kollar, S. Dickerson, M. R. Walter, A. G. Banerjee, S. Teller, and N. Roy. Un- derstanding natural language commands for robotic navigation and mobile manipulation. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2011.\n\n[18] J. Thomason, S. Zhang, R. J. Mooney, and P. Stone. Learning to interpret natural language commands through human-robot dialog. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2015.\n\n[19] J. Thomason, J. Sinapov, M. Svetlik, P. Stone, and R. J. Mooney. Learning multi-modal grounded linguistic semantics by playing “I spy”. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI), 2016.\n\n[20] J. Thomason, J. Sinapov, R. J. Mooney, and P. Stone. Guiding exploratory behaviors for multi-modal grounding of linguistic descriptions. In Proceedings of the National Conference on Artificial Intelligence (AAAI), 2018.\n\n[21] A. Aydemir, A. Pronobis, M Göbelbecker, and P. Jensfelt. Active visual object search in unknown environments using uncertain semantics\" IEEE Transactions on Robotics 29, no. 4 (2013): 986-1002.\n\n[22] J. K. Tsotsos.On the relative complexity of active vs. passive visual search. International Journal of Computer Vision 7, no. 2 (1992): 127-141.\n\n[23] L. E. Wixson, and D. H. Ballard. Using intermediate objects to improve the efficiency of visual search. International Journal of Computer Vision 12, no. 2-3 (1994): 209-230.\n\n[24] S. Ekvall, D. Kragic, and O, Jensfelt. Object detection and mapping for service robot tasks. Robotica 25, no. 2 (2007): 175-187.\n\n\n\nUPDATE AFTER AUTHOR RESPONSE AND DISCUSSION\n\nThe authors clearly made an effort to respond to the questions and concerns that I raised in my initial review and the follow-up discussion.\n\nThe authors addressed some of the issues that I initially identified. However, I still feel that the paper should be evaluated as a systems paper that describes the integration of existing methods to perform semantic search for occluded objects. As I mentioned in my reply to the authors, it is my view that a systems paper should either provide novel insights into the problem of mechanical (active visual search), e.g., through a compelling experimental evaluation, or show how using existing methods requires combining them in novel ways. I'm just not sure that the paper does this. The paper lacks comparisons to relevant active visual search methods and instead compares to OWL-ViT and a CLIP baseline that, to the best of my knowledge, are not designed to deal with occlusions. Are the\n\nI note that regarding the discussion of labeling the method as \"mechanical search \" vs. \"active visual search\" the authors point to the definition of active visual search (AVS) in Aydemir et al. [1] and suggest that it differentiates from mechanical search in that the latter must deal with occlusions. However Aydemir et al. also discuss the need to mitigate occlusions: \"Some of the advantages of an active strategy discussed are robustness to occlusions, possible increase of resolution, and use of motion to disambiguate vision-related aspects of the world such as varying illumination conditions.\" Later, in differentiating their work from a specific POMDP-based search strategy [2], they state that \"This [visibility assumptions that Hollinger et al. [2] make] is a big simplification of detecting an object with a camera since the task of finding an object in a place as big as a room involves many difficulties such as calculating a good viewing position, dealing with occlusions, and detecting objects that appear small in the image.\" I fail to see a fundamental distinction between searching for an occluded object on a shelf using a robot arm and searching for an occluded object in a multi-room environment. Again, this goes beyond the simple question of whether \"mechanical search\" vs \"active visual search\" is appropriate and instead relates to the importance of putting this work in context.\n\nMinor: I thank the authors for looking at the myriad of papers that I listed in my initial review. Admittedly, I may have gone a bit overboard with such a long list and It was not my intention that the authors cite most of these papers, but rather the ones that are particularly relevant with regards to using information extracted from textual descriptions to find an object that is not visible to the robot and the location of which is unknown. This goes beyond instances of natural language understanding as cited in the revised paper.\n\n[1] Aydemir, Alper, Andrzej Pronobis, Moritz Göbelbecker, and Patric Jensfelt. \"Active visual object search in unknown environments using uncertain semantics.\" IEEE Transactions on Robotics 29, no. 4 (2013): 986-1002\n\n[2] G. Hollinger, D. Ferguson, and S. Srinivasa, S. Singh, “Combining search and action for mobile robots,” in Proc. IEEE Int. Conf. Robot. Autom., May 2009, pp. 800–805.", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "WWua1K9muJt", "reviewer": "Reviewer_5M8F", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposed the task of open-world mechanical search, which extends mechanical search to the open-world (unseen vocabulary, unseen objects) evaluation setting.\nAlong with the task, the paper introduced an approach for combining off-the-shelf pre-trained VLMs, LLMs, object detectors with a prior works' mechanical search solution.\nUsing LLM completion logprobs as a proxy for semantic distance (in the taxonomy tree) between detected objects, the approach combines these semantic distances into one semantic map.\nThe manipulation policy uses a map that combines the aforementioned semantic map (1D for shelf mechanical search, 2D for object navigation) and a prior work's spatial occupancy distribution map to decided which object to manipulate.\nThis approach improves mechanical search success rates marginally and action-efficiency significantly on average across the pharmacy, kitchen, and office domains in simulation, as well as in real world.", "summary_of_recommendation": "My biggest concern is about the evaluation, which I believe can be more systematic, complete, and reproducible. As with many other open-world papers leveraging large pre-trained models, the approach lacking novelty is not a basis of rejection as long as the proposed system design is sufficiently ablated and insights about the system's design are discussed such that others can learn from and improve upon the results. With the paper's current evaluation, my recommendation is a strong reject.\n\n**Update**\nThe biggest change the authors made in my opinion include a change in scope from open-world to closed-world. This reduced the systematic requirements for evaluation, my biggest concern. The authors have ran a lot of experiments which ablated various parts of the proposed pipelines in closed-world settings, and provided an un-ablated result on an open-world object navigation search task. Although the approach is still not novel, the experiments are quite thorough and well-executed.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "### LLM completion probability design \n\nThis is more of a minor observation than a question, but it could be discussed in the limitations on how completion probabilities is only a proxy for semantic distance.\n\nThe authors described on L183 \"The affinity score for the target object with each label is the completion probability for the tokens that represent the target object\".\nSince completion probability for the label is proportional to exponential of sum of token log probs, longer object names will likely result in lower logprob.\n\nFor instance, using `text-davinci-003`  with probabilities enabled on the [OpenAI playground](https://platform.openai.com/playground), I used the approach's prompt to find a `toothbrush`:\n```\nI see the following in a room: {obj}. This is likely to be the closest object to toothbrush.\n```\n\nreliably gave the following log probabilities for `{obj}`:\n\n- `colgate toothpaste tube`: -33.71 logprob, 5 tokens. \n- `toothpaste tube`: -26.04 logprob, 3 tokens. \n- `paste`: -21.23 logprob, 1 token. \n- `toothpaste`: -19.83 logprob, 2 tokens. \n- `tooth`: -16.87 logprob, 1 token. \n- `cat`: -15.55 logprob, 1 token. \n\nWhich means the approach will prioritize searching behind the cat first.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Strengths\n - Mechanical search system was demonstrated on real robot\n - Large number of simulation-based evaluation episodes with standard errors for mechanical search.\n - Interesting Google Taxonomy based approach for semantic arrangement of simulation scenes. This has potential to be scalable approach to generate a large-scale simulation evaluation suite.\n\n## Weaknesses\n\n### Incomplete, Unsystematic Evaluation\n\nThe proposed solution is a common zero-shot recipe (e.g., [4,5]) of fusing VLMs and LLMs into a plug-and-play scene representation (e.g. [3,6]) for downstream low-level manipulation.\nSince this zero-shot recipe is not novel, demonstrating that it improves over prior work without VLMs and LLMs is not a contribution.\nHowever, applying the recipe to different robotic zero-shot tasks is still educational and informative because it establishes empirical rankings between different VLMs and how to use them for each task, usually with high statistical confidence (using lots of episodes, because pre-trained models are used zero-shot).\nFurther, it allows *systematic* variation in open-world properties (uncommon objects, appearance descriptions, etc. in [3], unseen objects, vocabulary, domains in [2]).\nCoW [3] extensively and systematically evaluated 3 VLMs, multiple different grounding approaches with CLIP, prompts, on two different simulation benchmarks.\n\nFrom empirical findings in [3-6], comparing LAX-RAY with OWSMS (Table 2, 3) is necessary but not informative.\nAblations like OWSMS-E and OWSMS-LLM are useful, but not complete (e.g., lacks insights in prompts to justify L181-183, object detectors other than ViLD).\nThis means the proposed approach's system vision pipeline of ViLD + OCT => SAM => BLIP-2 => CLIP is a large, unablated pipeline, making its design difficult to learn from for readers.\n\nFurther, their evaluation for unconstrained environments does not include mechanical search and only object navigation, has a small sample size (10 scenes), is not systematic (does not vary different open-world property separately), and is not reproducible (real environment not accessible for follow-up work, robot was teleoperated).\n\nI would like to see empirical results using a simulation benchmark that is reproducible that systematically (different system level design, VLM prompt, and VLM model choices, and evaluate the same design choices for all settings, not just for unconstrained environments) ablates the proposed system.\n\n### Lack of Visualizations\n\nI would like to see more visualizations of the real world object navigation example scenes, and the complete list of objects in the ten scenes.\nWithout this, it becomes difficult to interpret the paper's custom unconstrained evaluation benchmark's quantitive results.\n\n### Open-Vocabulary, not Open-World\n\nFrom [9]'s visualizations, FOSS only contains abstract extruded cylinders and cuboids, devoid of visual-semantic meaning. This means visual distribution shifts -- a core part of open-world generalization -- is not tested.\n\n\n\n## Areas for Improvement\n\n - **Evaluation**:\n   - Make evaluation systematic and reproducible.\n   - If the authors wish to claim a plug-and-play representation for open-world search (encompass both open-world mechanical search and object navigation) then compare approach on CoW benchmark, to allow for reproducible open-world evaluation.\n   - If the authors wish to claim a plug-and-play representation for only open-world mechanical search, then provide open-world (i.e., unconstrained) evaluations for mechanical search.\n - Add more real-world unconstrained visualizations\n - Add complete list of real-world 5 real world objects from all 10 scenes in Supplementary Materials\n - **Second contribution**. \"An empirically evaluated pipeline for generating object labels in open-world scenes without access to any object lists\" was not ablated. For instance, using all class names in Google Taxonomy's relevant class (e.g., OWL-VIT [8] uses all category names in LVIS) is a good baseline.\n - **Fair pre-trained baseline comparisons**: large pre-trained models are useful but noisy (as mentioned in L172), and using averaging over cropping and flipping augmentations is a well explored technique. For instance, [2] found that cropping + jittering leads to better CLIP relevancy maps. The baselines in open-world evaluation should also use cropping augmentations for CLIP and OWL-VIT baselines.\n - **Writing**:\n   - \"For unconstrained environments, results suggest that our semantic distributions outperform those generated by CLIP-based methods\" L20-22: This was neither in open-world mechanical search nor open object navigation. Meanwhile, open-world open navigation contains only one non-reproducible example.\n   - LLM only approaches are mentioned in related works (L122-127), with their disadvantages being that they require non-language information and domain-specific policies. However, LAX-RAY, the \"speciallized planning and manipulation [policy]\" (L130) that OMSMS was designed to work with, is a domain-specific policy which uses non-language information.\n   - L129 \"distills\" typically implies some learnable parameter. Consider changing to another word, such as \"uses\".\n   - What is \"Spatial NN\" (Table 2)? NN was not defined anywhere.\n\n\n## Citations\n\nSide note: [1-2, 5] are relevant missing citations. [3-4,6-8] have been cited in the work.\n\n - [1] [Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers](https://hila-chefer.github.io/publication/generic-attention-model-explainability/)\n - [2] [Semantic Abstraction: Open-World 3D Scene Understanding from 2D Vision-Language Models](https://semantic-abstraction.cs.columbia.edu/)\n - [3] [CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation](https://cow.cs.columbia.edu/)\n - [4] [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://say-can.github.io/)\n - [5] [Grounded Decoding: Guiding Text Generation with Grounded Models for Robot Control](https://grounded-decoding.github.io/) \n - [6] [Open-vocabulary Queryable Scene Representations for Real World Planning](https://nlmap-saycan.github.io/)\n - [7] [LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action](https://sites.google.com/view/lmnav)\n - [8] [Simple Open-Vocabulary Object Detection with Vision Transformers](https://arxiv.org/pdf/2205.06230.pdf)\n - [9] [Mechanical search on shelves using lateral access x-ray](https://research.google/pubs/pub50619/)", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "-DDinHzttaW", "reviewer": "Reviewer_D6LF", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents Open-World Semantic Mechanical Search (OWSMS), a large language model driven approach to mechanical search, in which locating and retrieving a target object may require navigation and or manipulation. CLIP approaches often fail since the objects being searched for are often dissimilar to the objects around them. The proposed approach queries a large language model for semantic information about the relative location of observed objects to the target object and then uses this \"semantic distribution\" to inform where to look. In image space, segmented patches from Segment Anything are used as inputs to generate the semantic distribution, which then provides affinity scores for each patch, each describing how likely the object in image space is likely to be most near to the object of interest. The affinity scores are then used for downstream tasks including shelf searching and object-goal navigation, showing improvements in both simulated and real-world environments.\n", "summary_of_recommendation": "This paper is well written and presents a creative and effective solution to a common problem in household robotics. The experiments are also impressive, showcasing the possibility of pretrained VLMs and LLMs to aid in increasingly sophisticated robot tasks. Additional visuals (and in particular a rework of Fig. 1) would greatly help readability and understanding of the impact of results.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "None. The paper is clear and easy to follow and the results well-motivated and impressive. I am happy to work with the authors during the rebuttal phase if they have thoughts about adding additional visuals and the trade offs space limitations would make necessary.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is well written and presents a creative and effective solution to a common problem in household robotics. The experiments are also impressive, showcasing the possibility of pretrained VLMs and LLMs to aid in increasingly sophisticated robot tasks. There is ultimately not too much to say about this work: it presents a well-scoped solution to a common problem and contains an impressive amount of experiments in both simulation and in the real world.\n\nAs such, most of my comments are mostly optional suggestions for improved readability and clarity.\n\nThe paper's biggest weakness is its density, something difficult to remedy without significant changes. As written, the paper favors building intuition via prose to including additional visuals to aid in understanding the low-level methodology and results. For example, Figure 1 is clean and nice to look at, but does not do a particularly effective job at showcasing the low-level process by which the different stages are related to one another. For instance, it was unclear until reading on page 4 (two full pages after the figure appears) what how the \"semantic distribution\" and \"affinity scores\" were related to one another. Similarly, only a single results figure is provided in the main body of the text and, while helpful, it only helps to visualize the results for Sec. 5.2. Additional figures (even those in the appendix) to help better understand the results from 5.1 would be a welcome addition.\n\nThese are perhaps smaller issues, since indeed the full method is understandable in the current version of the work and the paper stands well on its own. However, some effort from the authors to favor more visuals (or at least include more such visuals in the appendix) would likely improve the impact of the work.\n\nScattered other comments:\n- I don't believe it is mentioned which LLM is used for these experiments. I understand these days that naming which LLM is being used can often de-anonymize authors, so an acknowledgment by the authors that this information will be included in the final version will be sufficient here.\n- [typo?] The title of Fig. 2 is \"Object navigation experiments in real\". Perhaps this should end with \"real world environments\" or similar.\n- The limitations section should perhaps include a comment about the quality of the individual tools upon which the proposed method relies. For example, Sec. 5.2 mentions that performance is limited by the quality of labels from BLIP-2. Reiterating this point (and similar) in limitations if there is space would be welcome, but is not necessary.\n", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a new method for semantic mechanical search using large vision and language models.

The paper causes heat discussion during the rebuttal period and the reviewer discussion period.
In the end, majority of  the reviewers agree on accepting the paper. However, there is still concerns on the novelty of the proposed method.
Therefore, I recommend accepting the paper as a poster presentation.
Decision: Accept (Poster)