Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs
Authors: Haonan Chang
Kowndinya Boyalakuntla
Shiyang Lu
Siwei Cai
Eric Pu Jing
Shreesh Keskar
Shijie Geng
Adeeb Abbas
Lifeng Zhou
Kostas Bekris
Abdeslam Boularias
Venue: CoRL 2023 Poster (2023)
Keywords: Open-Vocabulary Semantic
Scene Graph
Object Grounding

Abstract: We present an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework for grounding a variety of entities, such as object instances, agents, and regions, with free-form text-based queries. Unlike conventional semantic-based object localization approaches, our system facilitates context-aware entity localization, allowing for queries such as “pick up a cup on a kitchen table” or “navigate to a sofa on which someone is sitting”. In contrast to existing research on 3D scene graphs, OVSG supports free-form text input and open-vocabulary querying. Through a series of comparative experiments using the ScanNet dataset and a self-collected dataset, we demonstrate that our proposed approach significantly surpasses the performance of previous semantic-based localization techniques. Moreover, we highlight the practical application of OVSG in real-world robot navigation and manipulation experiments.  The code and dataset used for evaluation will be made available upon publication.

Context-Aware Entity Grounding with Open-Vocabulary 3D Scene Graphs
,Shiyang Lu', Siwei Cai², Eric Pu Jing', Shreesh Keskar! Haonan Chang', Kowndinya Boyalakuntla', Shiiayk
Shijie Geng', Adeeb Abbas?, Lifeng Zhou?, Kostas Bekris', Abdeslam Boularias'
hc8866, kb1204, 011220,ss11642, sl1642, epjs. htt.:h/....g0....oo...oaaaaaaaaa www.:aili..ttttteeeemmmmmo
*sc35680drexel. edu, adesbabbs@gmail. aed.aaaaeeeaaabbeeaaaaaa....... 1o,, edd edu
'Rutgers University Drexeel University
Abstract: We present an Open-Vocabulary 3D Scene Graph (OVSG), formal framework for grounding a variety of entities, such as object instances, agents, and regions, with free-form text-based queries. Unlike conventional semantic- based object localization approaches, our system facilitates context-aware entity localization, allowing for queries such as "pick up cup on a kitchen table" "navigate sofa which someone is sitting". In contrast to existing research 3D scene graphs, OVSG supports free-form text input and open open-vocabulary querying. Through series of comparative experiments using the ScanNet [1] dataset and self-collected dataset, we demonstrate that our proposed approach significantly surpasses the performance of previous semantic- techniques. Moreover, we highlight the practical application of OVSG real- world robot navigation and manipulation experiments. The code and dataset used for evaluation can be found https:/ggitbb.ccmmcchahhnnnnnnnn
localization
Keywords: Open- Vocabulary Semantics, Scene Graph, Object Grounding
Introduction
In this paper, we aim address a fundamental problem in robotics grounding semantic entities within the real world. Specifically, we explore how unambiguously and accurately associate entities present in commands, such as object manipulation, navigation specific location, or communication with particular user.
Currently, the prevailing method for grounding entities in the robotics domain is semantic detec- tion [2]. Semantic detection methods are intuitive and stable. However, in scenes with multiple en- tities of the same category, semantic labels alone cannot provide unique specification. In contrast, humans naturally possess the ability to overcome this grounding ambiguity by providing context- aware specifications, such detailed descriptions and relative relations. For example, rather than simply designating "a cup", humans often specify "a blue cup on the shel?", "a coffee cup in the kitchen", or "Mary's favorite tea cup".
Inspired by this, a series of recent works introduce context relationship into grounding problem [3, 5. 6, 7]. These approaches employ 3D scene graphs as scene representation that concurrently accounts for instance categories and inter-instance spatial contexts. In 3D scene graph, concepts such people, objects, and rooms depicted nodes, with attributes like color, material, and affordance assigned as node attributes. Moreover, spatial relationships are represented graph edges. Such structure enables 3D scene graphs to seamlessly support context-aware object queries,
7th Conference Robot Learning (CoRL. 2023), Atlanta, USA.

---

such as "the red cup on the table in the dining room", provided that the attribute, the semantic category, and the relationship have been predefined in the graph.
However, this inevitably brings us to more crucial question that this paper aims to answer: how do we cope with scenarios when the class category, relationship, and attribute are available in the constructed 3D scene graph? Tackling this question is vital if we wish to effectively integrate robots into real-world scenarios. To resolve the challenge, we present a novel framework in this paper - the Open- Vece-Voaa 3D Scene Graph (OVSG). To the best of our knowledge, OVSG is the first 3D scene graph representation that facilitates context-aware entity grounding, even with unseen semantic categories and relationships.
To evaluate the performance of our proposed system, conduct series of query experiments ScanNet [1], ICL-NUIM [8], and self-collected dataset DOVE-G (Dataset for Open- Vocabulary Entity Grounding). We demonstrate that by combining open- ppen-vocabulary detection with 3D scene graphs, we can ground entities more accurately in real-world scenarios than using the state-of-the-art open-vocabulary semantic localization method alone. Additionally, we designed two experiments to investigate the open-vocabulary capability of our framework. Finally, we showcase potential applications of OVSG through demonstrations of real-world robot navigation and manipulation.
Our contributions are threefold: 1) A new dataset containing eight unique scenarios and 4,000 lan- guage queries for context-aware entity grounding. 2) A novel 3D scene graph-based method to address the context-aware entity grounding from open- vocabbaayyy queries. 3) Demonstrate the real- world applications of OVSG, such as context-aware object navigation and manipulation.
2 Related Work
Open-Vocabulary Semantic Detection and Segmentation The development of foundation vision- language pre-trained models, such as CLIP [9], ALIGN [10]. and LiT [11], has facilitated the progress of 2D open-vocabulary object detection and segmentation techniques [12, 13, 14, 15, 16, 17, 18]. Among these approaches, Detic [16] stands out by providing open-vocabulary instance- level detection and segmentation simultaneously. However, even ttate-of-the-- art single-frame meth- ods like Detic suffer from perception inconsistency due to factors such as view angle, image quality, and motion blur. To address these limitations, Lu et al. proposed OVIR-3D [19], method that fuses the detection result from Detic into an existing 3D model using 3D global data association. After fusion, the 3D scene is segmented into multiple instances, each with unique Detic feature attached. Owing to its stable performance, we choose OVIR-3D as our semantic backbone.
Vision Language Object Grounding In contrast with object detection and segmentation, object grounding focuses on pinpointing an object within 2D image or 3D scene based on textual input. In the realm of 2D grounding, various studies, such as [20, 21, 22, 23], leverage vision-language alignment techniques to correlate visual and linguistic features. In the 3D context, object grounding is inherently linked to the challenges of robot navigation, thus gaining significant attention from the robotics community. For instance, CoWs [24] integrates CLIP gradient detector with navigation policy for effective zero-shot object grounding. More recently, NLMap [25], ConceptFusion [26), CLIP-Fields [27] opts to incorporate pixel-level open-vocabulary features into a 3D scene recon- struction, resulting in queryable scene representation. While NI.Map overlooks intricate relation- ships in their framework, ConceptFusion claims to be able query objects from long text input with understanding of the object context. Thus, we include ConceptFusion one of our baselines for 3D) lasgunelanguage grounding.
3D Scene Graph 3D scene graphs provide an elegant representation of objects and their rela- tionships, encapsulating them as nodes and edges, respectively. The term "3D" denotes that each node within the scene possesses a dimenssonlii position. Such graph structure has been widely researched for decades in robotics community [3, 4, 5, 7, 28, 29].. In [3], Fisher et al. first in- troduced the concept of 3D scene graphs, where graph nodes categorized by geometric shapes. Armeni et al. [4] and Kim al. [5] then revisited this idea by incorporating semantic labels to graph

---

X Fly Position Inpuc
odjjctt
e aggent
region
AA ubstruut
spaciin
Gs
yf
G Lyg User Input
O RGBD Scan
Fi Fom
tab
pdd,; belong
R
OVIR-3D
Encoding
fimee
fix
fi
foiaka
uzu
leg "I want to find Tom's 's bottle laboratory"
object
aggen
regiion
L.l, abstruct
p.2,0; spacial
Graph Simblarity
1% bottle
Tom
Ft Priteennccy
1aio; berfony
(a200
== 0.6
+
Ga
D LLM Parse
Trup
Fib
12
Encoders: Sentence-BEir
O
GloVe
O
Spotial Encoder
O
CLIP Encoder
Figure 1: This is an illastation of the proposed pipeline. The system inguts the positional isput P.., use ingut Lu.. RGBD Scno and query language Lig- The top depicts the constraction of C... Both P., L., disectly fed O., The RGBD language descriptions for agent region converted
inguts
open-oocabulayy
system referred
to OVIR JD. system ouguus a positicn fentures via difforent encoders, unique Sparal Relatioushpp Encoder is employed to escode spatial relationship features from pose pairs. The bottom section shows the building of Gg- The query Lg used in this etample "I want to find Tom's bottle in the laboratory." An LLM used pane it imo various elements, each with description and typo. These descriptions are then encoded features different encoders, forming G. Finally. grounding the query language within soone becomtes a probiem locating G. within C.. candidate proposal makking algorition are introduced represented by the central mode of the selecied candidate.
for obyect. Subsequestly,
purpose. The cstity we
nodes. These works establish good foundation for semantic- aware 3D scene graphs, demonstrating that objects, rooms, and buildings can be effectively represented as graph nodes. Recently, Wald et al. [7] showed that 3D feature extraction and graph neural networks (GNN) can directly infer seman- tic categories and object relationships from raw 3D point clouds. Rosinol et al. [6] further included dynamic entities, such as users, within the scope of 3D scene graph representation. While 3D scene graphs exhibit great potential in object retrieval and long-term motion planning, none of the existing methods support open- vocabulary queries and direct natural language interaction. Addressing these limitations is crucial for real-world deployment, especially for enabling seamless interaction with users.
3 Open-Vocabulary 3D Scene Graph
3.1 Open-Vocabulary 3D Scene Graph Representation
An Open-Vocabulary 3D Scene Graph (OVSG) is denoted as G = EV, where V signifies graph nodes and E stands for graph edges. Each node o² in V = [0'} (t', P. con- sisis of a node type tt, a open-vocabulary feature f a language description 16 (optional), and the node index. In this study, we identify three primary node types, t': object, agent, and region. The open-vocabulary feature f' associated with each node vs is contingent
3D position p' (optional);
the node type t, The encoder utilized for f is accordingly dependent on 4. The 3D position p¹ = {æe, Ve, Se, Tmin, Imin, Zmin, Emar: Vmazz Zmaz) of each entity is de- fined by a 3D bounding box and its center position. Edges in the graph are represented by E = fer lu', wd E V), e = {rijjk = [tt.ik fi.jkk k p.id |kk = O, O...... Each edge e en- capsulates all relationships phcik between the nodes v² and v.. The triplet notation (i, k) refers the th relationship between node of and v9, tissk indicates the type of this relationship. We primarily categorize two relationships in this study: spatial relations and abstract relationships. short sen- tence (' is optionally provided to describe this relationship. The feature fiijk encodes the semantic meaning of the relationship, whose encoder depends on fiijkk For a more detailed definition of these types, please refer to Section 3.3.
The primary distinction of OVSG from conventional 3D scene graph work is its utilization of se- mantic features, instead of discrete labels, characterize nodes and relationships. These features either directly trained within the language domain like Sentence- BERT [30] and GloVe [31], or aligned to it, as seen with CLIP [9] and Detic [16]. The versatility of language features enables
3

---

OVSG to handle diverse queries. The degree of similarity among nodes and edges is depicted using a distance metric applied to their features:
{1 {1-
00 dot(f', f3)
if * t
dist(v", w²)
;dist(e"d, (²0))
<kelet"/,. dist(r"alle pume.
min
else
0c 1 , faa,, SRP(FIAA piikk f
if tith + t"
dist(rrisik, ranpe, yomiee
if ti..k = tu,,, * spatial
if tf = t".u," = spatial
where the |e-0| and |e" are the number of relationships inside eiJ and e" SRP refers to a Spatial Relationship Predictor. Check Section 3.3 and Appendix B for more details. Noticeably, the distance across different types will not be directly compared. These distances will be used to compute the type-fite free index in Section 3.4.
3.2 Context-Aware Open-Vocabulay Entity Grounding
The problem we address can be formally defined using the open-vocabulary scene graph concept as follows: Given a scene, represented as S, our objective to localize an entity, referred to as 8, using natural language, represented as Lg. within the context of the scene S. Essentially, we seek to establish mapping æ such that π (S). An RGBD) scan of the scene user linguistic input Lu, and position input Pa provided to facilitate this process. Significantly, the query language by may encompass entity types and relationship descriptions not previously included in the scene graph construction phase.
Our proposed procedure can be separated into two main stages. The first stage involves the con- struction of the scene graph. From the user imput L, and the RGBD scan we construct an open- vocabulary scene graph (OVSG) for the entire scene, denoted G,. This is one-time process that can be reused for every subsequent query. When new query is introduced, OVSG using the query Ly. denoted as Gq- Once we have both scene graphs G2 and Gy, we proceed to the second stage, which is the graph matching stage. Here, we match the query scene graph, Gg. with a sub-graph from the whole scene graph, G,. The queried entity is situated within the matched sub-graph.
also construct an
3.3 3D Scene Graph Building
Type definition Prior to delving into the scene graph construction procedure, we first delineate the categories of node types and edge types this paper pertains to. The term Object signifies static ele- ments within a scene, such as sofas, tables, and so forth. The term Agent is attributed to dynamic, interactive entities in the scene, which could range from humans to robots. Region indicates a spe- cific area, varying in scale from the surface of tabletop to an entire room or building. Regarding relationships, spatial describes positional relationships between two entities, such Tom being in the kitchen. Conversely, abstract relationships are highly adaptable, enabling us elucidate rela- tionships between an agent and object (for instance, cup belonging to Mary) or the affordance relationship between two objects, such as key being paired with a door.
Input process The inputs for G, consist of an RGBD-scan set J, user language input I.,, and a user position input Pa. The L,, input assigns names to agents and regions and provides descriptions of abstract relationships. P. provides the locations for the agent and region (not including object position), and it can be autonomously generated using existing algorithms like DSGS [6]. Since this process is not the focus of our study, we assume Pa is pre-determined in this paper. The input is an RGBD scan of the entire scene, which is fed into the Open- Vocabulary 3D Instance Retrieval (OVIR-3D) [19] system, a fusion system operating at the instance level. OVIR-3D returns a set of objects, each denoted by position p and a Detic feature Detie
Gg accepts a language query Ly its input. An exemplary query, depicted in Figure 1, is "I want to find Tom's 'ss bottle in laboratory". To parse this language, we utilize large language model (LLM),

---

such as GPT-3.5 or LLAMA. Utilizing a meticulously engineered prompt (refer to Appendix A for more details), we can interpret different entities within the query.
Feature encoding As specified in Eq. 1, the calculation of the similarity between nodes and edges relies heavily on their features. This operation of computing features termed the feature encoding process.
Instead of using a unified encoder as in previous works [25, 26], we choose different encoders for various node and relationship types. Since the inputs of Ge and Gg differ, the selection of encoders for each graph also varies. Object features in G. are generated by deploying OVIR-3D to the 3D scan of the scene. These features are Detic features. Meanwhile, objects in G. are encoded from their names (parsed from LLM during the input process) using the CLIP-text encoder. Because the Detic feature is directly trained align with the CLIP-text feature,
compute distances for object nodes between G, and G3 using Eq. 1. For agent and region nodes in G., they are identified by their names in the user input, L.. Whereas in Gg. agent and region nodes are also specified by names For both of them, we employ Sentence-BERT [30] to encode the language features. As for relationships, we differentiate between spatial relationships and abstract relationships. In G., the input for spatial relationships comes from the positions of the corresponding nodes. In contrast, in Gg. the input for spatial relationships comes from language descriptions 1 parsed from ly by LLM. Given the absence of a standardized approach for spatial-language encoding, spatial encoder for this purpose (see Appendix B). Finally, for abstract relationship features in G., the input is language from user input, L.. In Gg. the input is also textual. We use GloVe to encode these texts on both sides.
trained
Multiple distinct encoders are utilized during the feature encoding step. Different encoders have var- ied emphases, and using combination improve the robustness of OVSG. For instance, GloVe trained to be sensitive to nuances like sentiment, while Sentence-BERT is not. Therefore, GloVe for abstract relationships to better distinguish relationships such as "like" and "dislike". Con- versely, while GloVe does have a predefined vocabulary list, Sentence-BERT does not. Hence, for encoding the names of agents and regions, we prefer Sentence- BERT. Moreover, OVSG is designed with a modularized structure, allowing future developers to easily introduce new types and feature encoders into OVSG.
3.4 Sub-graph Matching
Subsequent the phases of input processing and feature encoding, two OVSG representations are constructed: one for the scene and another for the query, denoted by Ge and G, respectively. The problem of grounding L,, within the scene S can be converted now effectively translates to locating Gg within G.. Generally, the subgraph-matchie assumptions to simplify this problem. In this study, we assume that our Gg is a star graph, signifying that a central node exists and all other nodes are exclusively linked to this central node. (If G2 is not a star-graph, we will extract a sub-star- star-graph from it, and use this sub- -bb-rraph as our query graph.)
problem is NP-hard, prompting us to make several
The pipeline of sub-graph matching is illustrated the right side of Figure 1. This two-step procedure: candidate proposal and re-ranking. Let's denote the center of Gy as ug- Initially, we traverse all nodes, vi, in V., ranking them based on their distance to ey. computed with Eq. 1. Subsequently, we extract the local subgraph, G surrounding each candidate, vi. These extracted subgraphs serve our candidate subgraphs. In the second phase, we re-rank these candidates ing graph- sruph-simillrrtty metric, gG G To evaluate graph similarity, examine three distinct methodologies: Likelihood, Jaccard coefficient, and Szymkiewicz-Simppan index.
Likelihood Assuming the features of nodes and edges originate from a normal distribu- tion, we can define the likelihood of nodes and edges being identical as follows: L(ve, w²)
ff for edges. Here de and oe exp( -iis(("" for nodes and L(c's, e" ezp( de 6g balancing parameters. From this, can derive the graph- likelihood T as:
5

---

TI TG Go)) = L 2gy)) x
II argmax [L(e 00) L(eg e kEIV, jevv
where f.: is the center node of G The insight behind this formula is iterate over all possible node-level associations and select the one that maximizes the overall likelihood that G2 matches with Gz Noticeably, we use and de to balance the node-wise and edge-wise likelihood. In practice, we use 0, = 1.0 and de 2.0 to make the matching more sensitive to node-level semantics.
Jaccard-coefficient & Stymkiewicz-Simpss
index In addition to the likelihood index, we also consider other widely used graph similarity indices such the Jaccard and Seymkiewicz-Simpson indices. Both indices measure the similarity between two sets.
We adopt a similar method as in [7], generating set S(G) for each graph G by combining nodes and edges, such that |S(G) = |V| + E). The Jaccard coefficient Tу and Sonnkiewwizz-Simsso index Ty are then defined as follows:
S(G)))
Tr(G, G.)
(s(G, G:)
min(IS((,)))) (SG))
Given that we already know SS(G)) and [ we simply need to compute [S(G,) n S()))) which consists of nodes or edges that belong to both Gg and G We can define this union by applying distance thresholds Tu and Te for node and edge separately:
S(G)) 1(0 w7th (dist(s (7(K)) ev (le ( e(()) Eec
(4) Here, и is data association between Gg and G where x (k) = argmin (R) (dist( $x(K))) eu and c are threshold parameters. The differences between TL, TJ, and Ty can be understood as follows: T7 describes the maximum likelihood among all possible matches between Gg and G Both Tソ and 7s thresholds c.. t, to convert the node and edge matches to binary, and they measure the overall match rate with different normalization.
4 System Evaluation
Our OVSG framework experiments addressed these research questions: 1) How does our context- aware grounding method compare to prevailing approaches, including the SOTA semantic method and the recent work in the landscape of 3D semantic/spatial mapping, ConceptFusion [32] 2) How well does OVSG handle open-vocabulary queries? 3) What differences do our graph similarity- based methods show? 4) How well does OVSG perform inside real robot environment?
These questions are imperative as they not only test the robustness of the OVSG framework but also its comparative efficacy against notable methods like ConceptFusion in the ability to handle the intricacies of context-aware open-vocabulary queries.
4.1 Queries, Dataset, Metrics & Baselines
Queries We have two categories of queries for evaluation:
Object-only Queries These queries are devoid of any specific agent or region preference.
They are less generic and assess the system's 's grounding ability based purely on objects. An example might be: "Can you identify a monitor with keyboard positioned behind it?"
Whole Queries These queries inherently contain mix of agent, region, and object prefer- ences. For instance, these queries may include agents and other different entity types. An example would be: "Loocate the shower jet that Nami loves, with mirror to its right."

---

ScanNet We employed ScanNet's validation set (312 scenes) for evaluation. Since ScanNet only includes objects, we emulated agents, induced their abstract relationships to objects, captured spatial relationships between objects, and extracted object features via OVIR-3D before integrating the dataset into our evaluation pipeline. Resource limitations prevented manual labeling of scenes; hence, we synthetically generated 62000 queries (approx.) for evaluation (details Appendix E.1).
DOVE-G We created DOVE-G to support open-vocabulary queries within scenes using natural language. Each scene includes manually labeled ground truth and 50 original natural language queries (Tag). Using L.Mss, we expanded this by generating four extra sets of queries, totaling 250 queries per scene and 4000 overall to test OVSG's capabilities with diverse language expressions.
utilized the ICL- NUIM dataset[8]. We have created 359 natural language queries for the 'Whole Query' category and 190 natural language queries for the 'Object-only Query'. It should be noted that our approach is not merely a superficial addition of another dataset; instead, we have adapted and generated natural language queries for each scene within ICL-NUIM, emulating our methodology with DOVE-G, To adapt it to our framework, we performed similar preprocessing steps as with DOVE-G, importantly manually labeled ground-trune -runh annotations and leveraging OVIR-3D for feature extraction. Using this dataset, we demonstrate the superiority of our proposed method over ConceptFusion, especially concerning complex natural language queries that hinge multiple relationships context.
ICL-NUIM To thoroughly compare method, notably with ConceptFusion,
Evaluation Metrics
For each query, we evaluated the system's performance using three distinct metrics:
LOUBB For each query, this measures the 3D bounding box IoU between the ground truth and the top-k candidates yielded by our system.
IoUnD For each query, this measures the loU between the point cloud indices of the ground truth instance and the predicted instance.
Grounding Success Rate For each scene, this measures the fraction of queries where the system's predictions accurately match the ground truth given that the overlap is significan(loOUn > 0.5 IoUnD 0.5). The overlap threshold be adjusted to alter the strictness of the success criteria.
We reported the Topl and Top3 Grounding Success Rates and average IoU scores for each scene, reflecting the performance of our system in the Top-k results returned for each query.
Avg. Topl per Query
Avg. Topa scores per Query
Quary Type
Queries
OVIR-DD
OVSGJJ
OVSG.S
OVSG-L
OVIR-3D
OVSGJ
OVSG-S
OVSGLL (Ous)
LoUpe
0.52
Object-only
18.683
ToUan
0.38
0.22
0.44
0.55
Grounding Success Rote
38.52
15.29
40.99
51.18
5295
41.25
5366
Groundnng Success Roleap
45.13
17.22
47.79
68.35
ToUnn
0.37
0.22
0.44
0.55
0.53
0.45
0.55
0157
Whole
20,175
IoUlu
Gronding Sucuess Ro
58.56
24.33
47.77
58.85
$6.28
47.54
59.87
61.6
Gronding Success Rolegu
63,88
Tabll 1: Performance GVSG ScanNai
Baselines We assessed five methods in our study. The SOTA open-vocabulary grounding method, OVIR-3D, is our primary baseline as it will not leverage any inter-notion relations, providing a com- parative measure for the effectiveness of contextual information integration in the other methods. Unlike OVIR-3D, ConceptFusion integrates spatial relationships implicitly. The other three meth- ods, namely OVSG-J, OVSG-S, and OVSG-L (for Jaccard coefficient, Szymkiewicz-Simpsst index, and Likelihood, respectively) implement Context-Awarc Entity Grounding using different sub-graph matching techniques, as detailed in Section 3.4.
7

---

Query Type
Quenies
Metne
Avg. Topl Sccces
Avg. Top3
per Query
OVER-3D
OVSG-J
OVSG-S
OVSG-L (Ours)
OVIR-3D
OVSGJ
OVSG-S
OVSG-L
LoUnn
0.14
0.39
0.49
0.36
0.56
Object-only
320
ToUlu
0.1
0.14
0.41
0.54
Gronding Success Rolenn
36.56
13.75
39.06
48.44
34.06
56.25
56.56
Grouding Sucaess Retegu
49.09
18.44
53.13
67.81
LoUBB
0.35
0.2
0.41
0.51
0.55
0.41
0.55
0.56
Whole
400
0.37
0.21
0.43
0.55
Grounding Success Rus
35.5
23.0
44.75
54.25
56,0
41.0
56.75
57,0
Grounding Sucvess Ra
41.5
25.25
50.25
65.75
Table 2: Performanc of DVSG on DDVF-G
Query Type
Queries
Avg. Top) Soves per Quay
Ang.
Top?
sCNO pei Query
ConceptFuaion rel
(VBR-D
(VVS-)
OY3U-5
(VSC-LL(0mn)
OVIKJD
GV3G-J
OVSG-
(VSG-L
InC
0:15
0.32
8.5
0.53
0.49
Oboesi-oely
IoUlo
0.13 (0.3)
0.06 (0 15)
0.37
0.19
0.41
11.56
Grouncing Sacoos Raaig
35.26
16:34
36355
51.
54.74
48.95
54.24
55.79
Cooanding Sacses Rott
(.37
95
77.64
11.15
68:15
loUpn
0.47
0,2
0,56
Whole
119
ToAlao
Coostinng RoMfen
Groundnng Sacoes Ruttop
61.84
78,56
Table 3. Performance of OVSS & Cuncept on ICL-NUIM
4.2 Performance
ScanNet Table I averages results across 312 ScanNet scenes. Contextual data greatly improved en- tity grounding, with graph similarity variants (OVSG-S, OVSG-L) surpassing OVIR-3D, especially in scenes with repetitive entities like bookstores. More details are in Appendix E.
DOVE-G Table 2 averages performance over DOVE-G scenes for five query sets. OVSG-L consis- tently led, further detailed in Appendix F.3. While OVSG-J and OVSG-S were competitive in some scenes, OVSG-L was generally superior. OVIR-3D shined in the Top3 category, especially since DOVE-G scenes had fewer repetitive entities. Additional insights in Appendix
ICL-NUIM Table 3 shows ICL-NUIM results with OVSG-L outperforming other methods, espe- cially in the 'Whole Query' segment, contrasting with ScanNet and DOVE-G performances. Con- ceptFusion's 's performance was inconsistent across ICL-NUIM scenes (see Appendix G.3), with no- table success in one scene (highlighted in orange in Table 3). Simplified queries improved Con- ceptFusion's 's results, as depicted in the 'ConceptFusion (w/o rel)' column. Due to its point-level fusion approach, we evaluated different point thresholds and found optimal results at the Top 1500 points. Metries like IoUBB BB are not applicable for ConceptFusion. Further details on ICL-NUIM are in Appendix G. Despite ConceptFusion's strategy to avoid motion-blurred ScanNet scenes [32], its efficacy was still suboptimal in certain clear scenes.
Apart from these results, we also provide vocabulary analysis on OVSG as well as two robot exper- iments. Due to space limits, we put them to Appendices and D.
Conclusion & Limitation
Although we have demonstrated the effectiveness of the proposed OVSG in a set of experiments, there still remains three major limitations for our current implementation. First, OVSG heavily relies on an open- vocabulary fusion system like OVIR-3D, 3D, which may lead to missed queries if the system fails to identify an instance. Second, the current language processing system's strong dependence on LLMs exposes it to inaccuracies, any failure in parsing the query language may yield incorrect output. Third, as discussed in Section 3.4, calculating graph likelihood by multiplying nodes and edges likelihoods may not be optimal, as likelihoods from distinct types might carry varying levels of importance and distribution. Accurately balaneing these factors remains a challenge for future research, as our efforts with a GNN have not yielded satisfactory results.
Despite the aforementioned areas for improvement, we observe that OVSG significantly improves context-aware entity grounding compared to existing open-vocabulary semantic methods. Since OVSG only requires natural language as the query input, we believe it holds great potential for seamless integration into numerous existing robotics systems.
8

---

Acknowledgments
This work supported by NSF awards 1846043 and 2132972.
References
[1] A. Dai, A. X. Chang, M. Savva, M. Halber, Funkhouser, and M. Nießner. Scannet: Richly- annotated reconstructions of indoor scenes. Proc. Computer Vision and Pattern Recog- nition (CVPR), IEEE, 2017.
[2] K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[3] M. Fisher, M. Savva, and P. Hanrahan. Characterizing structural relationships in scenes using graph kernels. The first paper that uses 3d acene-graph.
[4] I. Armeni, Z.-Y. He, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese. 3d scene graph: structure for unified semantics, 3d space, and camera. 10 2019. URL http:// arxiv. texiv.org/a www./ogg//////oo/////o/oo 1910.0257........
[5] U. H. Kim, J. M. Park, J. Song, and J. H. Kim. 3-d scene graph: A sparse and semantic rep- resentation of physical environments for intelligent agents. IEEE Transactions on Cybernetics, 50:4921-4933, 12 2020. ISSN 21682275. doi: 1et11..110//CCYB.19992223
[6] A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone. 3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans. 2 2020. URL http://aare org/ abs/ 2002. 06289.
[7] Wald, Dhamo, N. Navab, and Tombari. Learning 3d semantic scene graphs from 3d indoor reconstructions. 4 2020. URL http: //axxiv. org/abs/2004 03967.
[8] A. Handa, T. Whelan, J. McDonald, and A. J. Davison. A benchmark for rgb-d visual odom- etry, 3d reconstruction and slam. In 2014 IEEE international conference on Robotics and automation (ICRA), pages 1524-1531. IEEE, 2014.
[9] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, al. Learning transferable visual models from natural language supervi- sion. In International conference machine learning, pages 8748-8766 PMLR, 2021.
[10] C. Jia, Yang, Y. Xia, Y.-T. Chen, Parekh, Pham, Q Le, Y.-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-langee language representation learning with noisy text supervision. In International Conference Machine Learning, pages 4916 PMLR, 2021.
[11] X. Zhai, X. Wang, B. Mustafa, A. Steiner, D. Keysers, A. Kolesnikov, and L. Beyer. Lit: Zero- shot transfer with ticked-image text tuning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123-18133, 2022.
[12] B. Li, K. Q Weinberger, S. Belongie, V. Koltun, and R. Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.03540 2022.
[13] G. Ghiasi, Gu, Y. Cui, and T.-Y. Lin. Scaling open- open-vocabulary image segmentation with image-level labels. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 540-557 557. Springer, 2022.
[14] J. Xu, S. De Mello, S. Liu, W. Byeon, T. Breuel, J. Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. arXiv preprint arXiv:2202. 2022.
[15] X. Gu. T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and lan- guage knowledge distillation. In International Conference Learning Representations.

---

[16] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty- tonnt-thoouaad classes using image-level supervision. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part IX. pages 350-368. Springer, 2022.
[17] M. Minderer, A. Gritsenko, A. Stone, M. Neumann, D. Weissenborn, A. Dosovitskiy, A. Ma- hendran, A. Arnab, M. Dehghani, Z. Shen, et al. Simple open- vocabulayy object detection with vision transformers. arxiv 2022. arXiv preprint arXiv:2205.062.0 062300.
[18) L.. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.- N. Hwang, et al. Grounded language-image pre-training. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10965-10975, 2022.
[19] S. Lu, H. Chang, E. P. Jing, A. Boularias, and K. Bekris. Ovir-3d: Open-vocabulary 3d instance retrieval without training on 3d data. In 7th Annual Conference on Robot Learning, 2023.
[20] Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprehension unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1 11-20, 2016.
[21] V. K. Nagaraja, V. I. Morariu, and L. S. Davis. Modeling context between objects for refer- ring expression understanding. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part IV 14, pages 792-807. Springer, 2016.
[22] L. Yu, P. Poirson, S. Yang, A. C. Berg, and T. L. Berg. Modeling context in referring ex- pressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October I 11-4, 2016, Proceedings, Part II 14, pages 69-85. Springer, 2016.
[23] W. Kuo, F. Bertsch, W. Li, A. Piergiovanni, M. Saffar, and A. Angelova. Findit: Generalized localization with natural language queries. 3 2022. URL http://arxiin www.wiiw..............eeeeeeeeee 2200. 17273.
[24] S. Gadre, M. Wortsman, G. Ilharco, L. Schmidt, and S. Song. Cows on pasture: Base- lines and benchmarks for language-driver zero-shot object navigation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23171-23181, 2023.
[25] B. Chen, F. Xia, B. Ichter, K. Rao, K. Gopalakrishnan, M. S. Ryoo, Stone, and D. Kappler. Open-vore vrcebvocayulary queryable scene representations for real world planning. In 2023 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 11509-11522 IEEE, 2023.
[26] K. Jatavallabhula, A. Kuwajerwala, Q Gu, Omama, Chen, Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, J. Tenenbaum, C. de Melo, M. Krishna, L. Paull, Shkurti, and Tor- ralba. Conceptfusion: Open-set multimodal 3d mapping. arXiv, 2023.
[27] N. M. M. Shafiullah, C. Paxton, L. Pinto, S. Chintala, and A. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. arXiv preprint arXiv:2210.0566: 2022.
[28] H. Chang, D. M. Ramesh, S. Geng, Y. Gan, and A. Boularias. Mono-star: Mono-camera scene-level tracking and reconstruction. arXiv preprint arXiv:2301.1324. 2023.
[29] H. Chang and A. Boularias. Scene-level tracking and reconstruction without object priors. In 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3795-3792. IEEE, 2022.
[30] N. Reimers and Gurevych. Sentence-hert: Sentence embeddings using siamese bert- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. 11 2019, URL https:/ //appsi///rxxvvv org/ abs/1908 10084.
10

---

[31] J. Pennington, R. Socher, and C. D. Manning. Glove: Global vectors for word representation.
In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532-1543, 2014.
[32] K. M. Jatavallabhula, A. Kuwajerwala, Q. Gu, M. Omama, T. Chen, S. Li, G. Iyer, S. Saryazdi, N. Keetha, A. Tewari, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint arXiv:2302.07241, 2023.
[33] C. Campos, Elvira, J. J. G. Rodríguez, M. Montiel, and J. D. Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874 187--1890, 2021.

---

A Prompt Engineering for Query Parse
As Chain of Thoughts (COT) has demonstrated, by providing with series of detailed examples, can guide large language model to generate our desired output while maintaining some format requirement. The design of these examples also known prompt engineering.
A.1 Prompt Example Illustration
Consider this natural language query as an example: "Could you point out Zoro's go-to cup, which we usually keep to the right of our espresso machine, on the left of the trash can, and in front of the coffee kettle?"
In this query, the user is asking about the location of cup, which has three different spatial rela- tionships with other reference entities and one abstract relationship with user named Zoro.
The desired output we provided is shown as below:
There are three notions here: zoro, cup, espresso machine, trash can, coffee kettle. can only use the relation provided.
The query target is cup. The relationship between zoro and cup is like. This relationship is abstract relationship.
The relationship between cup and espresso machine is right to. This relationship is spatial relationship.
The relationship between cup and trash can is left to. This relationship is spatial relationship.
The relationship between coffee kettle and cup is behind. This relationship is a spatial relationship.
The notion, target, and relationship are:
target @ cup {object}
zoro (user) like [abstract] cup {object)
cup {object) right to [spatial] espresso machine {object]
cup (object) left [spatial] trash can (object) coffee kettle (object} behind [spatial] - cup {object)
This example starts from some reasoning process in natural language, and ends with structured output which can be parsed by code. A breakdown of the structure is as follows:
target @ cup {object]: This line specifies the target object, which is a cup.
zoro {user] like [abstract] cup {object]: This line represents a relationship between user named Zoro (user) and the cup (an object) that Zoro likes the cup (Zoro's favorite). In our current implementation, like is relation of type abstract.
cup {object] - right to Ispatial] relationship between the cup (an object) and the espresso machine (an object). The cup is positioned the right of the espresso machine.
- espresso machine {object]: This line represents a spatial
cup {object} left to [spatial] trash can {object]: This line represents a spatial relationship between the cup (an object) and the trash can (an object). The cup positioned to the left of the trash can.
coffee kettle {object] behind [spatial] cup {object]: This line describes spatial relationship between the coffee kettle (an object) and the cup (an object). The coffee kettle is positioned behind the cup.
12

---

A.2 More prompt examples
Before asking LLM process the real user input, we will first input around 10 examples as prompt to control the output format. We select few examples to show here.
Question: want to get the cracker box around the table the kitchen.
There are three notions here: cracker box, table, and kitchen. I can only the relation provided.
The query target is the cracker box.
This is query for an object of the known category: cracker box.
The relationship between the cracker box and the table is 'near'. This relationship is spatial relationship.
The relationship between the table and the kitchen 'in'. This relationship is a spatial relationship.
The notion, target, and relationship are:
target @ cracker box object cracker box object- near [spatial] table object
table object in [spatial] kitchen region
Question:
Bring Tom his favorite drink.
There are two notions here: Tom and drink. I can only use the relation provided. This is query for object of known category: drink. The relationship between me and drink 'like". This relationship is spatial relationship. The query target 'drink'.
The notion, target, and relationship are:
target @ drink object Tom user like [spatial] drink object
Can you find Marry's ' favourite coffee cup? It might be the kitchen. There are three notions here: Mary, coffee cup, and kitchen.
Question:
This is query for object of known category: coffee cup.
The relationship between Mary and coffee cup is like. This relationship user relation- ship.
The relationship between coffee cup and kitchen in. This relationship is a spatial relation-
ship.
The query target is coffee cup.
The notion, target, and relationship are:
target @ coffee cup object
Mary user like [user] coffee cup object coffee cup object in [spatial] kitchen region
13

---

Ppose (A, B)
Six ("front")
fext ("leftt)
fpose (C,
Figure 3: The illustration highlights non lineurn mature of the spatia language feature. Let's assume both spatial pose feature and spotial text feature can represented within singular lincar space. For instance, consider A being to the left and in front of B, while is to the left but behind The pose festure for A relative B should align closely with features "left" and "from"". Com-enely, aall
festure for relative should close for foose B).
feature "left" distant
muppee
space, pose feature f fpose (A, would paralosically
B Spatial Relationship Prediction Pipeline
Likelibond; (N,
#Clayar: 1104.11
PClyyr, (7024 1924)
FCleyer: [1024, 1024)
Sputial feature; (N, 512)
Spatial feature;
RCAaye: (0024512]]
PC-layer [1024,1024]
IC-layer, (022,1024)
CLIP text encoder
PC PC-ayen (1024,
Folayer (18.1024)
S
Relationship
Figure Architecture spatial language predictior.
sputial pose encoder, the yellow sputial relationsip prodictor.
The Spatial Relationship Predictor module aims to estimate the likelihood between pose pairs and alignment
language descriptions. Given that there is no standard solution to this challenge, we have developed our own encoder-predictor structure.
Network Structure The input for the spatial pose encoder (depicted as blue block in Figure is pose pair defined by (N. 18). An entity's pose in the OVSG characterized by the boundaries and center of its bounding box, specifically (Imins Imin: Amins Tmaz, Imazz Amaz Tcenters Veenter, Zcenter). We employ five-layer MLP to encode this pose pair into spatial pose feature. For the encoding of the spatial relationship description, we utilize the CLIP-text encoder, converting it into a 512-dimensional vector.
Distance Design These encoders serve as the foundation for constructing the OVSG. When per- forming sub-graph matching, the predictor head estimates the distance between the spatial pose feature and the spatial text feature. We do not use cosine distance because the spatial relationship is highly non-linear. Figure 3 illustrates why cosine distance is not sufficiently discriminative for spatial-language alignment.
Training process We train this encoder and predictor module using supervised learning. The train- ing data is generated synthetically. We manually defined 8 different single spatial relationships, i.c. left, right, in front of, behind, in, on, above, under. From these 8 basic spatial relationships, can generated more than 20 different meaningful combinations, e.g. "on the right side", "at the left front part". Each combinations can also have more than one descriptions. Finally, we collected 90 descriptions in total. The training loss we used is a binary cross entropy loss.
14

---

C Robot application
Manipulation In order to exemplify the utility of OVSG in real-world manipulation scenarios, devised a complex pick-and-place experiment. In this task, the robot is instructed to select one building block and position it on another. The complexity of the task stems from the multitude of blocks that are identical in both shape and color, necessitating the use of spatial context for differentiation. Each task consists of a picking action and a placing action. distinct tasks for this purpose (please refer to Appendix C.I for detailed setup). The effectiveness of the manipulation task was evaluated by comparing the success rate achieved by OVIR-3D and our newly proposed OVSG-L. The outcome of this comparative study is depicted in the accompanying table. The results demonstrate that our innovative OVSG-L model significantly enhances the object grounding accuracy in manipulation tasks involving a high prevalence of identical objects. This improvement highlights the potential of OVSG-L in complex manipulation scenarios, paving the way for further exploration in the field of robotics.
formulated nine
object
shoe
bottle
chair
trash candt
trash cant2
drawer
cloth
Method
sconel
scone2
Scone3
success rate (%))
100.0
100.0
100.0
100.0
100.0
100.0
OVIR-3D (98
OVSG (9)
0.88
0.75
0.75
0.33
Table
object navegation task
Table 3: Success rule of manipulatio task
Navigation We conducted a system test ROSMASTER R2 Ackermann Steering Robot for an object navigation task. The detailed setup can be found in Appendix C.2. We provided queries for seven different objects within a lab scene, using three slightly different languages to specify each object. These queries were then inputted into OVSG, and the grounded positions the entities were returned the robot. We considered the task successful the robot's final position was within meter of the queried objects. The results are presented in Table 4. From the table, is evident that the proposed method successfully located the majority of user queries. However, there one query that was not successfully located: "The cloth chair in the office." In this case, we found that OVIR-3D incorrectly recognized the cloth as part of a chair, resulting in the failure to locate it.
C.1 Manipulation Experiment Setup
Robot Setup All evaluations were conducted using a Kuka IIWA robot arm equipped with a Robotiq 3-finger adaptive gripper. The arm was augmented with an Intel Realsense D435 camera, which was utilized capture the depth and color information of the scene in RGB-D format, offering resolution of 1280 720. The gripper operated in "Pinch Mode," whereby the two fingers on the same side of the gripper bent inward.
To initiate the process, the robot arm was employed to position the camera above the table, orienting it in a downward direction. Subsequently, the RGB-D data, along with a query specifying the object be picked and a target object for placement, were inputted into the OVSG system. Upon acquiring the bounding box of the query object, the robot gripper coordinates of the target box by utilizing the ROS interface of the robot arm.
directed to move towards the center
Block building task To evaluate the application of the proposed method in real-world manipulation tasks, we designed block- boolkk-b task. The task is to pick one building block from a set of building blocks and place it on another building block. The picking block and placing block are separately specified by different natural language query. The difficulty of this task is that each building block has many repeats around it have spatial context specify the building block. And need to succeed twice in row to complete task.
15

---

Object
Query Id
Query
shoe
shoe that is in front of the monitor and in the office.
2
A shoe positioned both before the monitor and within the office. shoe rests in front of the monitor. The shoe is inside the office. The bottle that is right to Tom's keyboard
3
bottle
4
The bottle to the right of Tom's 's keyboard.
6
The bottle positioned to the right of keyboard which belongs to Tom. The chair that is behind the TV.
chair#1
7
8
The chair is situated behind the TV The chair with a TV front of The chair that is in front of the car.
9
chair#2
10
11
The chair rests before a car.
12
The chair with car behind it. The trash can that is behind the refrigerator. The trash which
trash can#1
13
14
be found behind the refrigerator.
15
The trash can that is situated at the rear of the refrigerator. The trash that is in the lab and under table.
trash can#2
16
17
The trash can is located in the lab and beneath table.
18
The trash can situated within the lab, positioned under table.
drawer
19
The drawer that is behind a box in the office.
20
The drawer is positioned behind box in the office.
21
The drawer is situated at the rear of box within the office.
cloth
22
The cloth that is on a chair
23
The cloth is resting
chair
The cloth positioned atop a chair.
Table & for navigation
Figure one our suvigation task, ROSMASTER Ackermann Steering Robot. right our manipulation tack, IWA
C.2 Navigation Experiment Setup
Robot Setup All evaluations were conducted using ROSMASTER R2 Ackermann Steering Robot. For perception, utilized an Astra Pro Plus Depth Camera and YDLidar TG 2D lidar sensor, both mounted directly onto the robot. The robot is equipped with built-in Inertial Measurement Unit (IMU) and wheel encoder. The Astra camera provides video stream at resolution of 720p at 30 frames per second, and the lidar operates with a sampling frequency of 2000 Hz and a scanning radius approximately 30 meters. The overall configuration of the setup is depicted Figure 4.
Demonstrations and Execution Prior the evaluation process, employed Intel RealSense D455 camera and ORB-SLAME [33] generate comprehensive map of the environment. This generated both the RGB-D and pose data, which could be subsequently fed into the Open-
16

---

pipeline. For the demonstration of locating with the vren-Vocabulary 3D Scene Graph (OVSG), we developed a 3D to 2D conversion tool. This tool takes the point cloud from the comprehensive 3D map and converts it into a 2D map by selecting a layer of points at the height of the lidar. The resultant 2D map could then be utilized by the ROSMASTER R2 Ackermann Steering robot for navigation. To achieve goal-oriented navigation, we incorporated the Robot Operating System (ROS) Navigation stack and integrated it with the Timed Elastic Band (TEB) planner. The initial step involved establishing pose within the environment. Subsequently, the Adaptive Monte Carlo Localization (AMCI)) leveraged lidar scan inputs and IMU data to provide a robust estimate of the robot's pose within the map. The move base node, key component of the ROS navigation stack, used the converted map and the item's 's position provided by the OVSG and conversion tool to for- mulate a comprehensive global plan targeting the goal position. Concurrently, the TEB local planner consolidated information about ROSMASTER R2's kinematics and lidar input to generate short- term trajectory. The result was a locally optimized, time-efficient plan that adhered to the robot's pre-set velocity and acceleration limits. The plan also included obstacle avoidance capabilities, enabling the robot to identify and circumvent barriers detected by the lidar system.
Object navigation task To evaluate the application of OVSG in real-world navigation problems, a language-based object navigation task is proposed. We selected seven different objects inside a laboratory. Each object is paired with three different queries. All queries for three objects are listed in Table 6.
Falleyy
Faltiy
11.14
2111
1114
11.6
11
11.01
19:55
B-14
12.1
2180
1134
6335
2011
$8.00
6627
$2.9
1611
42.66
02:10
4025
16.99
72.8
46.12
Denei
9023
71.31
0721
11
E11
111
4.1
18
8.1
4.1
19
84h
881
841
036
Table 7: Perfontaacee comparisan against five different varied open-
Talee 8: Performance compatsson igninst five differest varie relunoouhhip sets
vocabulary se5s
D Open-Vocabulle Analysis
Having presented insights on our system's is performance on natural language queries for DOVE-G (as shown in Table 2), we proceed to deepen our investigation into the system's 's resilience across diverse query sets. To accomplish this, we instead average the results from all scenes for each of the five vocabulary sets (refer to Table 7). By doing so, we aim to provide a robust evaluation of our system's 's performance across a variety of query structures and word choices, simulating the varied ways in which users may interact with our system. In addition to experimenting with object vocabulary variations (a 'coffee maker' to 'espresso machine' or 'coffee brewer'), and altering the order of entity referencing in the query, we also studied the impact of changing relationship vocabulary. In this experimental setup, the LLM is not bound to map relationships to pre-determined set as before. Instead, the graph-based query contains a variety of relationship vocabulary. To illustrate, consider the queries "A is to the left back corner of B" and "A is behind and left to B". Previously, these relationships would map to a fixed relation like 'left and behind'. Now, 'front and left' as interpreted by the LLM can variate to 'leftward and ahead', 'northwest direction', or 'towards the front and left', offering broader range of relationship descriptions. The evaluation results for these query sets are presented in Table 8.
Varying object names Across all evaluated vocabulary sets, OVSG-L demonstrates the highest Topl and Top3 Grounding Success Rates outperforming the remaining methods. This pattern also persists for scores in the IOUBB category. Notably, OVSG-L's Grounding Success Rates span from 44.86% to 57.43% for Topl, and 56.57% 65.43% for Top3, All in all, contextual under- standing of the target again proves improve results from 35.83% (OVIR-3D) to 50% (OVSG-L) for Topl Grounding Success Rate and 0.32 to 0.44 for the Topl loUBB.
17

---

Varying relationships As shown in Table 8, we observe noticeable decrease in performance for the methods under the OVSG framework (compared to Table 7). This likely due to the increased complexity introduced by the varied word choices for edges (relationships) in the sub-graph being matched. Despite this, two of the OVSG methods still outperform the OVIR-3D method, with the OVSG-L method delivering the strongest results.
E
More on ScanNet
E.1
Synthetic Query Generation for ScanNet
In the ScanNet dataset, each scene comes with ground-truth labels for its segmented instances or objects. We began by calculating the spatial relationships between these ground-truth objects or entities. Subsequently, agents were instantiated into the scene, and abstract relationships were ran- domly established between the agents and the entities present in the scene. After generating the OVSG for each scene, our next step involved the creation of graph-based queries (refer to syntax and details in Appendix A) for evaluation purposes. For each of these queries, we randomly selected reference entities from the OVSG that shared a relationship with the target entity. This formed the basis of the synthetic generation of the graph-based queries for the ScanNet dataset.
E.2 Grounding Success Ratenв
OVSG's Grounding Success Ratepg on ScanNet Scenes
On 1/2 zovees with otject guenter
Oe acuser with okjeciantly querte
DMM--3
oV86.J
NUH Scetts
DVVS-- (Ours)
Kure of Scones
CV5G- (Oun)
100
16
x= 75
x:c
26
80
x6
Topt Grounding Sussess Rane X%
Topa Grounding Sucoess Rate vt
with
428
OVIR-3D
3D
-V88--8
0V8G-8
308
OVVC.LL (Ours)
300
CVBC- (Dun)
of Soames
308
Num
Top1
Figure Perfommince wir Grounding Success Ralega ScanNer Scenes
In this section, provide the number of ScanNet scenes that correspond to various success rate thresholds (at 15%, 25%, 50%, and 75%). We provide four-fold results containing Topl and Top3 scores for 'Object-only' and 'Whole Query' categories (as shown in Figure 5).
E.3 Grounding Success Ratesp
In this section, provide the various success rates for different IoUaD thresholds (at 0.15, 0.25, 0.5, and 0.75). We provide two-fold results containing scores for 'Object-only' and 'Whole Query' categories (as shown in Figure 6).
18

---

OVSG's Grounding Success Ralegp ScanNet Queries
On 037 sijec-onyy querko
On 28 wmslt quetic
100
0MR--0
100
0VIR-D
Grouniing Stotceso Rate (hi
(VVG-J
Grounding (N)
DVSGJJ
OVVS-- (Ouns)
DVVG-L [Oung]
0-15
2.0000
0.75
0.15
1- 0.25
5
30 IoU
3D lnd
Higure 6: Performance OVSG Grounding Success Kalegrr>
ScraNNtt Queries
F
More DOVE-G
F.
Grounding Success Rate
OVSG's Grounding Success Rategs on DOVE-G Scenes
On scaoee with skjeci-olly quarles
De scaces with otjech earioe
OMR 3D
(VVB--J
oVBC-J
L0V804L (0ur))
0V864 (Ouns)
Num
Top1 Grounding Success Rate
Top8 Grounding Suceess Rate
DVSGJ
-vvs--5
ovaG-s
Top1 Orounding Bucceus
Tops Grounding Success Rate
Figure Performunce OVSG wxl Grounding Success Rategg um DOVE-G Sernes
In this section, provide the number of DOVE-G scenes that correspond to various success rate thresholds (at 15% 25%, 50%, and 75%). We provide four-fold results containing Top1 and Top3 scores for 'Object-only' and 'Whole Query' categories (as shown in Figure 7).
F.2 Grounding Success RateaD
In this section, provide the various success for different IoUaD thresholds (at 0.15, 0.25, 0.5, and 0.75). We provide two-fold results containing scores for 'Object-only' and 'Whole Query' categories (as shown in Figure 8).
19

---

OVSG's Grounding Success Ralegp DOVE-G Querles
Oo 32V ebjecionlly quorits
On 400 whole qwonita
100
0MR--0
100
0VIR-D
CVSG-J
DVSGJJ
Grouniing Stotceso Rate
OVSG- (Ouns)
0VVG-- [Oung]
Grounding
1-15
2.50
0.75
0.15
1- 0.25
0.75
30 IoU x-
3D lod
Pigure Performance of UVSG Grounding Suceess Ralegry DOVE-G Queries
F,3
Performance of the OVSG Framework Various Scenes in DOVE-G
Topl Grounding Success Ratepp
Top3 Grounding Success Ratepn
Scene
OVIR-3D
OVSG-J
OVSG-S
OVSG-L (Ours)
OVIR-3D
OVSG-J
OVSG-S
OVSG-L (Ours)
Room #1
44.4
36.8
61.2
79.6
88.4
72.8
98.4
99.6
Kitchenette
31.0
24.0
49.0
65.0
51.0
71.0
71.0
Bathroom
44.8
23.2
57.2
58.0
59.6
41.2
66.0
69.6
Kitchen
44.8
36.0
51.2
55.6
53.6
50.8
$5.2
56.8
Room
26.0
62.0
65.2
Computer Lab
38.0
30.8
38.0
41.6
54
41.2
50.4
49.2
Room #3
32.8
16.8
40.4
40.4
47.2
34.4
47.2
47.2
Hallway
8.4
40.0
40.0
40.0
Overall
35.2
25.2
45.4
51.1
63.6
47.1
64.8
65.7
Top1 loUps
Top3 IoUpp
OVIR-3D
OVSG-J
OVSG-S
OVSG-L (Ours)
OVIR-3D
OVSG-J
OVSG-S
OVSG-L (Ours)
0.55
0.60
Kitchenette
0.28
0.23
0.34
0.42
0.59
0.47
0.62
0.62
Bathroom
0.30
0.13
0.36
0.36
0.41
0.30
0.44
0.46
Kitchen
0.35
0.28
0.38
0.42
0.47
0.42
0.46
0.48
Room #2
0.15
0.21
0.31
0.34
0.51
0.41
0,53
0.52
Computer Lab
0.32
0.22
0.33
0.39
0.49
0.39
0.45
0.46
Room #3
0.34
0.13
0.37
0.39
0.50
0.36
0.49
0.50
Hallway
0.27
0.10
0.24
0.31
0.37
0.34
0.38
0.41
Overall
0.28
0.18
0.34
0.39
0.49
0.39
0.49
0.50
Table 9: Performance of the OVSG framewek on natural language sceme quertes in DOVE-G
In Table 9, we present the performance of OVSG framework natural language queries in DOVE-G.
E4
50 Sample Natural Language Queries for Scenes in DOVE-G
In Table 10, we provide list of 50 sample queries for scenes in DOVE-G.
F.5 More on Scenes in DOVE-G
In Figure 9 and Figure 10, we display eight different scenes included in our DOVE-G dataset.
20

---

Query No.
Natural Language Query
Locate the vanity sink, which is positioned to the right side of a door latch. Identify the hand basin that has face cleanser situated in front of it. Is there hand basin that has both facial scrub and hand soap placed in front of it? I'm looking for a wash-han basin with a facial scrub directly in front of it and a door latch to its right.
2
3
4
5
Locate the shower jet that Nami loves, with a mirrored surface to its right and hair cleanser in front of it.
6
Can you find the shower sprayer with a face cleanser positioned behind it? Search for the shower sprayer that has a facial scrub behind it and a vanity mirror to its right. Identify the travel suitcase located to the right of backpack and ahead of a water bottle. Look for travel suitcase Zoro dislikes, it should be to the right of book and water bottle.
7
8
9
10
Can you find book positioned the left of a backpack?
11
Nami's preferred book should be positioned front of chair, can you find it?
12
Can you identify Zoro's ' liked book that's situated ahead of water bottle and another book? Locate Carry-on Luggage for me, please. It should have both water bottle and rucksack in front.
13
14
Is there a trolley bag with a water bottle and backpack up front, and also a desk chair in its rear? Find an ergonomic chair for me, but it has to have textbook situated to its left. Can you find the workbook that Luffy dislikes and is right of a desk chair? Is there headrest that has reference book positioned in front of it? Where's 's the cushion with a courschook and an ergonomic chair up front? I'm I'm searching for a backpack with a coursebook on its left.
15
16
17
18
19
20
Where's 's the table fan with desk chair bchind it?
21
Can you find table fan with computer chair and a reference book behind it?
22
Where's 's the reading lamp that's to the left a travel bag?
23
Can you spot the reading lamp that's 's to the left of a travel bag and behind a computer chair?
24
Is there reading lamp that's 'ss behind a reference book?
25
Can you find a pedestal fan with desk chair and a coursebook behind it?
26
Locate the headrest that's disliked by Nami, but Luffy indifferent to.
27
How about a cushion that Nami dislikes, Luffy is neutral to, and Zoro takes liking to?
28
Where's 's the reading lamp that's 's to the left of a knapsack?
29
Can you spot the reading lamp that's 's to the left of knapsack and hehind ergonomic chair?
30
Is there desk lamp that's behind workbook?
31
Where's the reading lamp that Nami fond of?
32
Can you locate a backpack with a table fan to its left?
33
Where's 's the knapsack with a tower fan on its left and Luffy hehind?
34
Can you find travel bag that has water bottle its left?
35
I'm searching for backpack with a textbook on its left.
36
Where's 's the pedestal fan with a computer chair behind it?
37
Where is the cup that's nestled to the right of the coffee maker, to the left of the coffee kettle,
and in front of the poster?
38
Identify the toy that's to the right of the espresso machine and the left of the trash can. Where's 's the doll with a cup positioned behind it?
39
40
Can you show me the water bottle that Luffy loves and has a coffee cup behind it? Locate the water bottle that's 's to the left of the checkerboard.
41
42
I want to know about the coffee cup that Zoro loves, which is also hehind the keyboard that
Nami is behind.
43
Can you identify the chair that the CPU machine is behind? Locate the chair that Nami likes and is also behind the CPU machine.
44
45
Can you identify the teacup that Luffy loves and is behind the CPU machine? Is there a computer chair that Zoro doesn't 't prefer? Identify a coursebook located ahead of water bottle.
46
47
48
Is there a workbook sandwiched between two water sipper bottles?
49
Nami's preferred coursebook should be positioned in front of a desk chair, can you find it?
50
Zoro's ' liked reading book, is it placed in front of water beverage bottle?
Table 10: Lis of 30 sample quenes for Scones DOVE
21

---

Room #1
Room #2
Room
Kilchen
Higure scenes
G More on ICL-NUIM
G.1 Grounding Success RateBв
In this section, we provide the number of ICL-NUIM scenes that correspond to various success rate thresholds (at 15%, 25%, 50%, and 75%). We provide four-fold results containing Topl and Top3 scores for 'Object-only' and 'Whole Query' categories (as shown Figure 11).
G.2 Grounding Success Rateap in comparison ConceptFusion
In this section, provide the various success rates for different IoUsD thresholds (at 0.15, 0.25, 0.5, and 0.75). We provide two-fold results containing scores for 'Object-only' and 'Whole Query' categories (as shown in Figure 12).
G.3 Scene by Scene Grounding Success Ratesp of OVSG & ConceptFusion on ICL-NUIM
Table 11 showcases the 3D Grounding Success Rate of various methods on different scenes in the ICL-NUIM dataset, highlighting the performance metries across different IoU thresholds.
G.4
Qualitative Performance Comparison between ConceptFusion and OVSG-L
In this section, providing qualitative results sample queries for the methods ConcepFusion and OVSG-L in Figure 13 and Figure 14 respectively.
22

---

Computer Lab
Hallway
Kitchenette
Bathroom
Pigure 10: Four the other scenes DOVE:
OVSG's Grounding Success Ratepe on ICL-NUIM Scenes
On scenes with onger--nyy quertte
De scaces with otjech-enty ouerter
DMR-33
(0080-1
0V86JJ
DVSG-- (Ours)
CVSG-4 (Oun)
Nure
X--16
X:
XF 69
75
x:
x6
Topt Grounding Sussess Rane X%
Topa Grounding Success Rate vtt
OVIR-3D Ouffascees
0V1R-0
-0V80-8
oV8G-6
OVVC.-L (Ours)
OVBC- (Duns)
of Stsmes
Mum
Fignee Performance Groanding Succoss Raligg Seanes
23

---

OVSG's Grounding Success Raleggo ICL-NUIM Queries
object only queritz
whole queritz
ConcepsFuiion
DVSG-J
ConceptFusion
Grouniing Suokese Rate
0V18-30
0V9G-L [Ours]
OVSG-S
Grouneing
6.500
3D xx
3D
Figure 12: Performance of OVSG Conceptaasion w.ct Grounding Suceess Katègry ICL-NUIM Queries
ICI-NUTM Scene
# Queries
Method
Grounding Success RaleyD
IoUso 0.15
IoUnp 0.25
IoUso 0.50
IoUau 0.75
ConceptFusion (wio rel)
89
88.89
0
ConcertFusion
5.56
OVIR-3D
83.34
83.34
50
22.23
living trajO frei pag
18
OVSG-J
5.56
5.56
5,56
OVSG-S
94.45
94.45
61.12
22.23
OVSG-L (Ours)
100
100
66.67
22.23
ConcepdFusion (wlo
61.77
50
41.18
ConceptFusion
26.48
26.48
14.71
living.room_rajj waannurajl_ffeiiie _frei_pag
34
OVIR-3D
OVSG-J
38.24
70.59
70.59
38.24
29.42
67.65
OVSG-L (Ours)
79.42
79.42
70.59
14.71
ConcepdFusiontw/o
46.43
14.29
ConceptFusion
3.58
0
0
OVTR-3D
50
50
50
28.58
living room traj2 pag
42.86
35.72
3,58
OVSG-8
82.15
82. 15
50
28.58
OVSG-L (Ours)
ConceptFusion(wwo tel)
0
0
0
ConcepuFuusion
Twing.room.rraj..fii...mg
17
OVIR-3D
OVSG-J
23.53
11.77
23.53
11.77
23.53
0
OVSG-S
41.18
23.53
11.77
11.77
OVSG-L (Ours)
82.36
64.71
52.95
29.42
ConcepiFusiiontvv rel)
0
ConceptFusion
0
0
0
oflice room traji) frei png
OVSG-J
44,83
44.83
41.38
OVSG-S
OVSG-L (Ours)
100
100)
96.56
ConcepdFusiontw/o
ConceptFusion
0
0
office.noom.tnil..ri...p.p
OVIR-3D
OVSG-J
42.11
68.43
68.43
42.11
68.43
42.11
21.06
31.58
OVSG-8
73.69
73.69
73.69
36.85
OVSG-L (Ours)
100
100
94.74
ConcepdFusion((vlc rel)
0
0
0
0
CuncepiFFusion
OVTR-3D
83.34
83.34
33.34
8.34
offie_roommmjj..fii..ngg.
12
OVSG-J
0
0
OVSG-S
83.34
34
33.34
OVSG-L (Ours)
100
100
41.67
16.67
ConcepiFusioatwv rel)
12
ConceptFPusion
0
0
0
oflice room Ing33 frei png
OVSG-J
48
48
28
OVSG-S
60
60
60
OVSG-L (Ours)
100
100)
80
Tible 11: Grounding Soccess Rilegrs of UVSG ConeeptPsson on ICL-NUIM

---

ConceptFusion's Best Pertorming Scene ICL-NUIM Dataset @ovng_room_teee _rapt_tre__ng
FAIL: "I need find sola that his the reading lamp lef?"
FAIL: "can you locare sots bordered on the right readinn lamp and wall paining, also aushooed
FAIL: "Pilow DQuery wio rel)
PARTIAL SUCCESS: "Can locate the chair with cobinet its left and deramio pot to is lett"*
PARTIAL SUCCESS; "Chai"" (Queyy wlo ni)
PARTIAL SUCCESS: "Sule" (Quary wio rel)
Figure ConceplFusion sample ICL-NUIM Queries
OVSG-L Performance on ConceptFusion's best performing scene in ICL-NUIM Dataset [wvingrroom_ htr__taal_______pp__
SUCCESS: need to find sola that has the reading lamp its
SUCCESS: "Can you locate a sola bordered the right by reading lamp and wall
painting, also cushioned pillow?
SUCCESS:
"Can you locate the chair with cabinet to Itss and ceramic pot to its let?"
SUCCESS: "Finn a pillow" [Topa]
SUCCESS: "Find pillow" [Top2]]
Figure Performance OVSG-1. sample ICI-NUM
25


Reviews:
[{"review_id": "vVz9cO5nxu", "reviewer": "Reviewer_2AQt", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces an Open-Vocabulary 3D Scene Graph (OVSG), a formal framework that enables context-aware entity localization in 3D scenes using free-form text queries. Unlike traditional semantic-based approaches, OVSG supports open-vocabulary querying, allowing more flexible and detailed queries. The proposed approach outperforms previous semantic-based localization techniques in comparative experiments using the ScanNet dataset and a self-collected dataset. The paper also demonstrates the practical application of OVSG in real-world robot navigation and manipulation experiments.", "summary_of_recommendation": "Overall, the paper was enjoyable to read. The proposed method is well described, and appears to outperform the OVIR baseline. However, I hope the authors could provide some additional baselines (i.e. a simple baseline using a combination of open-vocabulary object detector / VLM + LLM), to compare and potentially see the benefit to using a graphical approach, as opposed to using the implicit reasoning ability embedded within the LLM. For details, see review section.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "Overall, the paper was interesting to read. However, I have a few questions that I was hoping the authors could address:\n* How does the method compare with a basic implementation using an open-vocabulary object detector / VLM + LLM combined approach?\n* What is the reference frame for the position attribute? \n* How does the method handle situations where all matching values are low? \n* What is the runtime and compute requirements?\n* Have the authors tried capturing the representation in the GNN? Are there preliminary results on this / what is the bottleneck?\n* See review section for detailed descriptions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths\n* Uses semantic features instead of discrete labels\n* Paper is well written, notation is clear and defined, overall flow is great\n\nWeaknesses\n* Additional baselines:\n  * It would be interesting to see an implementation where an open vocabulary VLM is used to capture the scene and locations of objects, and a LLM is used to parse the query and select the object of interest. How would such an implementation compare with the proposed method?\n* Unclear points:\n  * What is the reference frame of the position? Is it with respect to some world frame?\n  * Regarding sub-graph matching, the node-level association with the highest likelihood will be selected. What happens in the scenario where all matching values are low?\n  * What is the runtime and compute requirements of the method? \n* Additional ablations:\n  * Using different encoding mechanisms: the authors use a combination of GloVe and Sentence-BERT. What about other encoding mechanisms?\n  * For sub-graph matching, the $\\sigma_v$ and $\\sigma_e$ are balancing parameters, what are they chosen to be? How do different values affect performance?", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "fLpZaQoQABU", "reviewer": "Reviewer_zmgd", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper aims to solve a problem of natural language grounding given a 3D scene graph. However, the grounding is challenging due to the variability of class category, relationship, and attribute that may be unforeseen during the construction of 3D scene graph. To figure it out, this paper proposes a context-aware entity grounding framework, an Open-Vocabulary 3D Scene Graph (OVSG), by leveraging the representation power of language models to represent unseen semantic categories and relationships. The main contributions of this paper are as follows:\n\n-\tA 3D scene graph-based grounding framework with an open-vocabulary query\n-\tA construction of dataset for the context-aware entity grounding\n-\tDemonstrations with navigation and manipulation scenarios", "summary_of_recommendation": "The proposed methodology is very interesting as an idea, but it is difficult to accept it without further experimental improvement due to the lack of a detailed description in the methodology and the lack of evaluation results (i.e., grounding success rates) supporting the mentioned contributions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "-\tWhy do Gs and Gq use different object feature encoders? Doesn’t it cause a bad distance estimation result?\n\n-\tIn Eq. (1), the spatial and abstraction distances require scaling to balance their effects. How do you determine? \n\n-\tThis paper uses different language model per feature. How do you select? Why don’t you use a specific model or all models per feature?\n\n-\tIn Eq. (2), what is this \\pi(k)? This is different from the explanation on L110. Further, this must be a deterministic function but equations use it as a random variable. Isn’t it mathematically incorrect?\n\n-\tL168: what is the central node and how do you select it given an instruction? \n\n-\tL169, L173: how do you extract a sub-star-graph?\n\n-\tL284-287: where is the result of the success rate?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Strengths – The main strength of this paper is the adoption of multiple language encoders to incorporate diverse linguistic expressions. \\\nWeaknesses – There are two major issues in this paper: clarity and effectiveness of the proposed algorithm\\ \n\nThis paper needs to clarify and improve a few things.\\\n\n-\tThe major concern is the lack of supporting evaluations. First of all, the evaluation results only show the graph matching performance and IoU which do not say the accuracy of the grounding result. Based on the provided tables, the proposed framework, OVGS-L, shows about 50-60 matching rates. However, this does not tell us the proposed framework could find correct targets. Although the matching rates are high, the grounding accuracies can be low. In that case, we cannot say the proposed method is meaningful. \n\n-\tFurthermore, another issue is the lack of baselines: this paper uses only one baseline method, OVIR-3D, that is not trained with relationships, so anyone can guess its lower performance than the proposed frameworks. For fair comparisons, this paper needs to train any methods with the relationship dataset using the dataset described in Appendix C. In addition, there must be other scene-graph-based grounding frameworks that can be used as baselines while handling relationships. We can find candidates via simple googling like [1].\n\n[1] Kim, Dohyun, Jinwoo Kim, Minwoo Cho, and Daehyung Park. \"Natural Language-Guided Semantic Navigation Using Scene Graph.\" In International Conference on Robot Intelligence Technology and Applications, pp. 148-156. Cham: Springer International Publishing, 2022.\n\n-\tAnother biggest problem is the unclear explanation to reproduce the proposed method. First of all, the proposed framework uses LLM to parse a given instruction. Although the proposed framework largely depends on the LLM’s parsing capability, this paper does not clearly explain how the authors extracted the required information such as object, agent, region, abstract and spatial relations. Appendix C is not clear. More detail must be needed to improve the clarity of this paper and its readers.\n\n-\tLastly, it would be better to provide deeper evaluation results and analysis about the spatial/semantic grounding accuracies instead of the open-vocabulary analysis since we know (large) language models have the capability of handling various similar expressions. \n\n-\tFigure 1 is really hard to understand. In particular, the encoding boxes are not interpretable.\n\n-\tL147-149 need to be improved for a better understanding", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}, {"review_id": "CoTCr-RQHZ", "reviewer": "Reviewer_UCiN", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The authors proposed a method of semantically localizing a query within a given 3D scene graph that is enhanced to include semantic embeddings (from various large language and vision language models). To do so they construct a small scene graph corresponding to the query and then search through the global scene graph using 3 proposed measures to quantify how well the small scene graph matches different subgraphs in the global scene graph. They demonstrate the system on synthetically generated queries over the ScanNet dataset, over manually and automatically generated queries in their own dataset, and for a robot manipulation task.", "summary_of_recommendation": "In summary, the authors have tackled a timely and novel problem, but I remain to be convinced that the challenges presented are very difficult to overcome or that the results have major significance based on the results they have presented.", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "I would be interested to hear the authors' perspectives on the \"Weaknesses\" listed above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I will start by summarizing what I see as the strengths and weaknesses of the paper and then follow up with some other general comments and inquiries. \n\nStrengths:\n\n - The paper presents a timely method that incoporates several state-of-the-art components such as LLMs, vision language models, and 3D scene graphs.\n\n - The paper has a strong systems contribution where many different models and econders are all incorporated into the method which is impressive from an engineering perspective and also useful from the standpoint of the wide variability of queries that can be treated.  \n\n - I believe this problem statement of semantic localization within a 3D scene graph is novel. \n\n\nWeaknesses:\n\n - I would argue that the paper lacks methodological depth. The crux of the method is Eq. (1) which defines a simple distance measure over the embedding space which is used to underpin the sub-graph matching methods. I have some doubts about the regularity of these embedding spaces with respect to how well calibrated these distance measures really are. I would be interested to hear the authors' perspective on this point.\n\n - Further, the authors assume that the global 3D scene graph is given and that the semantic embeddings for the global scene graph are also given, so finding the matching sub-graph for the new query seems relatively straightforward (unless I misunderstood some piece of complexity)\n\n - I find the level of rigour of the evaluation a bit troubling. The only baseline method considered is OVIR-3D is unpublished (there is some question about how the authors know who are the authors of this work and how they got access to the unreleased code...). Further to that, the DOVE-G which is created by the authors. \n\n - Further, since OVIR-3D does not have access to the additional information in the form of the 3DSG, is this really a fair comparison?\n\n\nOther Comments\n\n - I find the introduction of the manuscript a bit hard to follow. I would not consider \"fiducial markers\" to be a state-of-the-art method to perform natural language grounding. Other statements such as \"3D scene graphs ... bridge the gap between humans and robots\" are also difficult to interpret. \n\n - I feel that the related work should at least acknowledge the the work in language grounding that has taken place for the last decade or more before the widespread popularity of LLMs. One example from Stefanie Tellex's group (among others) is [1]. \n\n - The authors may also consider comparing to works such as ConceptFusion [2] which builds a 3D semantic/spatial map that can be queried in the way described in this work.\n\n - I had difficulty to understand what is the \"Positional Input\" and the \"User Input\". Is the \"Positional Input just the 3D scene graph without any semantic annotations?\n\n - The number of subscripts and superscripts get a little bit extreme at times. Perhaps it is unavoidable, but I found the mathematical notation somewhat laborious to follow. \n\n[1] Grounding natural language instructions to semantic goal representations for abstraction and generalization. Dilip Arumugam, Siddharth Karamcheti, Nakul Gopalan, Edward C. Williams, Mina Rhee, Lawson L. S. Wong & Stefanie Tellex. Autonomous Robots. 2018.\n\n[2] https://concept-fusion.github.io/", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission initially received mixed reviews.  The authors did a good job during the rebuttal phase, after which the reviewers all leaned toward acceptance.  The AC agrees with the recommendations.  Note that Reviewer UCiN pointed out that the recommendation is conditioned on the authors' promise to "release our code (including OVSG and OVIR-3D) and dataset (DOVE-G) for future robust evaluations."  The AC also agrees that the code and data release is critical given the nature of the work.  The authors should prepare code and data release along with the camera-ready version of the paper.
Decision: Accept (Poster)