CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding
Authors: Eslam Mohamed BAKR
Mohamed Ayman Mohamed
Mahmoud Ahmed
Habib Slim
Mohamed Elhoseiny
Venue: ICLR 2024 poster (2024)
Keywords: 3D referring
3D visual grounding
localization
3D
Primary area: general machine learning (i.e., none of the above)
TL;DR: We formulate the 3D visual grounding problem as a sequence-to-sequence (Seq2Seq) task, by predicting a chain of anchors first then utilizing them to predict the final target.

Abstract: 3D visual grounding is the ability to localize objects in 3D scenes conditioned by utterances. Most existing methods devote the referring head to localize the referred object directly, causing failure in complex scenarios. In addition, it does not illustrate how and why the network reaches the final decision. In this paper, we address this question “Can we design an interpretable 3D visual grounding framework that has the potential to mimic the human perception system?”. To this end, we formulate the 3D visual grounding problem as a sequence-to-sequence (Seq2Seq) task by first predicting a chain of anchors and then the final target. Interpretability not only improves the overall performance but also helps us identify failure cases. Following the chain of thoughts approach enables us to decompose the referring task into interpretable intermediate steps, boosting the performance and making our framework extremely data-efficient. Moreover, our proposed framework can be easily integrated into any existing architecture. We validate our approach through comprehensive experiments on the Nr3D, Sr3D, and Scanrefer benchmarks and show consistent performance gains compared to existing methods without requiring manually annotated data. Furthermore, our proposed framework, dubbed CoT3DRef, is significantly data-efficient, whereas on the Sr3D dataset, when trained only on 10% of the data, we match the SOTA performance that trained on the entire data. The code is available at github.com/eslambakr/CoT 3DV G.

COT3DREF: CHAIN-OF-THOUGHH DATA- 3D VISUAL GROUNDING
Eslam Mohamed Bakr, Mohamed Ayman, Mahmoud Ahmed, Habib Slim, Mohamed Elhoseiny King Abdullah University of Science and Technology (KAUST) (es 1am. abde midelrahmmnt mohamed. mohamed. 2, mahmoud. a ammd, habib. slim, mohamed. e elhoseiny eeu.st.duu.....
ABSTRACT
3D visual grounding is the ability to localize objects in 3D scenes conditioned by utterances. Most existing methods devote the referring head to localize the referred object directly, causing failure complex scenarios. In addition, it does not illustrate how and why the network reaches the final decision. In this paper, we address this question "Can we design an interpretable 3D visual grounding framework that has the potential mimic the human perception system?". To this end, we formulate the 3D visual grounding problem as a sequence-f--eqqueeee (Seq2Seq) task by first predicting a chain of anchors and then the final target. Interpretability not only improves the overall performance but also helps us identify failure cases. Following the chain of thoughts approach enables us to decompose the referring task into interpretable intermediate steps, boosting the performance and making our framework extremely data-efficient. Moreover, our proposed framework can be easily integrated into any existing architecture. We validate our approach through comprehensive experiments on the Nr3D, Sr3D, and Scanrefer benchmarks and show consistent performance gains compared to existing methods without requiring manually annotated data. Furthermore, our proposed framework, dubbed CoT3DRef, is significantly data-efficient, whereas on the Sr3D dataset, when trained only on 10% of the data, we match the SOTA performance that trained on the entire data. The code available at ligggiuub.com wtt.:/coomeeemmmmoo eslombakr Caannaaar//CoT3.DD CVV
INTRODUCTION
The 3D visual grounding task involves identifying and localizing objects in 3D scene based on a natu- ral language description or query. This task is crucial for many applications, such as roboties (Nguyen al.
bookshalf with the white and boon
2019; Karnan et al., [2222] Wijmans et al. 2019), ) vir-
tual reality (Puig et 20188 Ghasemi al.heemm Park et al. 2020 Osborne-Crooot [rowley][0200 Liu et al. 2019), and autonomous driving QQia et al. 2022
al. 2022
Cui et al. 2021; Jang al. 2017 Deng et al. 2021)
The goal is to enable machines to understand natu- ral language and interpret in the context of 3D environment. Although 3D visual grounding has sig- nificantly advanced, current solutions cannot imitate the human perception system nor be interpretable. To address this gap. we propose Chain-of- Thoughtsss 3D visual grounding framework, termed CoT3DRef. One of the biggest challenges in machine learning is understanding how the model arrives at its deci- sions. Thus, the concept of Chain-of- -hhiugghs (CoT)
Figure 1: Overview of our approach, where first predict a chain of anchors in logical order. In this example, reach the chair target, we first have to localize the white and red boxes, then the bookshelf.
comes in. Although CoT is widely applied in Natural Language Processing (NLP) applications Weei et al. 2022 Chowdhery et al. 2022 Lyu et al. 2023 Wang et al. 2022 Zhang et al. 2023 Madaan & Yazdanbakhsh| 2022), it less explored in vision applications. Understanding the Col crucial for several reasons. Firstly, it helps explain how the model arrived at its decision, which is essential for transparency and interpretability. Secondly, it helps identify potential biases or emrrs in the model,

---

MrlD
Sr10
Redentingheem
18. 543
00.5
MV7
Cet MVT
Redering
61a
764
CoT-5AT
MV7
SAT
Teatning Percentage
Training Data Percentage
5:10
Colvil VRL
CotVilL
Reteringgaag Acourcc
CAT
CCT-LAR LAR
25.3
Training Data Percentage
Training Data Percentage
Figure 2: Data efficiency results. To show the effectiveness our CoT architecture. we integrate into four architectures, i.c., MVT (Huang et al. 2022), SAT (Yang et al. (202T). LAR (Bakr et al. 2022) and ViL (Chen) ct al. 2022), across different amounts of training data (10% 100%).
which can be addressed to improve its accuracy and reliability. Third, it is critical step toward intelligent systems that mimic human perception. Similar to machine learning models, our perception system can be thought of as a Chain-of-Thoughh (Mc Vay Kane 2009 Chen al. (017)) scries of intermediate steps that enable us to arrive at our final perception of the world.
In this paper, we mainly answer the following question: Can we design an interpretable 3D visual grounding framework that has the potential to mimic the human perception system? To this end, we formulate the 3D visual grounding problem as a seuuuenneet--sqeenee (Seq2seq) task. The input sequence combines 3D objects from the input scene and an input utterance describing a specific object. On the output side, in contrast to the existing 3D visual grounding architectures, we predict the target object and a chain of anchors in causal manner. This chain of anchors is based on the logical sequence of steps a human will follow to reach the target. For instance, in Figure [ to reach the chair target, we first have to localize the white and red boxes, then the books f. By imitating the human learning process, we can devise a transparent and interpretable 3D framework that details the model's 's steps until localizing the target. To show that our proposed framework can be easily integrated into any existing architecture, we incorporated it into four different baselines: LAR (Bakr et al., 2022), SAT (Yang al.) (021), MVT (Huang al., (2022), and ViL. (Chen et al. 2022). CoT3DRef achieves state-of-the-ar results on Sr3D, Nr3D (Achlioptas et al., 2020), and ScanRefer (Chen et al. 2020) without requiring additional manual annotations by devising an efficient pseudo-label generator to provide inexpensive guidance to improve learning efficiency. Whereas boosts the performance by 3.6%, 4%, 5%, 0.5% on Nr3D and 10%, 11%, 9%, 1% on Sr3D, respectively. Proper design of such an approach is pivotal in attaining significant performance gain while circumventing the need for intensive human annotations. A pertinent example can be drawn from the labeling procedure employed in PhraseRefer (Yuan et al. 2022). which demanded a cumulative workforce commitment of 3664 hours, roughly equivalent to an extensive five-month timespan. Moreover, using additional manual annotations on the Nr3D dataset led to noteworthy enhancement in referring accuracy, boosting the performance by 9% compared to the baselines: LAR, SAT, and MVT, respectively. Consequently, using ScanRefer, by 6.5% and 6.8%, respectively. In addition, as depicted in Figure 2 CoT3DRef shows a remarkable capability of learning from limited data, where training on only 10% of the data is enough to beat all the baselines, which are trained on the entire data. Our contributions are summarized as follows:
approach surpasses SAT and MVT
We propose 3D data-efficient Chain-of-Thoughh based framework, CoT3DRef, that generates an interpretable chain of predictions till localizing the target.
We devise an efficient pseudo-label generator to provide inexpensive guidance improve learning efficiency.
Our proposed framework achieves state-of-the-ee performance on Nr3D, Sr3D, and Scan- Refer benchmarks without requiring manually annotated data.
Using 10% of the data, our framework surpasses the existing state-of-the-art methods.
2

---

Chain Thoughts
Rederring Hiad
Visual Encoder
Fusion
Chial
T
desce. booksted while boone.
Trrget
Head
bccoss
Bloes
Figure 3: An overview of our Chain-of-Thooee Thoughts Data-Efficient 3D visual grounding framework (CoT3DRef). First, predict the anchors O from the input utterance, then sort the anchors logical order O using the Pathway module. Then, feed multi-modas features parallel localized objects R and the logical path O to our Chain-of of-in-of-Te Thoughts decoder to localize the referred object and the anchors a logical order R"..
RELATED WORK
3D visual grounding. Significant progress has been made in 3D visual grounding thanks to ad- vancements in deep learning and computer vision, well the availability of grounded datasets
(Achlioptas et al. 2020; Chen et al. 2020 Abdelreheem et al., 2022a Yuan et al. 2022). One
approach involves using graph-based models (Achlioptas et 2020 Feng al. 2021 Yuan et al.)
202T; Huang et al. 2021) to represent the scene as nodes and edges, while attention mechanisms
help to focus on relevant parts of the scene. Another approach (Roh et 2022
convert the
visual input into language tokens using a classification head. These tokens can then be combined with input utterance and fed into a transformer architecture to learn the relationships between input sequence elements. Moreover, recent work (Bakr et al., 2022, Yang et al. (021) explores distilling knowledge from 2D to 3D in a multi-view setup. However, none of the existing works models the explicit reasoning process behind the prediction of the target object.
The chain-of-thoughh concept has been used in many different machine learning applications, including natural language processing (Wei al.] [022] Chowdhery al. 2022 Lyu
Chain of thoughts.
et al.
2023
Wang et al., 2022 Zhang et al. 2023) Madaan & Yazdanbakhshi 2022). and robotics (Ja
al.
2023
Yang et al. 2022). In the context of 3D Visual grounding, developing chain-of-thougee
approach provides a natural way to explicitly model the grounding reasoning process, which to the best of our knowledge has not been explored. An extended version is discussed in the Appendix A.9
COT3DREF
In this section, we propose a simple yet effective approach to decompose the referring task into multiple interpretable steps by modeling the problem Seq2Seq, as shown in Figure 3 First, briefly cover the general 3D visual grounding models' skeleton and its main components (Sec. 3.1).
Then, we present our method in detail (Sec. 3.2). Finally, we detail our loss function (Sec. 3.4
3.1 PRELIMINARIES
We built our framework in generic way where it could be integrated into any state-of-the-art 3D visual grounding model. An arbitrary 3D visual grounding model mainly consists of three essential blocks; a Visual Encoder, a Language Encoder, and a Multi-Modal Fusion module.
Visual Encoder. An arbitrary 3D scene S Nx6 represented by N points with spatial and color information, i.e., XYZ, and RGB, respectively. Using one the off-the-shelf 3D object detectors or manual annotations, we have access to the object proposals P = (P& H where P& E N x6, N' represents the number of the object's points and L is the number of proposals in the scene. Then, the visual encoder encodes the proposals into lower-resolution feature maps V = (V&)= where
VR E R xd and d is the number of hidden dimensions.
Language Encoder. Simultaneously, given input utterance that describes particular object in scene, called target, a pre-trained BERT model (Devlin et al. 2018) encodes the input sentence into T = (T)) 3=1 where W is the maximum sentence length. Then, language classification head
3

---

is utilized to predict only the referred object. We notice that limiting the language encoder only predict the target object restricts its capability of learning representative features.
Multi-Modal Fusion. After encoding the object proposals and the utterance, multi-modal fusion block is exploited to refine the visual features V based on the language embeddings T generating fused
features J { //2 where Fg Rixd Both graphs (Achlioptas et al. 2020 and transformers
(Huang et al., 2022 Bakr et al., 2022 Yang et al., (222T) are explored to capture the correlation between the two different modalities. However, our CoT framework can be integrated easily into any
existing 3D visual grounding architecture, regardless of how the multi-modal features F. are obtained.
3.2 CHAIN-OF-THOOUGII
We decompose the referring task into multiple interpretable steps, whereas to reach the final target the model must first predict the anchors one by one, in a logical order called Chain-of-Thoughts To this end, we first have to predict the anchors from the input utterance, then sort the anchors in a logical order using our Pathway module. Then, we replace the naive referring decoder with our Chain-of- -hhinhof-Thoughts decoder.
We formulate the referring task as a Seq2Seq problem by localizing the anchors as intermediate step. Instead of anticipating the target directly, we first predict the chain of anchors sequentially, then utilize them to predict the target.
Pathway generation. First, we extend the language head, ie.. Target Anchors Head, to extract both the target and the anchors OT {O,"," li=1 where M is the maximum number of objects in the sentence, depicted in the lower red part of 3 We add a "no_obj" class to pad the output to the maximum length M. However, the predicted anchors are unsorted or sorted based the occurrence order in the sentence, which can not be fit in our Chain-of-Thoughts Thoughts framework. Accordingly, we introduce "Pathway Head" which takes the encoded sentence T and the predicted objects of the utterance T to produce logically ordered objects O = =o!")
M
)===" A One possible solution is to exploit an MLP head to predict the logical order for each object. However, for better performance, we use a single transformer encoder layer to capture the correlation between different objects.
Sequeeceeto-Seqqeene Similar the language stream, we first employ a parallel referring head to localize the referred object alongside the anchors, ignoring their logical order. The parallel referring head only takes the multi-modal features F as input and localizes the target and the anchors R-== (R(() Ji=1. We experiment both with localizing the target and the anchors in parallel, and
R. = localizing them one by one in a sequential logical order using previously localized objects as prior information. In other words, we formulate the 3D referring task as Seq2Seq task, where the input is two sequences; 1) a set of object proposals P, 2) a sequence of words (utterance). The output is sequence of locations for the anchors and the target. Specifically, we add positioning awareness to the plain localized objects R using the predicted ordered objects OP, which act as a positional encoding. These positions indicate logical order that mimics human perception (My & Kane 2009; Chen et al. 2017). Furthermore, we employ a single transformer decoder layer, depicted in Figure 3 localize the objects in sequence w.r.t the predicted logical order OP. In this decoder layer, the queries are RF + O" and the values and keys are F. Accordingly, the attention maps denoted as A follow Eq.
J
A=o( RR + +OP)FT
Vd
where a is the Softmax function and d is the embedding dimensions. We use a masked self-attention layer to enforce the Chain-of-Thoughhe where while predicting the next object's location, we only attend to the previously located objects. This could be interpreted as a one- one-direction COT. We also experiment with another variant where no masking is applied so that we can attend to any object in the chain, as shown in Appendix A.6
3.3 PSEUDO LABELS
During the training phase, our proposed framework requires more information than the standard available GT in existing datasets (Achlioptas et al. 2020 Chen et al.) 2020). These datasets only annotate the referred object and the target. However, our framework requires anchor annotations. Three types of extra annotations are needed: 1) Given an input utterance, we need to identify the
4

---

mentioned objects other than the target; the anchors. 2) Once extract the anchors from the utterance, we need to know their logical order to create a chain of thoughts. 3) Finally, we need the localization information for each anchor, i... to assign a bounding box every anchor.
To make our framework self-contained and scalable, we do not require any manual effort. Instead, collect pseudo-labels automatically without any human intervention.
Anchors parser. We extract the textual information from the utterance using rule-based heuristics and scene graph parser (Schuster et al.) 2015; Wu et al.) (022). First, we extract the whole mentioned objects and their relations from the utterance using the scene graph parser. Then, we match the objects to their closest matching class from the ScanNet labels (Dai et al. 2017a) using SBERT (Reimers & Gurevych 2019). Due to the free-from nature of Nr3D, the anchors mentioned in the GT descriptions sometimes do not precisely match the ScanNet class labels. For instance, the GT description is "The plant at the far right-hand side of the bookcase tucked in the furthest corner of the desk." However, there no "bookcase" class in ScanNet. Therefore, we need match it to the nearest ScanNet class labels, which in this case will be "bookshelf."
Anchors pathway. We utilize GPT-3.5 (Ouyang et al. 2022) to extract the logical order of objects given an input utterance, using in-context learning (Brown et al. 2020). The full prompt used for pathway extraction is provided in Appendix(A.12
Anchors localization. The anchors localization module employs object proposals P, extracted
relations and utterance objects oT to establish associations between anchors and object bounding boxes within an input scene. Our method involves iterating over all anchors extracted from the utterance and searching for candidate objects P that share the same class. When the anchor class is represented singularly in the scene, return the matched object. However, in scenarios where disambiguation is required due to multiple objects belonging to the anchor class, we leverage the parsed spatial relations and the localized objects within the scene to identify the intended anchor accurately, termed FIND in Algorithm[] However, it is not guaranteed that the FIND function will be able to localize the remaining unlocalized anchors accurately. Thus, in this case, as shown in the last step in Algorithm 1, we randomly sample an object of the same class. We summarize our localization method in Algorithm T
Algorithm Localizing objects mentioned in an input utterance
Input:
-o (ax N unlogalized objects mentioned utterance, extracted using syntactic parser. scene P (B, C). object proposals bounding boxes b, E R and {cr ( classes of the object proposals for the
8 [b,) 1-
RELATE: OT R x OT: map objects mentioned the input utterance to (spatial relation, object) pair.
FIND: P x R pm P: map a localized object, a relation and set of candidate objects to output localized object.
Output:
A: localized anchors used the utterance, where A P.
funetion LOCALIZC(o,
Localizing single utterance
2:
IC +0+- [P, 1 Cj. E C)
b. Find a set of candidates with the same class > single object has anchor class
[[]
return IC
5: for (Cm, o RELATE(O) do
if A then: return FIND K.)
Dr The related object is already lncalized P. Otherwise, attempt to localize the related object
p LOCALIZEOO,, P. A) if # 0 then: return (p) U FIND(p, Km,
8:
return (p).p (pr p: ()) P)
If all else fails: randomly sample object of the same class
10: A
0
11: for OT do: A LOCALIZLOO P.A)
3.4 LOSSES
Mainly, three losses exist in most existing 3D visual grounding architectures. Two of them are considered auxiliary losses, i.e., 3D classification loss Lyt and language classification loss CT Hence, the third one is the primary loss, i.e., referring loss Cref- First, we extend the language classification loss LT, by recognizing the referred object class and the anchors based on the input utterance. Similarly, the referring loss extended to localize both the target and the anchors, termed as parallel referring loss C, as we localize the target and anchors in one step. Furthermore, we add another referring loss after the transformer decoder, termed CoT referring loss LCOT Finally, an
5

---

Table I: Ablation study for different components of our CoT3DRef framework. First. compare baseline. ie., MVT (Huang et 2022 against parallel and the chain of oh-hhggsst approaches. Then, we show the effect of adding distractor (Achlioptas et nl 2020
Table Ablation study for different numbers of the trans- former blocks used CoT decoder, based on MVT.
the experiments are conducted using Nr3D dataset
Transformer Blocks
Nr3D 1 1Sr3D
73,2
Data Percentage
+Distractor, Loss,
2
60.4
73,3
+ParalleI+CoI Nr3D Sr3D 1
4
60.1
72.9
(a)
10%
27.6
48,8
Table 3: Ablation study highlight the effect of the anchors' quality on the final target referring accuracy, based on Nr3D dataset and MVT as baseline.
(b) 10%
31.7
55.3
(c)
10%
37.5
65.2
(d)
10%
38.2
66.4
(e) 100%
55.2
66.0
Method
Ref. Aec. Ref. Acc.
Anchors Target
(8)
100%
/
57.0
71.5
Baseline
N/A
55.1
(g) 100%
60.0
72.7
.CoT +Zeroing Ancher
Labels
(h) 100%
60.4
73.2
-Human Labels
auxiliary distractor binary classification loss Laist is introduced to distinguish between the target and the distractors, i.e., objects with the same class as the target. Grouping all these losses, we optimize the whole model in an end-to-end manner with the following loss function:
L (Av f...) (AT -...) Aves (Ches + EreeT) ) Adiet Edist,
(2)
where À is the corresponding loss weight for each term.
4 EXPERIMENTAL RESULTS
Datasets. To probe the effectiveness of our proposed framework, CoT3DRef, we conduct evalua- tions on three 3D visual- -rouundig benchmarks, namely Nr3D, Sr3D (Achlioptas et al., 2020) and ScanRefer (Chen et al. 2020). Nr3D contains 41.5K natural, free-form utterances gathered from humans through a referring game, while Sr3D consists of 83.5K synthetic utterances. Consequently, ScanRefer provides 51.5K utterances 11K objects for 800 3D indoor scenes.
Network Configuration. We model the Pathway module using only one transformer encoder layer, and the CoT decoder using a single transformer decoder layer. The number of heads used are and 16 for the Pathway module and CoT decoder, respectively. The number of proposals L and the maximum sentence length are 52 and 24, respectively. L and W define the sizes of the input sequences to our CoT decoder. The maximum number M of objects in the sentence, the output sequence length for CoT decoder, is and 3 for Nr3D and Sr3D, respectively. Following previous works (Abdelreheem
2022b AAhlioptas et al. 2020 He et al. [021 Roh et al.) 2022 Jain et al. [2021] Yang et al. Qi et al.) 2017). we randomly sample 1024 points for each proposal, set the hidden dimensions d to 768, and train the model for 100 epochs from scratch using the weight initialization strategy described in (He et al. (015)). The initial learning rate is set to 10 and decreases by 0.65 every ten epochs. The Adam optimizer (Kingma & Bal 2014) and a mini-batch size of 24 per GPU are used for training all the models. We set the losses weights as follows: X 5. XT = 0.5, bref = 5, and Adist = We used the PyTorch framework and a single NVIDIA A6000 GPU for training.
al.
2021
4.1 ABLATION STUDIES
We conducted several ablation studies to validate each module in our framework, termed CoT3DRef.
CoT Parallel. To assess our CoT framework, we have disentangle the CoT from the pseudo label generation module. In other words, it could be thought that the achieved gain caused by just accessing additional supervision signal, i..., anchors' annotations. To this end, we implement a parallel approach, that has access to the anchors' labels, however, localizes the targets and anchors in one shot without any interaction between them. In contrast, our CoT3DRef framework leverages the causality between the anchors and the target through our chain-of of-thoughts decoder. As shown in Table [ on the challenging setup, where we assume access for only 10% of the training data while testing the entire testing dataset, the parallel variant boosts the performance by 4% and 6.5% over the vanilla MVT using Nr3D and Sr3D, respectively (row b). On the other hand, our CoT3DRef framework surpasses the vanilla MVT by 10% and 16.4% using Nr3D and Sr3D, respectively (row c). Consequently, using the entire data, our CoT surpasses both the parallel and the vanilla MVT approaches by 3% and 5% on Nr3D, and by 1% and 6.7% Sr3D, respectively.
6

---

Table 4: Benchmarking results on Nr3D and Sr3D datascts (Achlioptas ct al. 2020). We emphasize that we did not require any additional GT annotations, contrast SAT Yang al. 2021] which requires access to 2D images, and PhraseRefer (uaan et al. 2022) and ScanEnts (Abdelrcheem al. 2022a require manual annotations for whole anchors the data. We report the standard ddvviation a green.
Method
GT Anchors
NrAD 1
Sr3D
Overallie)
Easy
Hard
View dep.
View indep.
Overalli))
Eas
Hard
View dep.
View
Refertr3D
Achlioptou
botp
VIA
43.6
27.9
12.5
40.8
31.5
10.2
Goi--Guidoo
lostane:Reerr
N/A
460
31.8
3DRe/Tnms
N/A
39.0
454
32.0
34.7
41.2
47.0
50.7
44.3
47.1
3DVG-Tome
7r mi S1Y4
N/A
40.8
48.5
3488
3488
43.7
51.4
54.2
44.9
44.6
51.7
3DOG
NA
41.7
48.2
35.0
37.1
44.7
TransRefer317
HC
20:1
N/A
42.1
48.5
36,D
36.5
44.5
57.4
61.5
50.
49.9
57.3
langungRRefe
Rech
9023
N/A
43.9
51.0
36.6
41.7
45.0
$6.0
58.9
49.3
49.2
$6.3
3D-SPS
48.0
56.1
4G.7
=CoTTSricer (Oii))
52.3:0.1
59.4
45.7
50.4
53.6
.<Co133RRe (Ours)
58.0-0.2
65.1
50.9
54.6
60.0
69.0-110
74.2
59.4
58.9
70.1
SAT (Song et 2021
NA
49.2
56.3
12.4
16.9
50.1
57.9
61.2
50.0
49.1
58.3
.Phrátekater Your al. 2022
54.4
62.1
47.0
51.2
56.0
63.0
66.2
55.4
48.7
63.6
*Scantiass (Aloetienzem 2022a
59.8
45.6
51.3
53.2
-CCTT3RRTTO00
531-0.2
51.8
53.9
+CoTTDRee
51.6
57.0
71,9
6133
4901
54.3
61.3
66.9
553
554
64.7
*SSannzzss (Ancereneeee 2022a
59.3
65.4
53.5
57.3
00.4
-Phraseektter etane 2012
59.0
68.0
#CoT3DRRe (OBI))
60.4:0:2
66.2
54.6
58.1
61.5
#CuT3DRRf (Ours)
64.4-0.2
70.0
59.2
61.9
65.7
73.2-0.5 5
75.2
67.9
67.6
735
VII.DK (Chon
563
#CUT3DRenOOuS
57.5
-CoT3DRef (Ours)
6400002
70.4
57.3
61.5
6488
7.66:05
73.0
67.2
61.9
Table 5: Benchmarking results while training jointly Nr3D and Table 6: Benchmarking results on Scan-
Sr3D datasets (Achlioptas al. 2020)
Refer dataset (Chen al. 2020).
Method
NED+S3D
Method
Data Pencentage
Overalllo)
Easy
Hard
View-dop.
View-indop.
100%
Redertt:3
Acthlopptas et al 2000
37.2 0.3 44.0 30.6 + 0.3 333 D.
39.1 + 0.2
MVT (Huang et al 2022
36.4
51.5 51.5 57.4
SAT Minrer o 1
43,9 0.3
PhraseRater Tuai et 2022
59.9
TrandRfe
554
3933 0.5
5066
62.5
64.2
Yang 20040
16.7
SAT eH 12021
34.9
49.3
50.5
5388
Roin er
05
0.3
+PhraseReter Your dea.
2022
57.5
MVT COVIDCO1D Ourst
62.52004 7.00:0.5
53.0:002
58.3.0.1
63.90..3
:CoT3Dker (Ours)
60.3
Distractor Loss. We add an auxiliary distractor binary classification loss Caist to disambiguate the target and the distractors, as discussed in Sec. 3.4 As shown in Table [ rows d and h, incorporating boosts the performance by 0.5-1%.
Anchors Quality Effect. To establish the affirmative role of anchors in refining target localization accuracy without inducing detrimental effects, we conducted a worst-case simulation. In this simulation, all anchors were falsely detected. In other words, we deliberately zero the localization loss associated with anchors during training, causing the decoder randomly and inaccurately predict anchor locations. Remarkably, the target accuracy remained unaltered, as shown in Table 3 reflecting the robustness of the approach. The accuracy experienced a decrement of approximately 5%, dropping from 60.4% 55%. Encouragingly, even in this scenario, the accuracy mirrored the baseline performance, steadfast at 55.1%. This substantiates that while anchor detection may exhibit inaccuracies, the broader framework's efficacy in target localization remains largely unaffected. In contrast, we replaced the pseudo labels with manual annotations (Abdelreheem et al.) 2022a)). This substitution serves as an upper-bouee bound reference point for evaluation. As shown in Table the exchange of noisy pseudo labels with precise manual annotations led to noteworthy 4% enhancement referring accuracy for Nr3D, elevating it from 60.4% to 64.4%.
Number of Transformer Blocks. As shown in Table 2 we have explored using 1, 2, and transformer blocks in our CoT referring decoder. However, we didn't 't notice a significant gain in performance; therefore, we preferred to use a single transformer block.
4.2 COMPARISON STATE-OF-TIIB-AR
We verify the effectiveness of our proposed framework, termed CoT3DRef on three well-known 3D visual grounding benchmarks, i.e., Nr3D, Sr3D (Achlioptas et al. 2020) and ScanRefer (Chen et al. 2020). By effectively localizing chain of anchors before the final target, achieve of art results without requiring any additional manual annotations. As shown in Table 4 when we integrate
module into four baselines; LAR (Bakr et al. 2022). SAT (Yang al. 2021
MVT (Huang et al.

---

GT
MVT
Table at the end of the room in front of the black chairs
The table with chairs around it.
This chair is pulled from the tables and facing whiteboard.
With the counter front of you, it's the 2nd stool from left.
monitors on it.
Figure 4: Our qualitative results. First row indicates the GT w.r.t the input utterance, demonstrated in the last row. Second and third rows show the qualitative results for MVT our method, respectively. The success and the failure cases are shown in green and red boxes, respectively.
2022) and ViL (Chen al. 2022), it boosts the accuracy by 3.6%, 4%, 5%, 0.5% Nr3D and 10%, 11%, 9%, 1% on Sr3D, respectively. The disparity between the gain achieved on Nr3D and Sr3D due our pseudo label module that hinders achieving more gain on Nr3D. Exchanging of noisy pseudo labels with precise manual annotations led to noteworthy enhancement in referring accuracy, where our module boosts the performance by 9% compared to the baselines: LAR. SAT and MVT, respectively. This outcome underscores our model's 's ability to yield enhanced performance not only for simpler descriptions (Sr3D) but also in the context of more intricate, free-form descriptions (Nr3D). We have limited gain on only ViL. A detailed analysis that justifies this behaviour is mentioned in the Appendix A.4
Nr3D+Sr3D. In addition, we jointly train on Nr3D and Sr3D. Whereas, augment the Nr3D training data with Sr3D, while testing on the same original Nr3D test-set. Consistently with the previous results mentioned in Table and we surpass all the existing work by 3.5%. As shown in Table we achieve 62.5% grounding accuracy, while MVT only achieves 58.5% accuracy.
ScanRefer. To further show the effectiveness of our proposed method, we have conducted several experiments on the ScanRefer dataset across different data percentages on MVT and SAT baselines. As shown in Table data percentages, i.e., 10%, 40%, 70%, and 100%. More specifically, integrating our CoT framework into MVT has boosted the performance by 12.2%. 8.6%, 8%, and 6.8%, respectively. In addition MVT, we have integrated our CoT framework into the SAT bascline, where the same performance gain has been achieved, probing our method's effectiveness across comprehensive range of baseline models, datasets, and different available data percentages.
outperform both MVT and SAT by significant margin across the entire
Data Efficiency. To further validate the effectiveness of our framework, we assess it a more challenging setup, where we assume access to only limited data. Four percentage of data is tested, i.e., 10%, 40%, 70%, and 100%. As shown in Figure 2 on the Sr3D dataset, using only 10% of the data, we match the same performance of MVT and SAT that are trained on 100% of the data. This result highlights the data efficiency of our method. Furthermore, when trained on 10% the data on Nr3D with noisy pseudo labels (Sec. 3.3 we still surpass all the basclines with considerable margins.
Qualitative results. As shown Figure the first three examples show that our model successfully localizes the referred objects by leveraging the mentioned anchors, such as "the table with 5 chairs around". However, in the ambiguous description shown in the fourth example: "2nd stool from the left", the model incorrectly predicts the stool, as it is depeddennd.nt. In other words, if you look at the stools from the other side, our predicted box will be correct. Additionally, the last example shows challenging scenario where negation in the description is not properly captured by model.
8

---

5 DISCUSSION AND LIMITATIONS
Comparison with PhraseRefer (Yuan et al., 2022) and ScanEnts (Abdelreheem et al., 2022a). We acknowledge the great work proposed by PhraseRefer (Yuan et al. (022) and ScanEnts (Abdelreheem et al. 2022a), which paved the way for the importance of paying attention not just to the target object but also to the anchors. Thus, underlying approaches have some similarities in terms of demonstrating the importance of including anchors in the pipeline. However, there are a lot of significant differences: 1) Design CoT framework for 3D visual grounding that explicitly models causal reasoning while interpreting the instruction inspired by the human perception system. 2) Show that we can design an efficient framework that achieves tfat-of-theear results on three challenging benchmarks, i.e., Nr3D, Sr3D (Achlioptas et al., 2020). and ScanRefer (Chen et al., 2020), without requiring human labels. Proper design of such an approach is pivotal in attaining a good performance while circumventing the need for intensive human annotations. A pertinent example can be drawn from the labeling procedure employed in PhraseRefer (Yuan et al., 2022), which demanded a cumulative workforce commitment of 3664 hours, roughly equivalent to extensive five-month timespan.
Pseudo labels accuracy. The accuracy of the pseudo-labels plays a vital role in the overall perfor- mance. To evaluate its performance,
manually collect ground-truth labels for 1) the predicted orderings of the anchors in the utterance and 2) the final localization of the anchors as predicted by the geometry module, based on 10% of the Nr3D. To evaluate orderings predicted by in-context learning, we use the normalized Levenshtein edit distance between two sequences, where a length of I means that every object in the sequence is incorrect. We achieve an average distance of 0.18 between predicted and ground tround-truth orderings. We consider anchor-wiss localization accuracy to evaluate the geometry module's accuracy, where the anchors considered for each sequence are those mentioned in the input utterance. We achieve a 77 % accuracy for this task compared to human annotators. Overall, a significant accuracy gap is measured between automatically collected pscudo-labels and ground-truth data, contributing to the performance loss observed on the Nr3D dataset.
Pseudo module limitations. Despite designing a data efficient framework that could be integrated into any existing 3D visual grounding architecture, and achieve SOTA results without requiring any manual annotation efforts, our pseudo label module hinders achieving more gain on Nr3D. Accordingly, we encourage the future efforts to try enhance the pseudo module performance. In addition, the anchor localization block in our pseudo module is tailored on ScanNet dataset (Dai al. 2017a), and will thus need some adaptations be usable on other 3DD scene datasets.
Pathway module limitations. Our Pathway module is responsible of generating a chain of logical order for the extracted objects from the input utterance. However, it does not handle the multi-path scenario, where multiple paths are valid. For instance, given this utterance "It is the chair besides the desk, which has book and lamp above it", we have two possible starting points, i.e., locating the lamp first or the book. Thus, for simplicity, we start by the last mentioned object in the utterance, in the multiple paths scenario. Nevertheless, one possible solution to handle this limitation implicitly through building graph that reasons the different possibilities (Salzmann et al. 2020).
6 CONCLUSION
We propose CoT3DRef: a novel and interpretable framework for 3D visual grounding. By formulating the problem of 3D visual grounding from a natural language instruction as sepuence-to-sqqeemee task, our approach predicts chain of anchor objects that are subsequently utilized to localize the final target object. This sequential approach enhances interpretability and improves overall performance and data efficiency. Our framework is data-efficient and outperforms existing methods on the Nr3D and Sr3D datasets when trained on a limited amount of data. Furthermore, our proposed chain-of- thoughts module can easily be integrated into other architectures. Through extensive experiments, we demonstrate consistent performance gains over previous state-of-the-ar methods operating on Referit3D. Importantly, our approach does not rely on any additional manual annotations. Instead, we leverage automatic rule-based methods, syntactic parsing, and in-context learning to collect pseudo-labels for the anchor objects, thereby eliminating the laborious and cone-con process of manually annotating anchors. Overall, our work advances 3D visual grounding by making step towards bridging the gap between machine perception and human like understanding of 3D scenes.

---

REFERENCES
Ahmed Abdelrcheem, Kyle Olszewski, Hsin- Ying Lee, Peter Wonka, and Panos Achlioptas. Sca- nents3d: Exploiting phrase-to-3d-objeet correspondences for improved visio-linguistic models in 3d scenes. arXiv preprint arXiv:2212.06200 2022a.
Ahmed Abdelrcheem, Ujjwal Upadhyay, Ivan Skorokhodov, Rawan Al Yahya, Jun Chen, and Mohamed Elhoseiny. 3dreftransformer: fine-grained object identification in real-world scenes using natural language. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 3941-3950, 2022b.
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mohamed Elhoseiny, and Leonidas Guibas. Referit3d: Neural listeners for fine-grained 3d object identification real-world scenes. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I 16, pp. 422-440. Springer, 2020.
Eslam Bakr, Yasmeen Alsaedy, and Mohamed Elhoseiny. Look around and refer: 2d synthetic semanties knowledge distillation for 3d visual grounding. Advances in Neural Information Processing Systems, 35:37146-37111 2022.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners, July 2020. URL http://arxiv. / abs/ 2005 14165 arXiv:2005. 14165 [cs].
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. Scanrefer: 3d object localization in rgb-d scans using natural language. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XX, pp. 202-221. Springer, 2020.
Lang Chen, Matthew A Lambon Ralph, and Timothy T Rogers. A unified model of human semantic knowledge and disorders. Nature human behaviour, 133)00039, 2017.
Shizhe Chen, Pierre-Louis Guhur, Makarand Tapaswi, Cordelia Schmid, and Ivan Laptev. Language conditioned spatial relation reasoning for 3d object grounding. arXiv preprint mrXiv:2211.09644 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint 2xXiv:2204.03311 2022.
Kevin Crowston. Amazon mechanical turk: A research tool for organizations and information systems scholars. In Anol Bhattacherjee and Brian Fitzgerald (eds.), Shaping the Future of ICT Research. Methods and Approaches, pp. 210-221, Berlin, Heidelberg, 2012. Springer Berlin Heidelberg. ISBN 978-3-642-3511
Yaodong Cui, Ren Chen, Wenbo Chu, Long Chen, Daxin Tian, Ying Li, and Dongpu Cao. Deep learning for image and point cloud fusion in autonomous driving: A review. IEEE Transactions on Intelligent Transportation Systems, 23(2):722-733 2021.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes, 2017a
Bo Dai, Yuqi Zhang, and Dahua Lin. Detecting visual relationships with deep relational networks, 2017b.
Prithiviraj Damodaran. Parrot: Paraphrase generation for nlu., 2021.
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to- end visual grounding with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1769-1779, 2021.
10

---

Jacob Devlin, Ming-
Ming-Wei Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:181000488 04800, 2018.
Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2021.
Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffusion guidance for compositional text-to-image synthesis, 2023.
Yalda Ghasemi, Heejin Jeong, Sung Ho Choi, Kyeong-Beon Park, and Jae Yeol Lee. Deep learning- based object detection in augmented reality: A systematic review. Computers in Industry, 139: 103661, 2022.
Dailan He, Yusheng Zhao, Junyu Luo, Tianrui Hui, Shaofei Huang, Aixi Zhang, and Si Liu. Transre- fer3d: Entity-and-relation aware transformer for fine-grained visual grounding. Proceedings of the 29th ACM Infernational Conference Multimedia, pp.
2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-10344 2015.
Joy Hsu, Jiayuan Mao, and Jiajun Wu. Ns3d: Neuro-symbolic grounding of 3d objects and relations. arXiv preprint arXiv:2303. /343,, 2023.
Pin-Hao Huang, Han-Hung Lee, Hwann- Tzong Chen, and Tyng-Luh Liu. Text-guided graph neural networks for referring 3d instance segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 1610-1618, 2021.
Shijia Huang, Yilun Chen, Jiaya Jia, and Liwei Wang. Multi-view transformer for 3d visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 15524-15533, 2022.
Matthias Hutsebaut-Buyee Buysse, Kevin Mets, and Steven Latré. Hierarchical reinforcement learning: A survey and open research challenges. Machine Learning and Knowledge Extraction, 4(1):172-221, 2022.
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Looking outside the box to ground language in 3d scenes. arXiv preprint arXiv:21 2021.
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In Computer Vision- ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pp. 417-433. Springer, 2022.
Eric Jang, Sudheendra Vijayanarasimhan, Peter Pastor, Julian Ibarz, and Sergey Levine. End-to-end learning of semantic grasping. arXiv preprint arXiv:1707.01992 2017.
Zhiwei Jia, Fangchen Liu, Vineet Thumuluri, Linghao Chen, Zhiao Huang, and Hao Su. Chain-of- thought predictive control. arXiv preprint arXiv:2304.00776 2023.
Yong Jiang, Wenjuan Han, and Kewei Tu. Unsupervised neural dependency parsing, pp. 763-771, 01 2016. doi: 10. 00.18653vv//16.. 1073.
Justin Johnson, Agrim Gupta, and Li Fei-Fei. Image generation from scene graphs, 2018.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Ishan Misra, Gabriel Synnaeve, and Nicolas Carion. MDETR- - modulated detection for end-to-end multi-modal understanding. CORR, abs/2104.12763, 2021. URL.https: / /ttp//aavvvv//e////e//eee/...ava org. abs / /21 04. 12763
Haresh Karnan, Garrett Wamell, Xuesu Xiao, and Peter Stone. Voila: Visual-observation-oo imitation learning for autonomous navigation. In 2022 International Conference on Robotics and Automation (ICRA), pp. 2497-2503. IERE, 2022.

---

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Yikang Li. Wanli Ouyang, Xiaogang Wang, and Xiao' ouo'ou Tang. Vip-enn: Visual phrase guided convolutional neural network, 2017a.
Yikang Li, Wanli Ouyang, Bolei Zhou, Kun Wang, and Xiaogang Wang. Scene graph generation from objects, phrases and region captions, 2017b.
Luyang Liu, Hongyu Li, and Marco Gruteser. Edge assisted real-time object detection for mobile aug- mented reality. In The 25th annual international conference on mobile computing and networking, pp. 1-16, 2019.
Chao Lou, Wenjuan Han, Yuhuan Lin, and Zilong Zheng. Unsupervised vision- laiiu-la parsing: Seamlessly bridging visual scene graphs with language structures via dependency relationships, 2022.
Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haibing Ren, Hao Shen, Huaxia Xia, and Si Liu. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16454-16463, 2022.
Qing Lyu, Shreya Havaldar, Adam Stein, Li Zhang, Delip Rao, Eric Wong, Marianna Apidianaki, and Chris Kallison-Burchh Faithful chain-of tfuhh-t reasoning, arXiv preprint arXiv:2301. 13379, 2023.
Aman Madaan and Amir Yazdanbakhsh. Text and patterns: For effective chain of thought, it takes two tango. arXiv preprint arXhv:2209.07653 2022.
Jennifer C Vay and Michael J Kane. Conducting the train of thought: working memory capacity, goal neglect, and mind wandering in an executive-control task. Journal of Experimental Psychology: Learning, Memory, and Cognition, 35(1):196, 2009.
Khanh Nguyen, Debadeepta Dey, Chris Brockett, and Bill Dolan. Vision-based navigation with language-based assistance via imitation learning with indirect intervention. In Proceedings of the IEÉE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12527- 2019.
Katherine Esborne-Crowley, Social cognition in the real world: reconnecting the study of social cognition with social reality. Review of General Psychology, 24(2):144-15 2020.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow 2022.
instructions with human feedback. arXiv preprint
Kyeong-Beom Park, Minseok Kim, Sung Ho Choi, and Jae Yeol Lee. Deep learning- smart task assistance in wearable augmented reality. Robotics and Computer-Integratte Manufacturing, 63:101887, 2020.
Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 8494-8502. 2018.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. Advances in neural information processing systems, 30. 2017.
Rui Qian, Xin Lai, and Xirong Li. 3d object detection for autonomous driving: survey. Pattern Recognition, 130: 108796, 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, abs/1910. 10683, 2019. URL http: wwwp:/oggg/ab.//..
12

---

Nils Reimers and Iryna Gurevych. Bentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint 1908. 10084, 2019.
Junha Roh, Karthik Desingh, Ali Farhadi, and Dieter Fox. Languagerefer: Spatial-language model for 3d visual grounding. In Conference on Robot Learning, pp. 1046-1056. PMLR, 2022.
Tim Salzmann, Boris Ivanovic, Punarjay Chakravarty, and Marco Pavone. Trajectron+++: Dynamically- feasible trajectory forecasting with heterogeneous data. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XVIII 16, pp. 683-700. Springer, 2020.
Sebastian Schuster, Ranjay Krishna, Angel Chang, Li Fei-Fei, and Christopher D. Manning. Gener- ating semantically precise scene graphs from textual descriptions for improved image retrieval. In Proceedings of the Fourth Workshop Vision and Language, pp. 70-80, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10.18653/11W115-22222 URL htttps://aclattlie htt.:aatttt.oooooooooool----
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks, 2018.
Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint 1 2022.
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning point clouds, 2019.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting clicits reasoning in large language models. arXiv preprint arXiv:2201.11903. 2022.
Erik Wijmans. Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, and Dhruv Batra. Embodied question answering in photorealistic environments with point cloud perception. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6659-66688 2019.
Hao Wu, Jiayuan Mao, Yufeng Zhang, Yuning Jiang, Lei Li, Weiwei Sun, and Wei-Ying Ma. Unified visual-semantic embeddings: Bridging vision and language with structured meaning representations. In 2019 IEEE/CVF Conference Computer Vision and Pattern Recognition doi: 10. 10.1109/CVPR.2219.000
(CVPR), pp. 6602-6611, 2019.
Yanmin Wu, Xinhua Cheng, Renrui Zhang, Zesen Cheng, and Jian Zhang. Eda: Explicit
text-decoupling and dense alignment for 3d visual and language learning. arXiv preprint arXiv:2209. 14941, 2022.
Danfei Xu, Yuke Zhu, Christopher B. Choy, and Li Fei-Fei. Scene graph generation by iterative message passing, 2017.
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruy Batra, and Devi Parikh. Graph r-cnn for scene graph generation, 2018.
Mengjiao Sherry Yang, Dale Schuurmans, Pieter Abbeel, and Ofir Nachum. Chain of thought imitation with procedure cloning. Advances in Neural Information Processing Systems, 35:36366 36381, 2022.
Zhengyuan Yang, Songyang Zhang, Liwei Wang, and Jiebo Luo. Sat 2d semantics assisted training for 3d visual grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1856-1866, 2021.
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. Instancerefer: Cooperative holistic understanding for visual grounding point clouds through instance multi-level contextual referring. In Proceedings the the JEEE/CVF International Conference
Computer Vision pp. 1791-1800, 2021.
13

---

Zhihao Yuan, Xu Yan, Zhuo Li, Xuhao Li, Yao Guo, Shuguang Cui, and Zhen Li. Toward explainable and fine-grained 3d grounding through referring textual phrases. arXiv preprint arXiv:2207.0182./ 2022.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thought reasoning in language models. arXiv preprint arXi::2302.00923, 2023.
Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 3dvg-transe tevg-transformer: Relation modeling for visual grounding on point clouds. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2928-2937 2021.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yinggian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. A Survey of Large Language Models, 2023.
14

---

1x
Figure 5: Identification failure cases as a benefit of the interpretability. In this example, there are Two anchors mentioned in the description, desk and monitor, and the target is the chair. The correct chair should be number two, however, the model predicts number four. By visualizing the attention maps, the left, localize a wrong desk (desk #3), Therefore, the rest of the chain, i.c., the monitor and the chair are localized wrongly.
identify the main cause of the wrong prediction, whereas, the first anchor
A APPENDIX
Our Appendix contains the following sections:
Identification failure cases as benefit of the interpretabilityy.
Broader Impact and Ethical Considerations.
GPT-generated path influence on the overall grounding accuracy.
Justification for CoT performance when integration into Vi13DRef.
Do we use additional supervision?
Causal Non-Causal COT.
Nr3D Ambiguity.
Related Work.
Explore data-augmentation effect in the limited data scenario (10%).
Pseudo labels human-evaluation.
In-context prompt used for object order extraction.
Examples of generated paraphrases for an utterance.
A.1 IDENTIFICATION OF FAILURE CASES AS BENEFIT OF THE INTERPRETABILITY
To demonstrate interpretability abilities, visualize attention maps shown in Figure 5 In Figure 5) the input description is "the chair to the far left hand side of the taller desk with the computer monitor it." Accordingly, there are anchors mentioned in the description, i.e., desk and monitor, and the target is the chair. The correct chair should be number two, however, the model predicts number four. By visualizing the attention maps, on the left part of Figure [ we can identify the main cause of the wrong prediction, whereas, the first anchor localize wrong desk (desk #3) instead of desk #2. Therefore, the rest of the chain, i.e., the monitor and the chair are localized wrongly. This example also shows the ambiguity in Nr3D, where is hard to say which desk is the taller, desk #2 or desk #3.
A.2 BROADER IMPACT AND ETHICAL CONSIDERATIONS
3D visual grounding holds profound implications across diverse domains and applications, spanning from integrating outdoor systems, e.g., autonomous driving and indoor navigation systems.
Outdoor navigation systems: 3D visual grounding empowers the agents to execute a broad spec- trum of tasks, facilitating assistive systems catering to the needs of elderly individuals and those
15

---

with disabilities-Siie such as the visually impaired. For instance, in the context of visually impaired individuals, the technology's 's ability to ground objects within their 3D environment from language descriptions is pivotal in navigation tasks such determining the nearest exit chair.
Autonomous driving:
In the realm of autonomous driving, a 3D visual grounding system takes on critical role in translating navigational instructions into actionable responses within the dynamic visual landscape. This technology enables seamless communication between the language input, such navigation commands, and the 3D world the autonomous vehicle operates in. For instance, when given a directive like "merge into the right lane at the upcoming intersection where the three pedestrians are walking," " the 3D visual grounding system interprets this command by analyzing the spatial layout, recognizing lanes and pedestrians, and identifying the appropriate merging point. By bridging the gap between language instructions and the complex visual cues inherent in driving scenarios, this system enhances the vehicle's s ability to accurately interpret and execute instructions, contributing to safer and more efficient autonomous driving experiences.
Data-efficiency importance for autonomous driving: Introducing a data-efficient solution for the 3D visual grounding task holds immense importance in the context of autonomous driving, particularly considering the challenges associated with manually labeling vast amounts of 3D point cloud data from lidar sensors. The conventional approach of manually annotating such data is labor- intensive and economically burdensome. By proposing data-efficient solution, our system addresses a critical bottleneck in developing autonomous driving technology. This allows the autonomous vehicle learn and generalize from minimal labeled data, significantly reducing the dependency on large, expensive datasets. The capacity to make accurate 3D visual groundings with limited labeled information not only streamlines the training process but also makes deploying autonomous vehicles more scalable and cost-effective.
Federated learning for indoor navigation systems: Aligned with this imperative, the inherent data-efficient nature of our approach positions it as an ideal candidate for federated learning schemes, which is perfectly aligned with the indoor setup when privacy is required, e.g., the robot operating in personalized environment such as the client home. This becomes particularly relevant when robot introduced into a novel indoor environment and necessitates rapid learning from minimal interactions with its new user without sending the private data of the new environment to the server; thus, learning from limited data is essential.
A.3 GPT-GENERATED PATH INFLUENCE ON THE OVERALL GROUNDING ACCURACY
The GPT--ennrater path is only used as teacher label for training a simple MLP that predicts the path during the inference time. Therefore, GPT is not involved in the system during inference, as we train simple MLP layer, i.e., the Pathway-Hean Head in Figure 3 to predict the logical order. For instance, when the system is deployed on robot, it will be efficient as our simple MLP Pathway-Head is utilized to generate reasonable order based on the input description.
Accordingly, GPT could be seen as design choice to obviate the need for human annotations during training.
In Table 7 the Pseudo and GT labels are for the anchors' localization, not the path. For the path, we always use our MLP-predicted path. However, the accuracy will not be affected if we replace it with the GPT-generated path. As shown in Table 7 we achieve 64.4% and 64.5% using our predicted and GeT-generated paths, respectively. This an intuitive result, as when we assess the performance of the Pathway Head separately, we get 98% accuracy w.r.t the GPT-generated path. Therefore, replacing the Pathway Head generated path by the GPT ones expected not to impact the final performance.
A.4 JUSTIFICATION FOR COT PERFORMANCE WHEN INTEGRATION INTO VIL3DREF
For ViL, our observations, in Figure 2 reveal substantial performance enhancement in the de- manding data-efficient scenario, accompanied by marginal improvements when employing the entire dataset. These modest gains are likely due to nearing the peak attainable accuracy of the dataset, influenced by the limitations posed by annotation errors.
To substantiate hypothesis, we have conducted three experiments:
16

---

Table 7: Ablation study about the GPT- path influence on the overall grounding accuracy.
Method
Grounding Accuracy
MVT
55.1
MVT+ScanEnts
59.3
MVT+PhraseRefer
59.0
MVT+CoT labels)
60.4
MVT+CoT(GT labels)
64,4
MVT+CoT(GT Iabels+GPT Path)
64.5
Table 8: Comparison between causal and non-causal masking in our CoT transformer decoder-based, based on MVT (Huang et al. 2022).
+ Prrettrained P++ +Teacher Loss
ViL
Ours
X
x
59.4
70.3
x
69.5
72.7
72.8
73.6
Replacing our pseudo labels with GT: We replaced our pseudo labels with ground truth annotations (Yuan et al. 2022 Abdelreheem al. (2022a), which is intuitively guaranteed to enhance performance. As shown in Table the substitution our pseudo labels with ground truth annotations yielded performance improvements across the three alternative baselines; LAR (Bakr et al. (022). SAT (Yang et al. 202I). MVT (Huang et al.] 2022), with the exception of ViL.
Deactivation of ViL's Knowledge Distillation Loss (Teacher Loss): We disabled the knowl- edge distillation loss proposed by ViL. This maneuver, shown in Table g while signifi- cantly impairing ViL's performance, exhibited negligible influence on ours (ViL+CoT). Deactivation of P++ pre-training: Consequently, the deactivation of the teacher loss along- side P++ pre-training had a pronounced detrimental effect on ViL's performance, yielding referring accuracy of 59.4%. contrast, our own performance was only slightly affected, maintaining referring accuracy of 70.3%. These findings collectively serve to reinforce our hypothesis that our performance within the ViL architecture effectively limited by an upper bound.
A.5 DO WE USE ADDITIONAL SUPERVISION?
Our main thesis is to show that the design of our CoT framework can be accomplished with notable efficiency, obviating the need for human intervention. Furthermore, we emphasize its demonstrable capacity for substantial enhancement of existing methodologies, particularly within the context of data efficiency, as shown in Figure 2 To this end, we have devised an efficient pscudo-label generator to provide inexpensive guidance improve the learning efficiency of most existing methods, as we demonstrated our experiments in Table4aan and Figure 2 These pseudo labels are not required at test time. Proper design of such approach is pivotal in attaining significant performance gain while circumventing the need for intensive human annotations. A pertinent example can be drawn from the labeling procedure employed in PhraseRefer (Yuan et al. 2022), which demanded a cumulative workforce commitment of 3664 hours, roughly equivalent to an extensive five-month timespan. Hence, our pseudo-labe generation module is delineated into three core components: the anchor parser, the anchor localization module, and the pathway prediction module. Notably, it is essential to highlight that solely the pathway prediction component leverages pre-trained model to generate a ground truth but is not involved during training at inference. More specifically, the anchor parser and localization modules do not contain any learnable parameters nor pre-trained models, they use rule-based heuristics where only two pieces of information are needed: 1- objects' class labels. - objects' position. These two sources of supervision are utilized by all referring architectures, where object class labels used ground truth for the classification head, and object positions are incorporated in the model's pipeline either by concatenation with the objects' point clouds or input to the multi-modal transformer as a positional encoding. Accordingly, both
17

---

Chain-of Chain-af-Thougie Thoughts
Chein-of-Thouee
Fadlimay
Head
boom
Pothway
Head
Recashel
Bockhdrulll
Dooe
Red
What
Brren.
(ooooot a1l
Chhals
Dociahal
20088
Red
Vhile socertet
Figure 6: Comparison between two variants of our Ofain-of-Thoughe framework. On the left the causal version, where first localize the anchors in parallel (shown in green), then feed them to the masked transformer decoder to predict the refined anchors' location in auto-regressive manner. In contrast, the right is the non-causal variant of our CoT framework, where feed the multi modal features F directly to the transformer decoder and remove the causal constrains, i.e., all the anchors attend to each others. In both variants, the decoder outputs the anchors in auto-regressive way and respects the GT logical path for the chain of anchors.
Table 9: Comparison between causal and non-causal masking in our CoT transformer decoder-based, based on MVT (Huang et al. 2022).
Masking Type Nr3D 1 Sr3D
Causal
60.4
73.2
Non-Causal
60.9
73.2
modules do not require any additional supervision. Solely, the anchor pathway prediction module requires pre-trained model to generate the ground truth path, but is not involved during training or inference. The trajectory of reasoning (referred as the Shain-of-Thoughte produced by the pathway prediction module is indispensable within our framework. The omission of the anchor chain would inevitably result in the degradation of our framework, resembling the parallel scenario wherein anchors and the target object are independently detected. Conversely, our framework adopts a sequential, auto-regressive approach, wherein each anchor is predicted consecutively. To show the effectiveness of our proposed method, we compare it against the parallel approach, as shown in Table In summary, solely the pathway prediction component leverages a pre-trained model to generate a ground truth but is not involved during training at inference. Additionally, with zero human efforts, we achieve a significant performance gain on four distinct baselines, as shown in Figure 2
I
A.6 CAUSAL VS. NON-CAUSAL COT
contrast to the conventional teacher-forcing technique, our approach employs a parallel referring head to predict the initial, unordered locations of anchors, depicted in green within Figure 3 This implementation facilitates an investigation into the implications of causal masking, by removing the causality constraints. Essentially, this entails permitting the decoder encompass broader scope by attending to all anchors concurrently, irrespective of their order. However, our CoT decoder still adhere to the correct sequential order for prediction of both anchors and the target object auto- regressively. Figure 6 illustrates the modifications made to the architecture to accommodate the random masking approach. Empirical results, shown Table 9 indicate marginal enhancement in accuracy on the Nr3D dataset-almost 0.5 0p5% relaxation of the causality constraints. However, no discernible improvements materialize on the Sr3D dataset.
A.7 NR3D AMBIGUITY
The challenges posed by Nr3D not solely attributed to its free-form nature; rather, it is compounded by its susceptibility to ambiguity. A visual representation of this issue is provided in Figure showcasing instances of ambiguity where even human annotators encounter difficulties in accurately
18

---

With the counter in front you, it's the 2nd stool from left.
Two chair sitting together and one right side green.
Looking the table with monitors, pick the one on the right.
The leftmost pillow in the set.
Figure 7: A visual representation of Nr3D ambiguity view-dependent issue, where predictions hinge on the speaker's ' viewpoint. For instance, these four samples show that given an input description two options could be considered true based on the speaker's viewpoint. The green and orange arrows represent two different views. The color of the box indicates corresponding view-point.
19

---

localizing objects due to viewpoint-dependent considerations. In such cases, correct predictions hinge on the speaker's 'ss viewpoint, an aspect that often remains undisclosed.
To fortify this standpoint, conducted an in in-depth analysis of the consistency between two recently released manually annotated datasets (Yuan et al. 2022, Abdelrcheem et al. 2022a). Surprisingly, the matching accuracy between these annotations stands at mere 80%, signifying the extent of inherent data ambiguity and the ensuing challenges. In essence, divergent annotations from distinct human annotators due ambiguity impedes the potential for further enhancements.
A.8 ANCHOR LOCALIZATION ACCURACY
For each scene, we have the objects' proposals P = (B, C). where B = {b; M= bounding boxes b; E IRS = L is the class label and L is the number of objects in the scene. Given the extracted objects' names, OT {o: using our syntactic parser, the anchors localization module automatically assign each object's 's name to the corresponding box, A = (B, OT), where B S B and Ā S P. To assess the assignation quality, ask the annotators to assign box for each extracted object's 's name OT from the input utterance, creating the GT set AA = (B, OT), where B S B and A C P. Then, we measure the precision between the two sets, i.e., A and A.
A.9 RELATED WORK
Our novel architecture draws success from several areas, including 3D visual grounding, chain-of- thought reasoning, and scene graph parsers.
3D visual grounding. Existing work can be grouped into one- and two-stage approaches. Single- stage methods depend on detecting objects directly by fusing text features with point-cloud level visual representations and enable grounding multiple objects from a single sentence (Kamath et al.
2021 Luo et al., 2022). Two-stage methods dissociate the object detection task from selecting target among detected objects (Achlioptas et al. 2020 Chen et al.) 2020). by detecting objects during the first stage and then classifying the resulting objects. To benchmark the 3D visual grounding task,
ReferIt3D (Achlioptas et al., 2020) and ScanRefer (Chen et al. 2020) were introduced by collecting textual annotations from the ScanNet dataset (Dai al. 2017a). Early work depended on graph- based approaches /Veličković et al. 2018 Wang et al.) 2019 to infer spatial relations. The object graph is
constructed by connecting each object with its top nearest neighbors (Abdelrcheem al. [002bb] Feng iet al. 2021 Huang et al. [02 Yuan et al. 2021) based on Euclidean distances. Taking advantage of the Transformer's attention mechanism, which naturally suitable for multi- modill features fusion, (He et al.) 2021] Zhao et al., (02T] achieved remarkable performance by applying self-attention and cross-attention on input features. While methods such as BEAUTY-DETR (Jain et al., (002) and (Luo et al. 2022) adopt single-stage approach, most works follow the two-stage approach with pre-detected object proposals. LAR (Bakr et al. 2022). and SAT (Yang et al.) 202T) use multi-modal approach by introducing 2D images of the scenes to aid object grounding Conversely, MVT (Huang
ct al. 2022) focuses on 3D point clouds by extracting multiple views of the objects and aggregating them to improve view robustness make the transformer view-invariant. LanguageRefer (Roh et al. 2022) converts the visual grounding task into a language-based object prediction problem instead of the cross-modal approach. Similarly, NS3D (Hsu et al. (023) focuses on language features by translating language into hierarchical programs. EDA (Wu et al. (202)) explicitly separates textual attributes and performs dense alignment between language and point cloud objects. ViL3DRel (Chen) et al. 2022) includes a spatial self-attentio layer that considers the relative distances and orientations between objects in 3D point clouds.
Chain of thoughts. In machine learning, chain of thought refers to the series of intermediate steps that a model takes to arrive at its final decision or prediction. The chain-of-thought concept has been used in many different machine learning applications, including natural language processing (Weil ctt al. 2022: Chowdhery et al. 2022 Lyu et al. 2023 Wang et al. 2022 Zhang et al. 2023 Madaan & Yazdanbakhsh 2022), and robotics (Jia et al. 2023 Yang et al. 2022). In natural language processing, of thouugt prompting adds intermediate reasoning steps to the output of LLMs, which leads to increased reasoning capabilities (Zhao et al.) 2023 Wei et al. 2022). In robotics, understanding the chain of thought can help to identify which sensory inputs were used to arrive at a particular action or decision. Moreover, (Tia al. 2023) divides the primary goal into subgoals to tackle the complex
20

---

Table 10: Ablation study for different components of our CoT3DRef framework, such as different augmentation techniques and distractor loss. All the experiments are conducted on 10% of Nr3D dataset.
+Sene
+Point
+Lang. +Ref.
Ref. Acc.
Aug.
Aug.
Aug.
Aug.
27.6
(b)
37.5 /
34.5
(d)
/
34.0
/
/
35.5
/
37.5
sequential decision- makiing problems using the Hierarchical RL (HRL) (Hutsbbaut-Buyse et al. 2022).
Scene graph parser. Extracting scene graphs a fundamental task in NLP and multimodal vision.
This problem depends extracting directed graphs, where nodes objects and edges represent
relationships between objects (Lou et al., 2022) Xu et al.) 2017]] Wu et al. (019) Jiang et al. 2016
Yang et al. 2018 SSchuster al. 2015). Scene graphs have been used in scene understanding to
enhance image captioning and predicting pairwise relations between detected objects (Li et al. 201 7b
Dai al. 2017b Li al. 2017a). Other works have leveraged graphs enhance text-to-image
generation models by learning the relations between the objects in the sentence (Feng et al. 2023
Johnson al. 2018). Existing aff-the-shelf tools (Schuster et al.) (015)) can parse the language
description grammatically into scene graph with dependency tree (Wu et al. 2022).
A. EXPLORE DATA-AUGMENTATION EFFECT
LIMITED DATA SCENARIO (10%)
Visual-based Augmentation. Two visual augmentations are explored at the scene and point cloud levels. At the scene level, we randomly shuffle every object presented in the scene with another object of the same class from one the ScanNet scenes. At the point cloud level, we apply a small random isotropic Gaussian noise, rotate the points along each axis, and randomly flip the x and axes. In addition, we apply a small random RGB translation over color features. These visual augmentations used to enhance the robustness and generalization capabilities of our architecture.
Language-based Augmentation. We utilize a T5 (Raffel et al. 2019) transformer model fine-tuned paraphrase datasets to augment the input utterances. To ensure the quality of the generated paraphrases, we use the Parrot library (Damodaran) 2021 to filter them based on a fluency and adequacy score. This aims to enhance the naturalness and diversity of the generated utterances. We showcase an example of an input utterance alongside its corresponding generated paraphrases in Table
Referring-based Augmentation. The maximum number of objects in the sentence M is and for Nr3D and Sr3D, respectively. However, in 52% and 98.5% of Nr3D and Sr3D, respectively. two objects only are mentioned in the utterance; the target and one anchor. This motivates us to apply simple referring-baseee augmentation by swapping the target and the anchor while keeping the same semantic information. For instance, if the relation between two objects is "near to"", it should be the same after swapping. However, if it was "on the right of". should be swapped to "on the left of".
Results. We have explored several augmentation techniques to enhance architecture's robustness and generalization capabilities. However, we report the results on only 10% the data as we notice no gain is achieved when leverage more training data, e.g., 40%. As shown in Table 10 the language and referring augmentations (rows b and f) increase the performance slightly. However, the two visual-basel augmentations (rows c, d, and e) do not give any performance gain.
A.11 PSEUDO LABELS
The accuracy of the pseudo-labed labels plays a vital role in the overall performance. To evaluate pseudo- labels, manually collect ground-truth labels for:
21

---

[CoT 3D VG] Objects Info
Requeter: Valen Language
Revert: S0.6
Taats evailblle: 4002
Darator:: Hoon
Gualifiations Required: Acpotul Rain Reqwener It goatent then aquits
Number of Approvd 5000, Master
Instructions:
Start the task:
The sentence is: "The file cabinet on the left, when facing the windows."
object targed) arbitrary extract logical order objects reach the target
The target file cabinet
object.
other words, u need to find the target the scone with the help of
other mentiored ed object the sentence. Bo which object should
Objject
start with then whate the next ons, etc, sill reaching the target. Of course the order of the target will be the last one always.
Objjct
Order
Ex.1:
Glven the tollowing input sentance:
"The pillow on the couch that is on the right hand side of the when you facing the TV. and closest to the TV"
Object
Order
go your target you fraily can reach to the pillow.
try
objects
Object
Order
face it,
go
Therefora tha cutout bo:
Object
Order
pillow 3
couch 2 TV
Dijject
Order
Figure 8: Our User-Interface (UT), designed on Amazon- Mechaniialh-----ee-eaeaa (AMT) Crowston, 2012). We detailed the instructions on the left, and the task is mentioned the right. The annotators are asked to extract the mentioned objects the input sentence chronological order, then assign each object logical order to reach the end goal, the target. We demonstrated the instructions on the left for each sample for better performance.
IC01 Obarcs mealO'
Language
equalta
Namber of Alpooot graker tus 2000 Marars granted
Instructions:
Start the task:
facing
Given sentence, that describe
extract
objects
spatial relationship. spatial means any
Object
Relation
Subject 1
Extra Subject
such on front above
Also, the varha that indicats the spatial relation abould be extracted such us, hanged, blocking. touching, etc, Because all these verbe will help us locate the objects, is considered as spatial relation.
Object 2
Reletion
Subject 2
Extra Subject
Object
Relation
Subject 3
Extra Subject
1x
following imput
"The pillow on the couch that is on the right hand side of room, when you are facing the TV, and closest to the TV" Thenefore the curqut will be:
Object
Subject
Extra Subject
pillows couch pillow.
Object
Subject
2
"The cabinet
when facing the windows",
output will be:
Object
Reletion
Subject
Extra
file cabinet, on the left, windows
Bloooo
cometns ansiatinn Rssan
Calen Dudiiion
Figure 9; Our User- Interface (UI), designed on Amazon Mecionn-eeaanial----e Turk (AMT) Crowston, 2012). We detailed the instructions on the left, and the task is mentioned the right. The annotators are asked to chronologically extract the mentioned objects in the input sentence and their subject and relation triplet format. We demonstrated the instructions on the left for each sample for better performance.
22

---

scere0628.00 00 (18/75)
scine0265 90 (1/11)
scene05e6_D0
scan05555_00 (02266)
soene0262_00 (0/51)
soeme0184_00 (0166)
5oam00074_00 (0/13)
scene0138_0v
(0/115)
scane0147_00 (0/64)
soeme0286_00 (0/100)
4cHr#0462_00 (1/16)
The intro screen of our web-based Interfane (UI) for anchors localization task. On the intro page, we list the available scenes and the corresponding number of sentences for each scene in parentheses. More specifically, for each scene, demonstrate the number of annotated sentences divided by the total number of sentences in that scene.
Figure 10:
Nr3D Browser
All-Scenes
Project Page
Sentences
The plant at the far right hand side the bookcase tucked in the furthest corner the desk (TO DO)
Facing please
The plant directy nest the long bookshelf that takes over well (DONE)
booksheif,
bookshelf
the tall plant near the window (DONE)
The that the
Our web- base particular scene from the intro page, the 3D interactive window will be displayed the top, and corresponding sentences will be displayed below it. To facilitate the annotators' task, we mark the annotated sentences in green and the remaining yellow.
Figure 11:
(UI) for anchors localization tasks. After clicking
23

---

Nr30
All-Scanes
the tall plant near the window
Done
Our web-based User-Interface (UI) for anchors localization tasks. After clicking specific sentence, the 3D interactive window will displayed at the top, and the corresponding sentence will be displayed below it. To facilitate the annotators' task, we mark the anchors' words in a clickable yellow box. Once a box is clicked, we show the corresponding boxes with the same class label as the clicked word. Finally, the annotator's task is to pick the correct box out of the colored boxes. In this example, we have a unique box for the word window; therefore, only one box shown.
Figure 12:
Nr3D Browser
AllScceees
Project Page
Book above the blue
Cchair
in the middle
Done
Our web-hased User-Interface (UI) for anchors localization tasks. After clicking on particular sentence, the 3D interactive window will be displayed the top, and the corresponding sentence will be displayed below it. To facilitate the annotators' task, we mark the anchors' words in clickable yellow box. Once a box is clicked, we show the corresponding boxes with the same class label as the clicked word. Finally, the annotator's 's task is to pick the correct box out of the colored boxes. In this example, we have nine boxes for the word chair. The selected box by the annotator marked in green, while the other distractors are marked in red.
Figure 13:
24

---

1. The predicted orderings of the anchors in the utterance. Figures 8 and 9 depict our User- Interface (UD) designed on order and relations collecting. We detailed the instructions the left, and the task is mentioned on the right. The annotators are asked to extract the mentioned objects in the input sentence in chronological order and their spatial relationship, then assign each object a logical order to reach the end goal, the target. We demonstrated the instructions on the left for each sample for better performance. To evaluate orderings predicted by in-context- learning, we use the normalized Levenshtein edit distance between two sequences, where a distance of I means that every object the sequence is incorrect. We achieve an average distance of 0.18 18 between predicted and ground-truth orderings.
(AMT) (Crowston 2012) for objects'
2.
The final localization of the anchors as predicted by the geometry module, as demonstrated To evaluate the geometry module's is accuracy, we consider anchor-wist wise localization accuracy. Therefore, we design a web-based interface to collect ground-truth data for 10% of Nr3D dataset. At the entry screen, Depicted in Figure TO we list the available scenes and the corresponding number of sentences for each scene in parentheses. More specifically, for each scene, demonstrate the number of annotated sentences divided by the total number of sentences in that scene. To facilitate the annotators' task, we mark the annotated sentences in green and the remaining in yellow. Afterwards, when the annotator clicks on a particular scene, the 3D interactive window will be displayed at the top. and corresponding sentences will be displayed below it, as shown in Figure [I]] We mark the anchors' words in clickable yellow box. Once box is clicked, we show the corresponding boxes with the same class label as the clicked word. Finally, the annotator's task is to pick the correct box out of the colored boxes. For instance in Figure [2] we have a unique box for the word window; therefore, only one box is shown. In contrast, Figure [3 shows more challenging sample, where we have nine boxes for the word chair. The selected box by the annotator is marked in green, while the other distractors are marked in red.
in Figures TO 11 12 and 13
A.12 IN-CONTEXT PROMPT USED FOR OBJECT ORDER EXTRACTION
We provide below the abridged prompt we use extract the logical order of objects to follow to find the target object, following a given input utterance. We find empirically that 1) providing a large number of examples, 2) mentioning reasoning explanations, 3) asking to first provide the list of mentioned objects, and 4) repeating instructions at the end of the prompt increases the quality of the predicted object orderings produced by GPT3.5.
Given a set of instructions to locate a larget object in room. your task is to extract the order objects mentioned in each instruction. First list all the unordered mentioned objects in the seutence using the format: "L: [objoct_ J. object_ 2, object_31 33", Then
objects using the format [object onder]" mark target object with "t" Only include names Additionally, please exclude any additional information or descriptors such as color or size adjectives. The target the subject of the sentence.
objects.
Q: "The pillow on the couch that is on the right hand side of the room, when you are facing the TV, and is closest to the TV" T: [TV, couch, pillow]
R: [l: TV, 2: couch, t. pillow]
(explanation:
first need find the TV orient ourselves,
then find the couch on the right hand side of the room, and then the pillow on the couch. Do not mention the explamations in the future.)
Q: "The pillow the bed that is the far end of the room and the reur and right hand side of the bed'
L: [pillow, bod)
R: [1: bed, pillow]
end of the looking towards the pillows, choose pillow L: [bed, pillow]
Q: "Standing
the front, smaller more right."
pillow|
Q: "There are two groups of kitchen cabincts; three along the wall with the stove, and two on the opposite wall.
Looking the wall opposite the stove, the with the refrigerator, please select the kitchen cahinet closest the refrigerator."
L: [kitchen cabinet, wall, stove, refrigerator] I...
Do not mention explamations. Only include objects the list.
Do not add any text beyond the reply. The target is the subject of the sentence. Mention all relevant objects in the reply. Don't I use color size adjectives.
Please exclude any color adjectives your response. Avoid using any descriptive related to color your response. Query:
13 EXAMPLE OF GENERATED PARAPHRASEE ASES
In Table below, we provide an example of an input utterance sampled from Nr3D and its associated generated paraphrases.
25

---

Table 11: Example of an input utterance from Nr3D and its associated paraphrases.
Input Utterance
When entering the room, look on the nighistand to the left of the hed headboard, you will find some books.
Paraphrase
You'll find few books nightstand just the left of the headhoard.
Paraphrase
You'll 'll find few books nightstand just the left of the bed headboard.
Paraphrase
Look the nightstand to the left of the headboard of the bed, and will find some books.
Paraphrase 4
When enter the room, look at the nightstand to the left of the headboard of the bed.
26


Reviews:
[{"review_id": "s7RAp6gowM", "reviewer": "Reviewer_hPTL", "summary": "This submission studies 3D object referring. A new architecture is proposed, which predicts several objects sequentially instead of a single object at last. To generate a dataset that makes this new task setting possbile, the authors exploit grammar parser, GPT in-context logical ordering and rule-based box matching. The new architecture boils down to a decoder that matches several different encoder. Experiments show that the method improves several baselines and notably work quite well under a data efficient setting.", "questions": "None", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ A new decoder head for sequential object grounding in a scene. This is a new paradigm that may inspired related fields.\n+ A data curation effort that makes sense to me. This makes the CoT training possible.\n+ Quantitative results showing positive margins on four baselines and notably good performance in a data-efficient setting.", "weaknesses": "I like the paper and don't see major weaknesses. Why I am not rating higher is the fact that the new thing comes from foundation models, which is in fact a system-level contribution instead of a principled contribution. But I feel it fine and timely to accept this in ICLR 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UBGBjrYrJx", "reviewer": "Reviewer_ywux", "summary": "This paper presents CoT3DRef, a novel and interpretable 3D visual grounding framework that aims to mimic the human perception system. The authors formulate the 3D visual grounding problem as a sequence-to-sequence task, predicting a chain of anchors leading up to the final target, which is a significant deviation from existing methods that directly localize the referred object. This approach not only improves the overall performance but also enhances interpretability, helping to identify and address failure cases.", "questions": "Can you provide a detailed analysis or examples of the identification of failure cases as a benefit of the interpretability?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Technical Novelty.** The chain-of-thoughts (CoT) approach is an innovative way of addressing the 3D visual grounding problem, providing a sequence of interpretable intermediate steps that lead to the final prediction. This approach is aligned with how humans might approach the task, thereby making the model’s predictions more understandable and transparent.\n\n2. **Data Efficiency.** The proposed framework demonstrates significant data efficiency, especially highlighted by its performance on the Sr3D dataset where it matches state-of-the-art (SOTA) performance with just 10% of the training data. This is a crucial advantage, particularly in domains where acquiring labeled data is expensive and time-consuming.\n\n3. **Applicability.** The authors show that CoT3DRef can be easily integrated into various existing architectures, demonstrating its versatility and applicability. This is substantiated by the comprehensive experiments conducted across different benchmarks and architectures, consistently showing performance gains.\n\n4. **Comprehensive Experiments.** The paper includes extensive experiments and ablation studies, providing a thorough evaluation of the proposed method. The results on Nr3D, Sr3D, and ScanRefer benchmarks are impressive, showcasing the framework's effectiveness and robustness.", "weaknesses": "1. **Detailed Failure Case Analysis.** While the paper mentions the identification of failure cases as a benefit of the interpretability, it does not provide a detailed analysis or examples of these cases. Including such an analysis could provide valuable insights into the limitations of the model and areas for future improvement.\n\n2. **Broader Impact and Ethical Considerations.** The paper could discuss the broader impacts and potential ethical considerations of deploying such a system, especially in the highlighted application areas like autonomous driving and robotics.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4CBCiy2uq6", "reviewer": "Reviewer_6KUg", "summary": "This paper proposed to model the referring task into a seq2seq task. The authors introduce two new modules: “target anchors head” and “pathway head”.  Basically, the main idea behind is to force the model not only to predict the target object but also pay attention to the rest object mentioned in the utterance. The “pathway head” forces the model to learn the logical order information that indicates how the mentioned target object is identified. However this module requires extra annotation, which they obtain from prompting the Language Language Models (LLMs). Benefit from extra clues, they improve the grounding performance on Various grounding datasets.", "questions": "I have listed my question above.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strengths:\n1. This paper introduces not only just predict the target object, but also to force the model to predict all mentioned objects (anchors) to best leverage all available clues. Which provides more information for models to use. \n2. The proposed pathway generation method leverages the ability of powerful LLMs (GPT), freeing the burden of human annotators and making the system easy to scale up.\n3. The proposed pipeline not only improve the performance compare with previous SOTA, but also has higher data efficiently when training data is limited. This is valuable since 3D visual language data is not always easy to obtain.", "weaknesses": "Weaknesses:\n1. The proposed idea which pays attention to not just the target object, has been proposed before in “PhraseRefer”. They use human annotators to obtain fine grained annotation. Which share the same underlying idea with this paper. \nTo me one of the major novelties of this paper is that they explicitly model the logical order while the “PhaseRefer '' uses implicit manner. I think these two works are highly related, but the author didn’t give enough introduction of this “PhraseRefer”, which makes it a bit difficult to understand the performance comparison e.g table 4. SAT with PhraseRefer/ScanEnts and yours. Besides, I believe some missing number in this table.4 e.g. MVT with PhaseRefer.\nThe authors should give more clear comparison with this line of works that use fine grain annotation.\n\n2. Do all numbers in Table 4 use a GPT-generated path? If not, answer my below question:\nIf you have access to the GT anchor label either from “PhaseRefer” or “ScanEnts”, but you still require GPT to give you extra information “the Path”, how is the performance going in this case?\n\n\n3. In Sec.3.3 Anchors parser “Then, we match the objects to their closest matching class from the ScanNet labels”, Would you explain this in more detail? Correct me if I am wrong: I assume if you assign a more fine grain label set , like ScanNet200, you are actually providing richer semantic knowledge to the model. My question is how do you assign the label, do you use the same level of semantic information with the others?\n\n4. From the numbers of MVT in Table.4 and Table.5, it seems that when MVT baseline has access to more data, the performance can improve quite a lot, So it is possible that your proposed method introduce explicit knowledge to the model while MVT can has a chain to learn that out from the data if the dataset is large enough?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yVbsNJyEHN", "reviewer": "Reviewer_kjGR", "summary": "The paper proposes an interpretable 3D visual grounding framework based on Chain-of-Thought in large language models, i.e., CoT3DReF.\nIt is a sequence-to-sequence (Seq2Seq) model by first predicting a chain of anchors and then the final target.\nTo provide supervision on the intermediate chaining, the paper devises a pseudo-label generator.\nThe proposed framework achieves state-of-the-art performance and shows its data efficiency on several 3D visual grounding benchmarks.", "questions": "See \"Weaknesses\".", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper integrate the idea of Chain-of-Thoughts into the 3D groudning tasks, and propose a 3D data-efficient and interpretable  framework CoT3DRef.\n2. The technical contribution is sound, including Seq-2-Seq formulation and pseudo labels generation.\n3. The proposed framework achieves state-of-the-art performance on 3D visual grounding tasks and shows its data efficiency.", "weaknesses": "In general, I am in favor of the paper, which makes an interesting first attempt toward CoT 3D grounding tasks. However, there are still some important while unexplored problems.\n\n1. To make CoT3DRef self-contained and scalable, it is important to generate accurate pesudo labels. Although it has been discussed in Sec.5, I have some remaining questions/concerns:\n- In Alg.1, FIND function maps a localized object, a relation and a set of candidate objects to an output localized object. What is the detailed process of FIND? Is the relations between objects known?\n- Alg.1 receive the input of object proposals P. Is it the output from the 3D object detection model which requires extra training? If so, the anchor localization may be bottlenecked by the object proposals.\n- Considering the elaborate process of psuedo label generation, the propose frameworks seems not \"data efficient\" in data preparation, though it is data efficient in 3D grounding training.\n\n2. The interpretability is limited. From the input end, GPT-3.5 is used for extract the logical order of objects given an input utterance, thus adding interpretability. But the logical relations of the objects is not explicit modeled. Instead, it is implicit learned in the Seq-2-Seq network.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)