ARKitSceneRefer: Text-based Localization of Small Objects in Diverse Real-World 3D Indoor Scenes
Authors: Shunya Kato
Shuhei Kurita
Chenhui Chu
Sadao Kurohashi
Venue: EMNLP 2023 Findings (2023)
Keywords: 3D
Dataset
Visual Grounding
Referring Expression Comprehension

Abstract: 3D referring expression comprehension is a task to ground text representations onto objects in 3D scenes. It is a crucial task for indoor household robots or augmented reality devices to localize objects referred to in user instructions. However, existing indoor 3D referring expression comprehension datasets typically cover larger object classes that are easy to localize, such as chairs, tables, or doors, and often overlook small objects, such as cooking tools or office supplies. Based on the recently proposed diverse and high-resolution 3D scene dataset of ARKitScenes, we construct the ARKitSceneRefer dataset focusing on small daily-use objects that frequently appear in real-world indoor scenes.
ARKitSceneRefer contains 15k objects of 1,605 indoor scenes, which are significantly larger than those of the existing 3D referring datasets, and covers diverse object classes of 583 from the LVIS dataset.
In empirical experiments with both 2D and 3D state-of-the-art referring expression comprehension models, we observed the task difficulty of the localization in the diverse small object classes.

ARKitSceneRefer:
Text-based Localization of Small Objects in Diverse Real-World 3D Indoor Scenes
Shunya Kato
Shuhei Kurita²
Chenhui Chu'
Sadao Kurohashi!
Kyoto University RIKEN {s-kato, chu, kurojenlp.ist. kyo.o-u.ac.........
shuhei kuri couritarriken
Abstract
3D Scene
3D referring expression comprehension is task to ground text representations onto objects in 3D scenes. It is crucial task for indoor household robots or augmented reality devices localize objects referred in user instruc- tions. However, existing indoor 3D referring expression comprchension datasets typically cover larger object classes that are easy to local- ize, such as chairs, tables, or dlorrs, and often overlook small objects, such as cooking tools or office supplies. Based the recently proposed diverse and high-resolution 3D scene dataset of ARKitScenes, we construct the ARKitSceneRe- fer dataset focusing small daily-use objects that frequently appear in real-world indoor scenes. RRiiSSceneRefer contains 15k objects of 1, 605 indoor scenes, which are significantly larger than those of the existing 3D referring datasets, and covers diverse object classes of 583 from the LVIS dataset. In empirical ex- periments with both 2D and 3D state-of-te the- referring expression comprehension models, we observed the task difficulty of the localiza- tion in the diverse small object classes. ARK- itSeeneRefer dataset is available https: //gi thub. com/ com/ku- wwt.k////////tteeeeeeeeeeeeeeeee
Close up
2D Image
Query: blue teapot top of a black colored counter near light blue pitcher
Figure An example of the ARKitSceneRefer dataset. We present the whole 3D scene, zoomed 3D scene, and 2D image zoomed for local structure.
robots, robots are expected to find cooking tools, foods, and ingredients. Similarly, to develop au- tonomous laundry robots, they need to find small objects such as clothes. We assume that finding these small objects in 3D scenes is meaningful for household robots, although these objects are dif- ficult to be captured and often overlooked from the existing real-world scan-based 3D scene REC datasets.
1 Introduction
3D referring expression comprehension (REC) is essential task of understanding 3D scenes and localizing objects in scenes into easy-to-interpret text representations. It has numerous applications, such as robotics and augmented reality. Recently, sophisticated datasets have been proposed for this purpose (Chen et al., 2020; Wald et al., 2019; Qi et al., 2020). These datasets are based on object segmentations in 3D scenes and cover relatively large objects, such as furniture in indoor scenes. However, when we develop robots that follow instructions and perform indoor household tasks, such robots are expected to find out and localize typically small objects that are required for house- hold tasks. For example, for developing cooking
In this study, we propose a new REC dataset named ARKitSceneRefer in 3D scenes in that we concentrate on the fineness and diversity of the objects referred to in instructions. ARKitSceneRe- fer is based on the recently proposed ARKitScenes dataset. ARKitScenes (Baruch et al., 2021) is a fine- grained photo-realistic 3D scan for diverse 1, 661 venues and 5, 047 scenes. Based on ARKitScenes. extract small objects that are covered in the previous 3D scene datasets well. We first apply the 2D object detector Detic (Zhou al., 2022) with LVIS (Gupta et al., 2019) object classes for the video frames (i.e., 2D images) from which 3D

---

scenes in ARKitScenes are constructed. Next, we extract object labels and positions and map them to 3D scenes with ray-casting and clustering by DB- SCAN (Ester et al., 1996). We confirm that most small objects are detected 3D scenes with this approach. We then annotate referring expressions that people use to locate the objects via Amazon Mechanical Turk, while manually revising the in- correctly detected object labels. Figure/ 1 shows an example of our ARKitScenRRefer dataset. We finally collect 1, 605 scenes with 583 object classes and more than 15k objects and their corresponding referring expressions.
1, 661 unique indoor environments and provides high-quality depth maps and 3D-oriented bounding boxes.
Based on these 3D indoor-scene datasets, several language-related 3D scene understanding datasets have been proposed. For 3D visual grounding or 3D REC, ScanRefer (Chen et al., 2020) and ReferIt3D (Achlioptas et al., 2020) have been pro- posed. These datasets are based on ScanNet and annotated with referring expressions for objects in 3D scenes. They are also used for the 3D dense captioning task. Similarly, the 3D question an- swering dataset ScanQA (Azuma al., 2022) was proposed based on ScanNet. Yuan et al. (2022) extended 3D visual grounding to 3D phrase-aware grounding with phrase-level annotations from ex- isting 3D visual grounding datasets (Chen et al., 2020; Achlioptas et al., 2020). Qi et al. (2020) annotated language instructions based on Matter- port3D (Chang et al., 2017) and proposed remote embodied visual referring expression in real 3D indoor environments. Xu al. (2022) proposed a large-scale 3D synthetic indoor dataset TO-Scene focusing on tabletop scenes. Unlike these datasets, our dataset focuses on a broader category of in- door small objects in real-world 3D scenes. Our dataset is more challenging because small objects are harder to recognize.
In addition, we conduct experiments with both 2D and 3D models to localize objects in ARK- RtSceneRefer. Our 2D models based the state-of-the-art 2D REC models MDETR (Kamath al., 2021) and OFA (Wang et al., 2022). Our 3D models are based on an adaptation of the state- of-the-art 3D REC models ScanRefer (Chen et al., 2020) and TDVG-Transformer (Zhao et al., 2021) on our dataset.
Our contributions are follows: (i) creating the first object localization dataset concentrating on the small objects in daily indoor scenes upon the high-resolution 3D scene dataset of ARKitScenes, (ii) attaching more than 15k referring expressions with human annotations with a significantly large number of object classes, and (iii) comparisons with the ttatc-of-the-art 2D and 3D REC models on ARKitSceneRefer.
2.2 Referring Expression Comprehension
REC is the task of localizing target object corresponding a referring expression. In 2D REC (Kazemzadch et al., 2014; Plummer et al., 2015; Yu et al., 2016; Mao et al., 2016), mod- els find the target object region specified by tex- tual referring expression in an image. Deng et al. (2021) use images with bounding boxes and queries for supervised REC. TransVG (Deng al., 2021) is a transformer-based framework for 2D visual grounding, outperforming existing one-stage and two-stage methods. These fully supervised REC, However, depends on large annotated datasets. Weakly supervised methods (Liu et al., 2019; Sun et al., 2021) don't require manually annotated bounding boxes and unsupervised methods (Jiang et al., 2022) that require neither manually anno- tated bounding boxes nor queries have also been studied. Pscudo-Q (Jiang et al., 2022) proposed method for generating pseudo queries with ob- jects, attributes, and spatial relationships as key components, outperforming the weakly supervised
2 Related Work
2.1 3D and Language
Recently, several photorealistic 3D indoor scene datasets (Nathan Silberman and Fergus, 2012; Song al., 2017: Dai et al., 2017; Wald et al.. 2019; Straub et al., 2019; Ramakrishnan et al., 2021: Rozenberszki al., 2022) have been con- structed. ScanNet (Dai et al., 2017) consists of 1,513 RGB-D scans of 707 unique indoor envi- ronments with estimated camera parameters and semantic segmentation. 3RScan (Wald et al., 2019) consists of 1, 482 RGB-D scans of 478 environ- ments across multiple time steps, including ob- jects whose positions change over time and notations of object instances and 6DoF mappings. ARKitScenes (Baruch et al., 2021) is the recently proposed high-resolution 3D scene dataset based on Apple's LiDER scanner. ARKitScenes con- sists of 5, 047 high-resolution RGB-D seans of

---

ARKitScenes
Object Detection
Projection of 2D Objects 3D
palsan
Bounding Box Selection
Class Correction
Choose "yes" the object is mostly covered by the bounding box, and the bounding box should almost fit the object. Otherwise, "no". yes no
Choose the best suitable class for this object. When you are sure that the present class is correct, correct. airplane
aerosol,
Referring Expression
Write relationships between target object and nearby object identifing it. corner room window
205: teapot center green
Object Selection
Annotation
Figure 2: Overview of our dataset construction pipeline. Green boxes in the 3D scene represent 3D bounding boxes.
methods.
key point selection. Other approaches (Chen et al., 2022; Jain al., 2022: Wu al., 2022) have been proposed for further improvements of the matching. Net (Chen et al., 2022) unified dense captioning and REC a self-critical manner. In this study, we adapt two-stage models for the proposed task.
Recently, pre-training large vision-and- language datasets become popular in image- understanding tasks. Many existing 2D REC meth- ods (Li* et al., 2022; Yang et al., 2022: Subra- manian et al., 2022; Kamath et al., 2021; Wang et ul., 2022) relied on some pre-trained models. MDETR (Kamath et al., 2021) is an end-to-end text-modulated detector derived from DETR (Car- ion et al., 2020) and achieved good performances on scene understanding tasks. OFA (Wang al., 2022) a unified soueenee-t--sqqeeeee pre-trained model that unifies multiple multi-modal tasks such as image captioning, VQA, visual ground- ing, and text-to-image generation. OFA achieved state-of-the-ar performances on several vision-and- language tasks, including the REC. In addition to 2D REC, Video-REC (Li et al., 2017; Chen et al., 2019) become major task. First-person vision REC RefEgo (Kurita al., 2023) shares the sim- ilar difficulties with 3D REC problems. Both OFA and MDETR are utilized in Kurita al. (2023).
Dataset
We describe the methods to construct the ARK- itSceneRefer dataset in this section.
3.1 Data Collection
We construct the ARKitSceneReffr dataset based on ARKitScenes (Baruch et al., 2021), which is a large-scale 3D indoor scene dataset. ARKitScenes has comparably higher resolutions in 3D scenes, and thus it is suitable for task that targets small object localization. Note that, in 3D scenes, some small objects often become unclear and difficult recognize. However, most them can clearly detected and classified corresponding 2D images. The performance of object detection in 2D images has been improved significantly, making it possible to find small objects in 2D images. Therefore, in this study, detect target objects in the video frames (i.e., 2D images) where the 3D scene is constructed and then localize them in 3D scenes. Figure 2 shows our dataset construction pipeline. In the following subsections, we describe each step in detail.
Compared to 2D and video REC, 3D REC is an emerging task. Two-stage (Chen et al., 2020; Zhao et al., 2021; Yuan et al., 2021) and single- stage (Luo et al., 2022) methods have been pro- posed for 3D REC. Two-stage methods generate object proposals and then match them with the query. These methods have the disadvantage that they don't take the query into account when gen- crating object proposals. To address this disadvan- tage, single-stage methods conduct language-awas aware

---

3.1.1 Object Detection
of the bounding boxes as the "ray." By doing so, we obtain multiple ray-mesh intersections for each scene. We then make clusters by DBSCAN (Ester et al., 1996) to summarize the intersections and create object points in 3D space. For clustering, we make clusters for the same class-label objects. We also impose threshold of 0.05m of distance for making single cluster to DBSCAN to keep the resolution of small objects in scenes. As a sult of clustering, we obtain 68. 68.25 clusters in a single scene on average on the validation set. Note that we didn't It use the clusters that consist of fewer numbers of intersections for further analyses and annotations.
In this step, we detect objects in video frames from which the 3D scene is constructed. The results of object detection are used to select target objects in the dataset. Detic (Zhou et al., 2022) is used as the object detection model. Detic is trained both object detection and image classification datasets. With that, Detic expands the vocabulary of object detection, reduces the performance gap between rare classes and all classes in standard LVIS (Gupta et al., 2019) benchmarks, and achieves state-of-the- art performance. Detic provides more detailed class information than conventional models trained on MSCOCO (Chen et al., 2015), which is helpful for the next class selection step, and the instance seg- mentation corresponding each 2D bounding box, which is helpful for obtaining 3D bounding boxes. We used pre-trained Detic in LVIS, MSCOCO, and ImageNet-21K: Because the same object appears in chronologically close video frames, it is unnec- essary to perform object detection on all frames. Therefore, we conduct object detection only for 1/10 frames uniformly sampled from all frames.
Finally, we use 2D instance segmentations in the images corresponding to each 3D object point to assign 3D bounding boxes. Similar to obtaining 3D object points, we use ray-casting for each pixel of the instance segmentation to project it onto the 3D scene. To reduce computational complexity, ray casting is only conducted for pixels whose posi- tions are divisible by 5. To eliminate noise, the top 5% and bottom 5% of the projected instance seg- mentation coordinates are removed, and the mini- mum and maximum coordinates within that range are considered as the 3D bounding box. Note that the 3D bounding boxes are assumed to be parallel to each axis because an object rotation is not taken into account in ARKitScencRefer.
3.1.2 Projection of 2D Objects to 3D
After we obtain the bounding box of objects de- tected by Detic, we project the position of the ob- ject point in video frames into the world coordi- nate of the 3D space using the provided intrinsic and extrinsic parameters of ARKitScenes. We first project the camera position for each video frame into the world coordinate with the extrinsic param- eters. The position of the detected objects is then projected by the intrinsic parameters of the camera. Here, two problems exist: the distance between the camera and the object remains unknown, and the projections of the same object from multiple video frames don't 't always converge cause of the noise of the projection parameters. is also important to isolate bounding boxes for differ- ent objects because the Detic results often contain multiple same-class-label objects in a video frame. Therefore, we apply the ray-casting and simple clustering based approach for summarizing multi- ple projections for a single object in the 3D scene. We use the ray-casting from the camera point the ARKitScenes mesh to obtain the first intersection of the mesh and the ray to the target object. Here we use the line from the camera point to the center
3.1.3 Object Selection
The object selection step consists of three sub-steps: class selection, scene selection, and target object selection. Firstly, as large objects and small objects are mixed in the results of object detection, we conduct class selection to select small objects. We choose the class of small objects based on the cri- teria that they can be grasped with both hands (e.g.. coffee maker, laptop computer, and microwave oven). As result of object selection, the number of object classes decreases from the Detic object detection results 1, 151 to 794. Next, we con- duct scene selection to guarantee the diversity of 3D scenes. We select one scene per room. In the same room, the scene with the largest number of objects and more than 20 objects is selected. As a result, the number of scenes decreases from 5, 047 to 1, 615. Finally, tion to decide on the objects be annotated. The number of target objects is set to 20 for each 3D scene. In order to increase the number of objects in rarely detected classes, instead of choosing tar-
single point be-
conduct target object selec-
'https://aa //dl. fbatp:: icf.les.ccc.......c...c LcOco121k_CLLB Swwnn__SS.B____ (ee__899b322_20 4x_ 4x_ft4x Ft4x_max-xattt wewxx_xxxsszzeeepe pth
www.iiee...o/eeee//eeeeeeeeeeeee com/detic/Detic

---

Objects Average Volumes Major Classes (ARKitScenellefer)
are asked to annotate referring expressions for the target objects; the referring expression for a target object must clear enough to distinguish the tar- get object from other objects. One task contains five target objects. Workers are restricted to coun- tries of USA, Canada, UK, and Australia. Workers are also asked to choose 3D bounding boxes in terms of whether their position is almost correct as 3D bounding boxes sometimes become noisy. As a result, workers answered 48. 15%1 of the 3D bounding boxes as correct. The wrong bounding boxes are excluded from the final dataset. Further- more, as the class obtained from Detic is sometimes wrong, the workers are asked to choose the correct class in the case that the class originally shown in the interface was wrong. The class vocabulary used for class correction is also LVTS, which is the same as the one used in Detic. As a result, workers modified 40. 40.72% classes throughout the annotation tasks, and the number of classes our dataset be- came 583. Note that 4.93% small objects wrongly detected by Detic are modified by the worker to larger object classes, and thus not all annotated target objects belong to small object classes.
Valuns 1002
bashi
plrte
Class
Objerts and Average Volumes in Majer (ScanRefar)
049
Ansnuge YOUr 12
100
130
dhvl
montt
dest
office offroudhal moo
giichen caltrett
tosel
bookshell
radibb
di Lish
cutal
whrchoon
briet
agrozand
bal
Figure 3: The distributions the number objects and average volumes in major 30 object classes for ARKitSceneRefer (upper) and ScanRefer (bottom).
get objects randomly, we select objects from low- frequent classes to high-frequent classes according to the detected class frequeney. As a result, the number of the entire object class of RRKitSceneRe- fer becomes 612.
3.2 Dataset Statisties
Finally, we collected 9.69 objects on average for each scene in 1, 605 3D scenes in ARKitScenes and 15, 553 referring expressions for these objects. Each object has 14.43 average length of the re- ferring expression. The referring expression covers 538 classes of indoor objects. Table shows a comparison of our data with existing 3D referring expression datasets. Our dataset is significantly larger than existing datasets in two aspects, 3D scenes, and object classes. Figure 3 shows distribu- tions of the number of objects and average volumes of each class in major 30 classes comparing our dataset with ScanRefer. We can see that ScanRefer includes many object classes for relatively large objects, such as "table" and "door." Compared to ScanRefer, ARKitSceneReffer includes many ob- ject classes for small objects. such as "bottle" and "box." Moreover, the volumes in ARKitSceneeeeer are not more than 0.10m while the volumes in ScanRefer are significantly greater than 0.10m3 The distribution indicates that our dataset is suc- cessfully focused on small objects. Figure 4 shows the most commonly used words for nouns, verbs, adjectives, and adverbs classified by NLTK (Loper
3.1.4 Annotation
With the above steps, we obtain the position infor- mation of small objects from 3D scenes without any human annotations. Then we annotate referring ex- pressions on Amazon Mechanical Turk as the last step for constructing our dataset. Before the anno- tation, we conduct a qualification test and create a worker pool to ensure the quality of our dataset. In the annotation website, We present workers 3D scene viewer with 3D bounding boxes made in Sec. 3.1.2, three images containing the target object with 2D bounding boxes detected by Detic, and the class of the target object. The images are randomly selected from video frames containing large detected bounding boxes from target objects. A 2D bounding box for the target object is shown in each image, aiming to make it easier for work- ers recognize the location of the target object. The 3D bounding boxes of all objects and all target objects of each task are shown in the 3D scene so that workers can clearly recognize the target ob- ject and other objects. While referring to the 3D scene and 2D images with target objects, workers
"Scc Appendix A for our annotation interface.

---

Dataset
Environment
Objects
Expressions
Average Length
Scenes
Venues
Object Class
Sr3D (Achlioptas et al., 2020)
ScanNet
8,863
83,572
9.68
1,273
613
Nr3D (Achlioptas al., 2020)
ScanNet
5,878
41,503
11.32
641
641
76
ScanRefer (Chen al., 2020)
ScanNet
11,046
51,583
20.27
800
800
279
ARKitSceneRefer
ARKitScenes
15,553
15,553
14.43
1,605
1,605
583
Table I: Overview of 3D referring expression comprehension datasets.
left stable
b sink sitting
OF green
Lue
small
Ierianntaion aa beside
ip
O Back
red
WOOdEn
directl
y back
top
cor the
Peu doo
placed atcchr manging
Wnite
grayo
left
closest
oven La Fight
Wa
Lose
Crrthhenpatii dom Teal Lee located berr
nextbl brown
(a) Noun
(b) Verb
(c) Adjective
(d) Adverb
Figure 4: Word clouds of (a) noun (b) verb (c) adjective (d) adverb for the ARKitSceneRefer. Bigger fonts mean high frequencies in the referring expressions.
Split
Expressions
Scenes
Object Class
the same method presented in Sec. 3.1.2. Finally, the center of the cluster with the most points is regarded as the center of the predicted target object on the 3D scene. Note that 2D models can't predict 3D bounding boxes because these models don't 't generate 2D instance segmentation maps.
Train
11,197
1,144
534
Val Test
2.732
285
363
1,624
176
310
Table 2: ARKitSceneRefer dataset statisties.
and Bird, 2002). In our dataset, "wall" and "sink" are commonly used as nouns, "hanging" and "sit- ting" as verbs, "white" and "black" as adjectives, and "close" and "directly" adverbs. Note that NLTK rarely fails to classify the part of speech, such as "oven" classified into adverbs. We further split our dataset into training, validation, and test sets. Table shows the statistics of our datasct after the split.
4.2
3D Models
Our 3D models are based on ScanRefer (Chen et al., 2020) and 3DVG-Transformer (Zhao et al.. 2021), which state-of-the 3D REC mod- els. We customize both ScanRefer and 3DVG- Transformer to fit to our task. Specifically, we don't 't adopt the vote regression loss introduced ScanRefer because there are no aeeeggraind in- stance segmentation labels in ARKitScenes, which means we define the object detection loss Lees as Litt = 0.5Cobjn-cls + Lbox + L where Lubju-els- Lbox. and Lsem-cls respectively represent the objectness binary classification loss, box gression loss, and semantic classification loss, all of which are introduced in ScanRefer. Our loss function defined as followings:
4 Model
Following the previous 3D referring expression studies (Chen et al., 2020; Zhao et al., 2021; Luo et al., 2022), we compare 2D to 3D REC models."
4.1 2D Models
Our 2D models are based on MDETR (Kamath et al., 2021) and OFA (Wang et al., 2022), which are ofate-of-the-n art 2D REC models. We first ap- ply 2D REC models, which take a video frame and referring expression as input and predict the bounding box corresponding to the referring ex- pression in the video frame. Then the centers of the predicted bounding boxes in video frames projected onto the 3D scene and clustered using
C= BL Lols
(I)
where Lloe and Lete respectively represent the local- ization loss and the tinguage-to-objeee classifica- tion loss, all of which are introduced in ScanRefer. and a, 8, and 7 represent the weights for each loss. Note that the loss function of 3DVG-Transformer are based on ScanRefer, but the weights are cus- tomized. We use the same weights introduced ScanRefer and SDVG-Transfoul
'More details are provided in Appendix "See Appendix D for a formulation of the task.

---

5 Experiments
Method
Split Dist@0.1 Dist@0.3 Dist@0.5
5.1 Evaluation Metrics
MDETR-rraddm
val
7,43
13.68
17.16
OFA-random
val
7.97
14.34
17.60
Following ScanRefer and Refer360" (Cirik et al.. 2020), we employ two evaluation metrics. The first metric is loU@k, kt where the predicted 3D bound- ing box is considered correct if its Intersection over Union (loU) with the ground truth 3D bounding box is equal to or greater than the threshold value This metric has been widely adopted in existing studies on REC. We set the threshold values & to 0.05, 051 0.25, and 0.5. The second metric is @l which considers the predicted 3D bound- ing box as correct if the distance between its center and the center of the ground truth 3D bounding box is equal to less than the threshold value 1. We use threshold values of 0.1, 0.3, and 0.5. Note that units of IoU & and Dist @ are percentiles.
OFA-Detic
val
13.39
25.51
31.62
MDETR-random
test
7.63
14.28
17.73
OFA-random
test
7.82
15.57
18.78
OFA-Detic
test
13.48
27.95
35,46
Table 3: Localization results by 2D models MDETR and OFA.
expression to word embeddings. Then all word embeddings in the referring expression concate- nated and input the 3D models. ScanRefer was trained for 200 epochs on the batch size of 32, and JDDG--ransfommn was trained for 500 epochs on the batch size of 8. The initial learning rate was 1e-3, 3, and Adam (Loshchilov and Hutter, 2019) was used for optimization. Following ScanRefer, we applied random flipping, random rotation in the range of -"",,",, random scaling in the range of [e [e-1.1 cod1, and random translation in the range of [-0.5m, 0.5m] point clouds for data augmenta- tion. The input features for the 3D models were xyz coordinates, colors, and normals, where the num- ber of vertices in the point cloud was 200, 000. All experiments were conducted on 1 NVIDIA A100 GPU.
5.2 Settings
2D Models We used MDETR and OFAlarge fine- tuned on RefCOCOg (Mao et al., 2016). We com- pared the following methods:
MDETR-randood and OFA-random: We ran- domly sampled input 1/10 video frames used to construct the 3D scene of ARKitScenes. Note that the target object may not appear in the randomly sampled video frames. If the target object not contained in a video frame, the 2D REC models may localize irrelevant regions, leading to noises.
5.3 Quantitative Analysis
Tables 3 and 4 present the evaluation results of 2D and 3D models on ARKitSceneReffer respec- tively.
This is a heuristic- method, OFA-Detic conducted object detection on video frames by Detic, and then used only video frames that contained the detected class appearing in the referring expression. If no class is included in the referring expression, used the same randomly sampled video frames OFA-random. Note that also used Detic for dataset construction, this method biased. We leave the comparison of using other object detectors for video frame selection as future work.
OFA-Detic:
For 3D models, 3DVG-Transformer outper- formed ScanRefer by a large margin. However, both of these models achieved lower performance than that previous datasets. For example, in terms of IoU@0.25, 15.90% the ScanRefer validation while only 2.21% on our validation set, which suggested that our dataset is insanely difficult compared to the existing datasets.
IoU
achieved
Dist Comparing 2D models of MDETR-random and OFA-random to 3D models for @.. and and OFA- OFA-random out- performed 3D models. However, 3D models were comparable to 2D models for D.s.@0.5. This is because the 3D models can make predictions based on the entire 3D scene. Even if the target object is not recognizable, the approximate position can be guessed. OFA-Detic significantly outperformed all
@ist@0.3,
We used DBSCAN (Ester et al., 1996) algorithm for clustering. We set the maximum distance between points in cluster to 0.02m, and the minimum number of points that make up cluster to 1.
3D Models We used NLTK (Loper and Bird, 2002) for tokenization. We used GloVe (Penning- ton et al., 2014) to convert tokens a referring
More discussions can be found in Appendix E.

---

Method
Split loU@0.05 IoU@0.15 1oU'0025 loU@0.5 Dist@0.1 Dist@0.3 Dist@0.5
ScanRefer
val
2.97
1.17
0.54
0.02
1.09
9.84
18.55
val
5.69
3.65
2.21
0.30
2.81
11.70
17.93
ScanRefer
test
3.00
1.13
0.46
0.03
1.02
9.26
17.82
3DVG Trenforrsfommer
lest
6.41
3,54
2.20
0.41
2.90
12.82
19.18
Table 4: Localization results by 3D models of ScanRefer and BDVG-Transformee
other methods, indicating the importance of select- ing video frames that contain the target object for the 2D model.
the 3D scenes of ARKitScenes fail to reconstruct accurately, which is common to ScanNet, resulting in missing parts or low resolution.
5.4 Qualitative Analysis
Human Annotation To ensure the quality of the dataset, we conducted a qualification test to gather highly skilled workers. However, still, subjectivity rarely leads to occasional errors in human annota- tions. Particularly in this paper, selecting accurate 3D bounding boxes is susceptible to such influ- ences.
Figure 5 shows the comparison of localization re- sults of SDVG-Transfommer and OFA-Detic. In the leftmost example, both 3DVG-Transformer and OFA-Detic successfully localized the referred ob- ject. Relatively large objects that are unique in the scene were easy to localize accurately. However, in the second example from the left, only 3DVG- Transformer successfully localized the referred ob- ject. This suggests that 2D models, which rely local information from video frames, struggled to consider the entire 3D scene simultaneously, re- sulting in overlooking relatively small objects. In the third example from the left, only OFA-Detic successfully localized the referred object. This sug- gests that 3D localizing models faces difficulties in accurately localizing quite small objects such bottles. In the rightmost example, both 3DVG- Transformer and OFA- -Dtic failed to localize the referred object. This suggests that objects in com- plicated scenes are still difficult localize even with current best models.
2D Models In this paper, we utilized off-the-shelf 2D models that were fine-tuned on RefCOCOg. These models already exhibit impressive perfor- mance, but we can expect further improvement on our task by fine-tuning them on our dataset. In our experiments, we employed simple heuristic video frame selection methods. It can potentially enhance accuracy to implement more optimized video frame selection methods tailored to our task.
3D Models ARKitScenes lacks semantic segmen- tation maps, which leads to the omission of the vote regression loss employed by ScanRefer and 3DVG-Transformer. Consequently, in our experi- ments, there is a possibility that the object detector is not fully optimized. However, there has been significant progress in recent research 3D scene understanding (Peng et al., 2023; Yang al., 2023). Leveraging these advancements to generate high- quality pseudo- labels could improve 3D model per- formance.
6 Conclusion
In this paper, we introduced a new 3D REC dataset, ARKitSceneRefeer, for small objects. ARK- RefeeneRefer consists of 15, 553 referring expres- sions for 1, 605 scenes in ARKitScenes. We found that conventional 3D models cannot get high ac- curacy on our dataset. We also confirmed that the performance of the 2D models varied significantly depending on the input video frames. In the future, we plan to use the confidence scores of 2D models for image selection. We hope that our dataset will be useful in the 3D REC community.
Ethical Statements
The annotations were conducted on Amazon Me- chanical Turk, and we ensured that the workers re- ceived fair compensation, taking into account mar- ket rates. As we utilized existing data and didn't 't collect any newly scanned data, the workers' pri- vacy can be well protected. The annotation process was conducted in compliance with the procedures established by our institution.
7 Limitations
Dataset ARKitSceneRefer only provides one re- ferring expression per object, which is less than in previous works. Additionally, some objects in

---

Query: trash can on the floor in between the sink and the toilet
Query: brown bucket in the bathroom under the sink next to the bathtub
Query: pink scap dispenser the bathroom sink white counter beside of a white dispenser
Query: black wine bottle on the top shelf of the entertainment center against the wall
Ground Truth
Zoomed 20 Image
3DVG- CVG--Tranformme
OFA-Detic
Figure 5: Qualitative analysis of the localization results of the TonG-Transformer and OFA-Detic models. Yellow, green, and red represent ground truth, correct predictions, and incorrect predictions, respectively.
Acknowledgement
ral Information Processing Systems Datasets and Benchmarks Track (Round 1).
This work was supported by JSPS KAKENHI Grant Number JP23H03454 and JP22K17983, and by JST PRESTO Grant Numjber JPMJPR20C2.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. 2020. End-to-end object detection with transformers. In Computer Vision ECCV 2020: 16th European Conference, Glasgow. UK. August 23-28, 2020, Proceedings, Part page 213-229, Berlin, Heidelberg. Springer- Verlag.
References
Panos Achlioptas, Ahmed Abdelreheem, Fei Xia, Mo-
hamed Elhoseiny, and Leonidas J. Guibas. 2020. Referlt3D: Neural listeners for fine-grained 3d object identification in roal-world scenes. In Proceedings of the 16th European Conference Computer Vision (ECCV).
Angel Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Niessner, Manolis Savva, Shuran Song, Andy Zeng, and Yinda Zhang. 2017. Matter- port3D: Learning from RGB-D data in indoor envi- ronments. International Conference on Vision (3DV).
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Mo- toki Kawanabe. 2022. Scanqa: 3d question answer- understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
ing for spatial
Dave Zhenyu Chen, Angel X Chang, and Matthias Nießner. 2020. Scanrefer: 3d object localization in rgb-d scans using natural language. In Proceed- ings of the European Conference on Computer Vision (ECCV). Springer.
Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Bran- don Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. 2021. ARKitscenes diverse real-world dataset for indoor scene understanding using mo- bile RGB-d data. Thirty-fifth Conference on Neu-
Dave Zhenyu Chen, Qirui Wu, Matthias Nießner, and Angel X. Chang. 2022. D3net: A lesteer-listener architecture semi-supervised dense captioning and visual grounding in rgb-d scans. In ECCV.

---

Xinlei Chen, Hao Fang, Tsung-Y- Yi Lin, Ramakr- ishna Vedantam, Saurabh Gupta, Piotr Dollár, and C. Lawrence Zitnick. 2015. Microsoft coco cap- tions: Data collection and evaluation server. CoRR, abs/1504.00325.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. 2014. ReferltGame: Referring to objects in photographs of natural scenes. In Proceed- ings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 787- 798, Doha, Qatar. Association for Computational Linguisties.
Zhenfang Chen, Lin Ma. Wenhan Luo, and Kwan- Yee Kenneth Wong. 2019. Weakly-Supervise Spatio-Temporally Grounding Natural Sentence in Video. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL).
Shuhei Kurita, Naoki Katsura, and Eri Onami. 2023. Refego: Referring expression comprehension dataset from first-person perception of ego4d. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 152144
Volkan Cirik, Taylor Kirkpairka,, and Louis- Philippe Morency, 2020. Refer360%; A referring expression recognition dataset in 360 images. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational L Linguistics, pages 7189- 7202, Online. Association for Computational Lin- guistics.
Liunian Harold Li*, Pengchuan Zhang", Haotian Zhang*. Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao. 2022. Grounded language-imed pre-training. In CVPR.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Mathhis Nießner. 2017. Scannet:
Zhenyang Li, Ran Tao, Efstratios Gavves, Cees G. M. Snoek, and Arnold W. M. Smeulders. 2017. Tracking by Natural Language Specification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
3d reconstructions of indoor scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR).
Xuejing Liu, Liang Li, Shuhui Wang. Zheng-Jun Zha, Li Su, and Qingming Huang. 2019. Knowledge- guided pairwise reconstruction network for weakly supervised referring expression grounding. In Pro- ceedings of the 27th ACM International Conference on Multimedia, MM '19, page 539-547 New York, NY, USA. Association for Computing Machinery.
Jiajun Deng, Zhengyuan Yang. Tianlang Chen, Wen- gang Zhou, and Hougiang Li. 2021. Transvg: End- to-end visual grounding with transformers. In Pro- ceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 1779-1779.
Martin Ester, Hans-Peter Kriegel, Jörg Sander, and Xiaowei Xu. 1996. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference Knowledge Discovery and Data Min- ing, KDD 96, page 226-231. AAAI Press.
Edward I Loper and Steven Bird. 2002. Nltk: The natural language toolkit. arXiv preprint cs/0205028.
Ilya Loshchilov and Frank Hutter, 2019. Decoupled weight decay regularization, In International Confer- ence Learning Representations (ICLR).
Agrim Gupta, Piotr Dollar, and Ross Girshick. 2019. Lvis: A dataset for large vocabulary instance seg- mentation. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR).
Junyu Luo, Jiahui Fu, Xianghao Kong, Chen Gao, Haib- ing Ren, Hao Shen, Huaxia Xia, and Si Liu, 2022. 3d-sps: Single-stage 3d visual grounding via referred point progressive selection. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 16454-16463.
Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. 2022. Bottom up top down detection transformers for language grounding in im- ages and point clouds. In ECCVV
Junhua Mao, Jonathan Huang. Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. 2016. Generation and comprehension of unambiguous ob- ject descriptions. In CVPR.
Haojun Jiang, Yuanze Lin, Dongchen Han, Shiji Song, and Gao Huang. 2022. Pseudo-q: Generating pseudo language queries for visual grounding. Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition.
Pushmeet Kohli Nathan Silberman, Derek Hoiem and Rob Fergus. 2012. Indoor segmentation and support inference from rgbd images. In ECCV.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. 2021. Mdetr - modulated detection for end-ta-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1780-1790.
Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser. et al. 2023. Openscene: 3d scene understanding with open vocabularies. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recog- nition, pages 815-824.

---

Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Confer- ence on Empirical Methods Natural Language Pro- cessing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
matching and reconstruction for weakly referring expression grounding. IEEE Transactions on Pat- tern Analysis and Machine Intelligence, 43(1 43(11):4188 18:4 4195.
Johanna Wald, Armen Avetisyan, Nassir Navab, Fed- erico Tombari, and Matthias Nießner. 2019. Rio: 3d object instance re-localization changing indoor environments. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, pages 7658-7667.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. 2015. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- to-sentence models. In 2015 IEEE International Conference on Computer Vision (ICCV), pages 2641- 2649.
Peng Wang, An Yang. Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. 2022. OFA: Unifying ar- chitectures, tasks, and modalities through a simple learning framework. In Pro- ceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 23318 233400. PMLR.
Yuankai Qi, Qi Wu, Peter Anderson, Xin Wang, William Yang Wang. Chunhua Shen, and Anton van den Hengel. 2020. Reverie: Remote embodied visual referring expression in real indoor environ- ments. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR).
Sequence-to-sequeee
Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. 2021. -aattot- 3d dataset (HM3d); 1000 large-scale 3d environ- ments for embodied AL. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track.
Yanmin Wu, Xinhua Cheng. Renrui Zhang, Zesen Cheng, and Jian Zhang. 2022. Eda: Explicit text- decoupling and dense alignment for 3d visual and language learning.
Mutian Xu, Pei Chen, Haolin Liu, and Xiaoguang Han. 2022. To-scene A large-scale dataset for understand- ing 3d tabletop scenes. In ECCV.
Yunhan Yang, Xiaoyang Wu, Tong He, Hengshuang Zhao, and Xihui Liu. 2023. Sam3d: Segment any- thing in 3d scenes. arXiv preprint arXiv:2306.03908
David Rozenberszki, Or Litany, and Angela Dai. 2022. Language-grounded
Language- grounded indoor 3d semantic segmenta- tion in the wild. In Proceedings of the European Conference on Computer Vision (ECCV).
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Unitab: Unifying text and box outputs for grounded vision-language modeling. In ECCV.
Shuran Song. Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. 2017. Se- mantic scene completion from a single depth image. Proceedings of 30th IEEE Conference on Computer Vision and Pattern Recognition.
I Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. 2016. Modeling context in referring expressions. In Proceedings of the Euro- pean Conference on Computer Vision (ECCV), pages 69-85.
Julian Straub, Thomas Whelan, Lingni Ma, Yufan Chen, Erik Wijmans, Simon Green, Jakob J. Engel. Raul Mur-Artal, Carl Ren, Shobhit Verma, Anton Clark- son, Mingfei Yan, Brian Budge, Yajie Yan, Xiaging Pan, June Yon, Yuyang Zou, Kimberly Leon, Nigel Carter, Jesus Briales, Tyler Gillingham, Elias Mueg- gler, Luis Pesqueira, Manolis Savva, Dhruv Batra, Hauke M. Strasdat, Renzo De Nardi, Michael Goe- sele, Steven Lovegrove, and Richard Newcombe. 2019. The Replica dataset: A digital replica of indoor spaces, arXiv preprint arviv: I 057977
Zhihao Yuan, Xu Yan, Zhuo Li, Xuhao Li, Yao Guo, Shuguang Cui, and Zhen Li. 2022. Toward explain- able and fine-grained 3d grounding through referring textual phrases. arXiv preprint arXiv:2207.0182..
Zhihao Yuan, Xu Yan, Yinghong Liao, Ruimao Zhang, Sheng Wang, Zhen Li, and Shuguang Cui. 2021. In- stancerefer: Cooperative holistic understanding for visual grounding point clouds through instance multi-leve contextual referring. In Proceedings of the IEEE/CVF International Conference on Com- puter Vision (ICCV), pages 1 1791-1800.
Sanjay Subramanian. William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. 2022. ReCLIP: A strong zero-shot baseline for re- ferring expression comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 5198-5215, Dublin, Ireland. Association for Computational Linguistics.
Lichen Zhao, Daigang Cai, Lu Sheng, and Dong Xu. 2021. 3dvg-transformer: Relation modeling for vi- sual grounding point clouds. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 2928-2931
Mingjie Sun, Jimin Xiao, Eng Gee Lim, Liu, and John Y. Goulermas. 2021. Discriminative triad

---

Xingyi Zhou, Rohit Girdhar, Armand Joulin, Philipp Krähenbühl, and Ishan Misra. 2022. Detecting twenty-thousand classes using image-level supervi- sion. In ECCV.

---

A Mturk Annotation Interface
of 50 objects (()+(i)((iii) are detectable, which certified the quality of small object representations. Although ARKitScenes provides high-quality 3D scenes, we didn't use them this paper because of the lack of computational resources. Indeed, we confirmed that we would not be able to see high-quality 3D scenes on the web browser. More- over, not only paper but also other papers (e.g., ScanRefer and SDVG-Transforme adopted down- sampling of 3D scenes before feeding them into models to reduce the GPU memory. We believe that future improvements in computational resources would make possible to handle high-quality 3D scenes while we can use the same annotations of ARKitSceneRefer.
Figure 6 present example of our annotation in- terface. Workers can see 3D scenes, object classes, and 3D bounding boxes. They can also rotate and zoom 3D scenes interactively.
B Unique/Multiple Objects
One of the major challenges for 3D REC is multi- ple objects with the same class as the target object can appear in the same scene. Basically, it's hard to determine unique/multiple objects precisely be- cause we sampled target objects before revising classes from Detic. Instead, we can determine unique/multiple objects on the dataset before the target object selection, which means the analysis is little noisy. We confirmed that class-unique and class-multiple objects the whole dataset are 61.5% and 38.5% respectively. As a result, the model should focus on not only object words but also other descriptions (e.g., other objects and re- lationships) in order to achieve higher scores. Fur- thermore, the performances of unique and multi- ple objects the test set by were 1.75% and 2.16% IoU@0.25 and 12.69% and 13.00% on @ist@0.3, respectively. might be because of the difference in the amount of the data. Specifically, we selected low-frequency ob- jects so that our dataset could cover extensive ob- ject classes, leading to the fact that the average number of objects in the whole dataset was 94.11 for class-unique and 199.45 for class-multiple, re- spectively.
D Task
We introduce a text-based small object localization task in 3D scenes, Our task is completely differ- ent from existing 3D referring expression tasks in terms of the object size, which means existing tasks mostly focus large objects. In our task, the input of the 3D REC model is fine-grained 3D indoor scene and referring expression that clearly describes the target object. 3D scenes are represented by xyz coordinates, colors, and nor- mals. The model predicts the 3D bounding box of the target object as:
3DVG erGaeema((Seeee Query) Box
(2)
where Box E R represents the xyz coordinates, height, width, and depth of the 3D bounding box.
E Discussion
C Human Score on the Small Dataset
Number of Points on 3D Scenes When the 3D models localize small objects, the number of ver- tices in the point cloud should be a very important parameter compared to localizing large objects. As shown in Table 5, investigated how the number of vertices in the point cloud affects the perfor- mance. We used 3DVG-Transformer because its performance was better than SeanRefer. However, the performance was comparable as we reduced the number of vertices. This is because the object de- tectors employed by 3D models were not optimized because of the lack of semantic segmentation maps.
To verify the dataset quality, we randomly tested 50 samples from val and test datasets. We carefully checked whether the objects were detectable and classified them into five categories: (i) we can local- ize objects 3D scenes without video frames (ii) we can localize objects in 3D scenes referring to video frames. (iii) we can't localize objects because of the ambiguity of referring expressions. (iv) we can't t localize objects because of the incorrectness of bounding boxes. (v) It's hard to localize objects. We confirmed that 29 of 50 objects can be localized by referring expressions and 3D scenes. Further- more, an additional five objects can be localized by using video frames. This result indicated that the performances of 3D REC models were much lower than those of humans. We also confirmed that 42
3D Features Chen et al. (2020) found that us- ing color and normal information improves perfor- mance in 3D models. Therefore, we conducted experiments to verify this claim in our task. We

---

Points
Split loU@0.05 IoU@0.15 IoU@0.25
loU@0.
Dist@0.1
Dist@0.3
D.st@0.5
50,000
test
6.82
4.31
2.57
0.23
2.72
12.95
20.35
100,000
test
7.27
4.72
3.04
0.49
3.26
12.82
18.90
200,000
test
6.41
3.54
2.20
0.41
2.90
12.82
19.18
Table 5: Comparison of different numbers vertices in the point cloud the 3DVG-Transfommer model.
trained the model with four features: (i) coordi- nates (xyz), (ii) coordinates and colors (xyz+rgb), (iii) coordinated and normals (xyz+normal), (iv) co- ordinates, colors, and normals (xyztrg for the DDVG-Tranfformr model. As shown in Table 6, rgb features were not effective in our task. This is because our dataset focuses on small objects and handles a wide range of object classes, mak- ing it more difficult associate rgb features with objects. By contrast, normal features were slightly effective our task.
oracle was slightly superior to OFA-Detic because many referring expressions include the object class in themselves. The results showed that if use video frames with target objects, the model can be further improved in our task.
Additionally, we compared 3DVG-Trane TDVG-Transformer trained with coordinates, colors, and normals (xyz+color+normaa) with a model trained with coordinates, colors, normals, and multiview fea- For the reduction of computational costs, changed the number of points, epochs, and object pro- posals from 200, 000 to 25, 000, from 500 200, and from 1, 024 to 256, respec- tively. As shown Table -ran-Transformer (xyztcolor+normal) was comparable to 3DVG- Transformer kyzzoolorr-oomal-mmiiiiii showed that the importance of 3D scenes might be comparable with video frames.
tures (xyz+tolorrtormall--uttie
This
Upper Bound of 2D Models We focused on that MDETR and OFA achieve high accuracy in 2D REC tasks, but the performance is significantly lower in our task. Therefore, we investigated the upper bound of the OFA-based 2D model, which outperforms the MDETR-based model, shown in Table 3. We further conducted experiments on the following two settings:
OFA-oracle: We conducted object detection video frames by Detic, and then used only video frames with the detected class corre- sponding with the one annotated in the ARK- Refeen
OFA-upper: We used only the video frames used for annotating referring expressions by crowdsourcing workers.
As shown in Table 8, OFA-oracle and OFA-upper were superior to OFA-random significantly. OFA-

---

Input
Split
loU@0.
1oU@0.15
IoU@0.25 loU@0.5 Dis1@0.1
Dist@0.3 Dist@0.5
xyz
test
6.60
4.18
2.36
0.16
2.57
12.46
18.30
xyz+rgb
test
6.19
3.90
2.51
0.34
2.62
11.09
17.78
xyz+normal
7.04
4.33
2.58
0.40
3.43
14.33
22.24
xyz+rgh+norrml
test
6.41
3.54
2.20
0.41
2.90
12.82
19.18
Table 6: Comparison of color and normal features for the 3DVG-Transfomeer model.
Input
Split loU@0.05
IoU@0.15
loU@0.25
loU@0.5 Dist@0.1
Dist@0.3
Dist@0.5
xyz+rgb+normml
lest
3.26
2.08
1.25
0.16
syzengo-noommal/mumiiive
test
2.62
1.65
0.92
0.08
5.74
Table 7: Comparison of multiview features for the EDVG-Transformer model. Please note that the experimental settings are different from others.
Method
|Split
D
D.st@0.3
D.5
OFA-random
test
7.82
15.57
OFA-Detic
test
13.48
27.95
35.46
OFA-oracle test 16.25
34.29
42.30
OFA-upper test 32,94
63.30
76.10
Table 8: Oracle and upper bound results for the OFA-
based model.

---

Bounding Box Selection Referring Expression
Stee Tite 3D deta and imagee You can zoom 3D dista scroiling on You can also move 3D data parallet with Cirl + drag
23 2014
47
Restrahh.com
Click 3D data to open 3D data new teb. you can't ore the object tebels or imgges please reloed this
Show the button
Boundiing Exictence
Expressions
"yoe" there object neer the 3o bounding
E.g. while cup top Altchen the clue Dic.
can see images
Short phrase.
NOT sentence!
Include the
bounding box.
covered by bounding but.
should be mambiiuuus and discernible for others.
The size the bounding
Please make sure that the phrase includes the object label as well as the
ahould the object, neither too barge
Do Please wrlte
object properties. egg- "toys are meant played with"
nor too small, and it should match
Write relationshiis batween the target object and nearby object for identifiig
"no" wxtte "the coundiing box incorrect" Reter data images.
bounding
there otject near the bounding box ifilt
doserf mukch lebd.
Please consider only the 3D bounding boxes the
boxes in images are not relevant the judgment. target box is highlligtted in red, while others are displeyed yellow.
basih_co can
Oyes Cio
lempehade
Oyes Cio
brash, can
One
lampohoude
Oyes Cio
Comments and feedbock (Please eave a comment you have any questions suggestions, etc.
Submit
Figure 6: Example of the annotation interface. We provided an opportunity to indicate if they are unable to locate the object at all.


Reviews:
[{"review_id": "qYpW3w1Dl2", "reviewer": "Reviewer_R12s", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### For Reviewer R12s\n\nWe would like to thank the reviewers for the positive comments and would like to address the points raised below:\n\n**\"This paper only constructed a dataset but did not propose any models to solve the 3D visual grounding problem of small objects, which lacks of novelty.\"**\n\nWe don't consider it as a reason for reject because our main contribution is on the construct of a new dataset, and it's not indispensable to propose a novel model.\n\n**\"The experiment was not solid enough. The experiment was only conducted on two existing models, and the workload was insufficient.\"**\n\nWe used two SoTA REC models for both 2D and 3D. We believe it's mostly sufficient to grasp the latest performance for the small object detection task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1.\tIt’s quiet important to construct such a dataset for small objects. Because this is very helpful for the applications of 3D referring expression comprehension.\n2.\tARKitSceneRefer contains 15k objects of 1, 605 indoor scenes, which are significantly larger than those of the existing 3D referring datasets.", "reasons_to_reject": "1.\tThis paper only constructed a dataset but did not propose any models to solve the 3D visual grounding problem of small objects, which lacks of novelty.\n2.\tThis study claims that the 3D ReC model used is a State-of-the-Art (SOTA) model, but one of them, ScanRefer, is severely outdated. The author needs to validate the dataset on more diverse SOTA (State-of-the-Art) models with different architectures.\n3.\tIn this dataset, the object only has one expression. The object is not described from different perspective,which will affect the effectiveness of the dataset.", "questions_to_authors": "", "paper_topic_and_main_contributions": "Existing indoor 3D referring expression comprehension datasets typically cover larger object classes that are easy to localize.This paper introduced a new 3D REC dataset, ARKitSceneRefer, for small objects. This dataset is based on the recently proposed diverse and high-resolution 3D scene dataset of ARKitScenes. ARKitSceneRefer consists of 15, 553 referring expressions for 1, 605 scenes in ARKitScenes.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "sYSI79tbzb", "reviewer": "Reviewer_E4hY", "summary": "", "questions": "1. As claimed in lines 355-358, workers are asked to judge whether the 3d bounding box is correct. It's not clear whether workers are asked to fix the wrong bounding box or whether wrong bounding boxes are excluded from the final dataset.\n", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### For Reviewer E4hY\n\nWe would like to thank the reviewers for the positive comments and would like to address the points raised below:\n\n**\"One of the major challenges for existing 2D or 3D Referring Expression Comprehension is that multiple objects with the same class as the target object can appear in one image or scene. The model must fully understand the expression rather than only the object word to determine the target object. However, in this dataset, during the Annotation or Dataset Statistic section, there is no discussion about this challenge. If most target objects are class-unique in the scene, models trained in this dataset will only focus on object words and neglect other descriptions. Authors may consider this point and build a harder test set.\"**\n\nBasically, it's hard to determine unique/multiple objects precisely because we sampled target objects before revising the class labels from Detic. Instead, we can determine unique/multiple objects on the dataset before the target object selection, which means the labels are perhaps wrong. We confirmed that class-unique and class-multiple objects in the whole dataset are 61.5% and 38.5%, respectively. As a result, the model should focus on not only object words but also other descriptions like other objects and relationships in order to achieve the higher scores. By the way, the performance of unique and multiple objects on the test set by 3DVG-Transformer were 1.75% and 2.16% on IoU@0.25 and 12.69% and 13.00% on Dist@0.3, respectively. It's probably because of the difference of the amount of the data. Specifically, we selected low-frequency objects so that our dataset could cover extensive object classes, leading that the average number of objects in the whole dataset was 94.11 for class-unique and 199.45 for class-multiple, respectively.\n\n**\"As mentioned in lines 208-210, small objects in 3D scenes are often unclear and hard to recognize. As a 3D Referring Expression Comprehension task, 3D scenes should be one of the important information resources. If not, this task may degenerate into 2D Referring Expression Comprehension on video frames. Unfortunately, there is no evidence to support this point. As one option, Authors may consider performing experiments of comparing ScanRefer(xyz+rgb+normal) with ScanRefer(xyz+rgb+normal+multiview). If these two models are comparable, the importance of 3D scenes may be comparable with video frames.\"**\n\nUnfortunately, we could not fine the experiments with xyz+rgb+normal+multiview within the short author response period. By the way, we think video frames and 3D scenes have different aspects. Video frames provide more detailed but localized information, and 3D scenes help compositional understanding of 3D spaces. Using only single modal would not be an optimal solution. Instead, we believe it's necessary to explore both video frames and 3D scenes for achieving better performance. For example, an object, which is included in a referring expression but not visible in video frames, might make 2D REC models confused. However, utilizing 3D scenes might complement the lacked information, leading a promising chance to detect the target object precisely.\n\n**\"As claimed in lines 355-358, workers are asked to judge whether the 3d bounding box is correct. It's not clear whether workers are asked to fix the wrong bounding box or whether wrong bounding boxes are excluded from the final dataset.\"**\n\nWe excluded the data with wrong bounding boxes from the final dataset.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The first 3D Referring Expression Comprehension datasets about small object grounding.\n2. They compare both 2D and 3D baselines on this dataset to verify the difficulty of this new task.\n3. This paper is well-written.", "reasons_to_reject": "1. One of the major challenges for existing 2D or 3D Referring Expression Comprehension is that multiple objects with the same class as the target object can appear in one image or scene. The model must fully understand the expression rather than only the object word to determine the target object. However, in this dataset, during the Annotation or Dataset Statistic section, there is no discussion about this challenge. If most target objects are class-unique in the scene, models trained in this dataset will only focus on object words and neglect other descriptions. Authors may consider this point and build a harder test set.\n\n2. As mentioned in lines 208-210, small objects in 3D scenes are often unclear and hard to recognize. As a 3D Referring Expression Comprehension task, 3D scenes should be one of the important information resources.  If not, this task may degenerate into 2D Referring Expression Comprehension on video frames. Unfortunately, there is no evidence to support this point. As one option, Authors may consider performing experiments of comparing ScanRefer(xyz+rgb+normal) with ScanRefer(xyz+rgb+normal+multiview). If these two models are comparable, the importance of 3D scenes may be comparable with video frames.", "questions_to_authors": "1. As claimed in lines 355-358, workers are asked to judge whether the 3d bounding box is correct. It's not clear whether workers are asked to fix the wrong bounding box or whether wrong bounding boxes are excluded from the final dataset.\n", "paper_topic_and_main_contributions": "Existing Referring Expression Comprehension in 3D environments focus more on localizing large objects. This work first proposes a dataset concentrating on localizing small objects. The dataset provides human-written referring expressions involving more target objects. They compare state-of-the-art 2D and 3D models in this dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "hWZQeYAniR", "reviewer": "Reviewer_2zzf", "summary": "", "questions": "A. Do you think its possible/reasonable to test human performance on a small subset of the 3D detection data to demonstrate task feasibility?", "limitations": "", "rating": 3, "confidence": 2, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### For Reviewer 2zzf\n\nWe would like to thank the reviewers for the positive comments and would like to address the points raised below:\n\n**\"The 3d scene quality of ARKitScenes is still quite low, which raises even further concerns to the quality of small object representations in the scene (whether they are detectable at all). Perhaps it would be useful to verify this on a sampled subset.\"**\n\nWe tested 50 objects whether they are detectable in the 3D scene. We confirmed that 42 of 50 objects are detectable, which certifies the quality of small object representations. By the way, ARKitScenes provides high-quality 3D scenes. However, we didn't use them in this paper because of the lack of computational resources. For example, we confirmed not to be able to see high-quality 3D scenes on the web browser. Moreover, not only our paper but also other papers like ScanRefer and 3DVG-Transformer adopted downsampling of 3D scenes before feeding them into models to reduce the GPU memory. We believe that future improvement of computational resources would make it possible to handle high-quality 3D scenes while we can use the same annotations.\n\n**\"The multiple choice annotation format does not give annotators the opportunity to indicate if they are unable to locate the object at all, which may force them to make a guess in some scenarios. Also there seems to be no post-annotation qualitative/quantitative examination to verify the quality of the annotations.\"**\n\nWe provided the opportunity to indicate such situations. We wrote an instruction for it as follows: Click \"yes\" if there is an object near the 3D bounding box, and the object is mostly covered by the bounding box. The size of the bounding box should be almost fit to the object, neither too large nor too small, and it should match the object label. Click \"no\" if there is no object near the bounding box or if it doesn't match the label. Please consider only the 3D bounding boxes in the 3D scene; 2D bounding boxes in images are not relevant to the judgment.\n\n**\" Do you think its possible/reasonable to test human performance on a small subset of the 3D detection data to demonstrate task feasibility?\"**\n\nWe tested human performance on 50 samples of the dataset. We confirmed that 29 of 50 objects can be localized by referring expressions and 3D scenes. Moreover, additional 5 objects can be localized by using video frames. We also confirmed that the main reason for  localization failures was ambiguity in the referring expressions, which raises a situation that multiple objects correspond with the referring expressions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivations of the paper makes sense as small object identification is indeed an important unsolved problem for embodied agents and virtual assistants. Current datasets suffer severely from long tail object category distribution when it comes to small objects.\n2. The paper is clear and generally well-written, providing intuitive explanation to the authors' approach.\n3. The dataset can be useful to promote research to improve 3d detection model robustness to small objects.", "reasons_to_reject": "1. The 3d scene quality of ARKitScenes is still quite low, which raises even further concerns to the quality of small object representations in the scene (whether they are detectable at all). Perhaps it would be useful to verify this on a sampled subset.\n2. The multiple choice annotation format does not give annotators the opportunity to indicate if they are unable to locate the object at all, which may force them to make a guess in some scenarios. Also there seems to be no post-annotation qualitative/quantitative examination  to verify the quality of the annotations.", "questions_to_authors": "A. Do you think its possible/reasonable to test human performance on a small subset of the 3D detection data to demonstrate task feasibility?", "paper_topic_and_main_contributions": "This work builds upon the existing ARKitScenes dataset to produce a new 3d object localization dataset focusing on small objects in indoor scenes. Authors compares the new dataset to existing ones and tests existing 3d and 2d models on the new dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a 3D recognition dataset for small objects. It’s based on the 3D scene dataset ARKitScenes, but adds 15,553 referring expressions of mostly small objects that have been overlooked in previous 3D recognition datasets that focused more on large objects like sofas and chairs.

Strengths:
- Reviewers unilaterally agree on the merit of the proposed 3D recognition dataset for small objects.
- Some reviewers highlighted that this dataset is also larger than prior 3D referring expression datasets, regardless of object size.
- Most reviewers commended the paper as well written and clear 

Main unresolved concern:
- A small fraction of the data is incorrect. In a random test of 50 samples, 16% of objects in the annotations were either difficult or impossible to localize.

After rebuttal, reviewers reached a consensus soundness and excitement scores of 3.

Despite the weakness, the dataset adds value to the community and addresses a gap in current datasets. The authors are urged to visibly highlight the 16% unrecognizable objects when publishing the dataset to allow users to make informed assumptions about the dataset.
Decision: Accept-Findings