SlowGuess/NeurIPS_2023_OCR huggingface dataset:
where the hyper-parameter  $\Delta$  defines the score margin. The losses  $\mathcal{L}_{\mathrm{rel1}}$  and  $\mathcal{L}_{\mathrm{rel2}}$  ensure that the contextual relation  $r_{\alpha, \beta}$  (between objects  $\bar{b}_{\alpha}$  and  $\bar{b}_{\beta}$ ) has a matching score surpassing the scores of other contextual relations (with target  $\bar{b}_{\alpha}$  or contextual object  $\bar{b}_{\beta}$ ) by a margin of  $\Delta$ . This encourages the network to explicitly differentiate related and unrelated contextual relations for the target object. We utilize the learned contextual relation features for target inference, as detailed in Section 3.1.3.
In addition to the previous losses for learning contextual objects and relations, we also supervise the target identification network on its estimated grounding scores  $[p_1^\star, p_2^\star, \dots, p_T^\star]$  for the  $T$  candidate object detections. We take the  $\alpha$ -th detected object that best matches the ground-truth label  $b_{\mathrm{gt}}$  as the positive sample, and we treat other detections as negative samples. We compute the binary
cross-entropy loss on their grounding scores as follows:
$$
\mathcal {L} _ {\mathrm {v g}} = - \log \left(p _ {\alpha} ^ {\star}\right) + \sum_ {\substack {k = 1 \\ k \neq \alpha}} ^ {T} \lambda_ {n} \cdot \log \left(1 - p _ {k} ^ {\star}\right). \tag{7}
$$
We take the sum of losses for all the modular networks to train the entire model. Following the previous works [6, 7], we apply auxiliary losses to the predictions from every decoder layer of the modular networks. The overall training loss is:
$$
\mathcal {L} = \frac {1}{N _ {d}} \sum_ {k = 1} ^ {N _ {d}} \left(\mathcal {L} _ {\mathrm {n l o c}} ^ {k} + \mathcal {L} _ {\mathrm {n c l s}} ^ {k} + \mathcal {L} _ {\mathrm {l o c}} ^ {k} + \mathcal {L} _ {\mathrm {c l s}} ^ {k}\right) + \frac {1}{N _ {r}} \sum_ {k = 1} ^ {N _ {r}} \left(\mathcal {L} _ {\mathrm {r e l} 1} ^ {k} + \mathcal {L} _ {\mathrm {r e l} 2} ^ {k}\right) + \frac {1}{N _ {g}} \sum_ {k = 1} ^ {N _ {g}} \mathcal {L} _ {\mathrm {v g}} ^ {k}, \tag {8}
$$
where the superscript  $k$  indexes the losses of predictions from multiple decoder layers.
Nr3D [6] is built on 3D indoor scene dataset ScanNet [35]. It contains 41,503 human-annotated text descriptions, covering 76 object categories and 707 indoor scenes. The dataset is divided into "Easy" and "Hard" subsets depending on whether there are objects that share the same category as the target in the scene. Based on whether a specific viewpoint is required to infer the target, the dataset is divided into "View-dep." and "View-indep." subsets.
Sr3D [6] contains 83,572 descriptions that are automatically generated using specific templates. Similar to the Nr3D dataset, the Sr3D dataset is divided into multiple subsets for evaluation.
ScanRefer [7] provides 51,583 text descriptions of 11,046 objects in 800 3D scenes from the ScanNet. On average, there are 13.81 objects and 64.48 text descriptions per scene. The official division takes 36,665 samples as the training set and 9,508 as the test set. According to whether the target object category is unique in the scene, the dataset is divided into "Unique" and "Multiple" subsets.
We utilize the AdamW optimizer [36] to train our model with a batch size of 24. For visual feature encoding, we utilize the PointNet++ network [37] with an initial learning rate of  $10^{-3}$ . The rest of the model has an initial learning rate of  $10^{-4}$ , and the weight decay value is set to  $5 \times 10^{-4}$ . We employ the first three layers of the RoBERTa [38] to extract text features. We train our model for 120 epochs on the Nr3D dataset, 60 epochs on the Sr3D dataset, and 100 epochs on the ScanRefer dataset. The hyper-parameter  $\Delta$  in Equation 6 is set to 0.4, and we set the weight hyper-parameters  $\lambda_{\mathrm{giou}} = 1$  and  $\lambda_{\mathrm{L1}} = 5$  for the  $\mathcal{L}_{\mathrm{box}}$  loss. The numbers of decoder layers for the text-guided object detection, relation matching, and target identification networks are set to  $N_d = 3$ ,  $N_r = 2$ , and  $N_g = 3$ , respectively. Following the previous work [39], we apply rotation data augmentation for the 3D point cloud scenes and augment supervision with detection prompts during training. We measure accuracy using Acc@0.25 for both Nr3D and Sr3D datasets, where a predicted bounding box is considered correct if its IoU with the ground truth target exceeds 0.25. For the ScanRefer dataset [7], we employ two metrics: Acc@0.25 and Acc@0.5.
4.3 Comparison with State-of-the-Art Methods
Table 1 presents the comparative results of our method with the current state-of-the-art methods on the  $\mathrm{Nr3D}$ , Sr3D, and ScanRefer datasets. Our method consistently outperforms the previous methods. For the  $\mathrm{Nr3D}$  and Sr3D datasets, it is standard practice to use ground-truth object boxes as object proposals to infer the referred object. However, considering that ground-truth boxes are usually unavailable in practical scenarios, our model is designed to locate objects of interest using a dedicated detection network. Thus, for a fair comparison, we quote the evaluation results from BUTD-DETR [39], where previous methods are evaluated with detected object proposals by a pretrained 3D object detector [40]. The quoted results are marked with  $\dagger$  and we denote this evaluation setup as "det" in Table 1. On the  $\mathrm{Nr3D}$  and Sr3D datasets, as shown in Table 1, our method exhibits
Table 1: Comparison with the state-of-the-art methods on Nr3D, Sr3D, and ScanRefer.
Method	Nr3D Acc@0.25 (det)	Sr3D Acc@0.25 (det)	ScanRefer Acc@0.25	Acc@0.5
ReferIt3DNet [6]	24.0†	27.7†	26.4	16.9
ScanRefer [7]	-	-	35.5	22.4
TGNN [8]	-	-	37.4	29.7
InstanceRefer [9]	29.9†	31.5†	40.2	32.9
FFL-3DOG [41]	-	-	41.3	34.0
LanguageRefer [13]	28.6†	39.5†	-	-
3DVG-Transformer [11]	-	-	45.9	34.5
SAT 2D [12]	31.7†	35.4†	44.5	30.1
BUTD-DETR [39]	43.3	52.1	52.2	39.8
CORE-3DVG	49.57	54.30	56.77	43.84
Table 2: Comparison with the state-of-the-art methods on different subsets of ScanRefer.
Method	Unique	Multiple	Overall
Acc@0.25	Acc@0.5	Acc@0.25	Acc@0.5	Acc@0.25	Acc@0.5
ReferIt3DNet[6]	53.8	37.5	21.0	12.8	26.4	16.9
ScanRefer[7]	63.0	40.0	28.9	18.2	35.5	22.4
TGNN[8]	68.6	56.8	29.8	23.2	37.4	29.7
InstanceRefer[9]	77.5	66.8	31.3	24.8	40.2	32.9
FFL-3DOG[41]	78.8	67.9	35.2	25.7	41.3	34.0
3DVG-Transformer[11]	77.2	58.5	38.4	28.7	45.9	34.5
SAT 2D[12]	-	-	-	-	44.5	30.1
BUTD-DETR[39]	84.2	66.3	46.6	35.1	52.2	39.8
CORE-3DVG	84.99	67.09	51.82	39.76	56.77	43.84
significant performance advantages over the previous methods. Compared with the recent BUTD-DETR, which employs a larger transformer encoder and decoder network, our method achieves absolute improvements of  $6.27\%$  and  $2.20\%$  on the  $\mathrm{Nr3D}$  and Sr3D datasets, respectively.
Table 2 provides a detailed performance comparison between our method and previous approaches on the ScanRefer dataset. Our method shows superior performance across various test settings and evaluation metrics. Notably, on the challenging "Multiple" subset, where multiple objects share the same category as the target objects, our method achieves  $51.82\%$  in Acc@0.25, surpassing the leading BUTD-DETR [39] by an appreciable margin  $5.22\%$ . Our explicit modeling of contextual information enables better handling of such challenging scenarios. For the simpler "Unique" subset, where the target object is unique, we also achieve higher or comparable results compared to previous methods. Our overall accuracy on the ScanRefer dataset reaches  $56.77\%$  (Acc@0.25) and  $43.84\%$  (Acc@0.5), respectively, outperforming all the previous methods.
In this section, we conduct the ablation studies on the Nr3D [6] dataset to validate our method.
Learning Contextual Objects and Relations. Table 3 presents the ablation study on the proposed explicit learning of contextual objects and relations. The first row of Table 3 represents the baseline model, which achieves an overall accuracy of  $34.66\%$  (Acc@0.25). Based on this baseline, we first introduce the learning of contextual objects, resulting in a significant improvement of  $9.76\%$  in overall accuracy. The improvements are consistently observed across all test subsets, as shown in the second row of Table 3. While the contextual relations are not modeled, the contextual objects alone provide crucial contextual information for target inference. In the third row of the table, we separately introduce the learning of contextual relations and observe an improvement of  $2.67\%$ . The "View-dep" subset exhibits the largest improvement of  $6.14\%$ , highlighting the effectiveness of relation learning for context modeling. Finally, by incorporating the learning of both contextual objects and relations, we achieve an overall accuracy of  $49.57\%$ . This surpasses the baseline model by  $14.91\%$  and achieves the best performance among these ablation variants, as shown in the last row of Table 3.
Table 3: The ablation study of learning contextual objects and relations.
Learning context. objects	Learning context. relations	Easy	Hard	View-dep.	View-indep.	Overall
✓	40.50	29.04	32.13	35.59	34.66
49.85	39.20	40.75	45.78	44.42
✓	43.34	31.55	38.27	36.98	37.33
✓	✓	53.91	45.39	48.19	50.07	49.57
Table 4: The ablation study of pseudo-label self-generation strategy.
Method	Easy	Hard	View-dep.	View-indep.	Overall
-	40.50	29.04	32.13	35.59	34.66
+ Supervised learning on labeled noun phrases	45.05	31.87	33.17	40.23	38.33
+ Learning pseudo labels for unlabeled noun phrases	49.85	39.20	40.75	45.78	44.42
Table 5: The ablation study of initial spatial relation features.
Distance & Orientation	Volume & Dimension	Perspective-related relations	Easy	Hard	View-dep.	View-indep.	Overall
✓	49.85	39.20	40.75	45.78	44.42
52.38	40.99	43.53	47.70	46.57
✓	51.62	40.70	42.94	47.20	46.05
✓	52.49	42.58	46.46	47.81	47.44
✓	✓	✓	52.55	43.42	46.85	48.28	47.90
Pseudo-Label Self-Generation Strategy for Contextual Object Learning. In Table 4, we investigate the pseudo-label generation strategy used for contextual object learning. First, we only employ the detection loss defined in Equation 4 to learn the objects corresponding to labeled noun phrases. The second row of Table 4 shows an overall accuracy improvement of  $3.67\%$  compared to the baseline model in the first row. This suggests that augmenting the textual diversity for object detection training also contributes to improved visual grounding. After that, we utilize the pseudo labels generated for unlabeled noun phrases to guide the learning of contextual objects. The third row of Table 4 demonstrates that this technique boosts the model's accuracy from  $38.33\%$  to  $44.42\%$ $(+6.09\%)$  and exhibits significant performance enhancements across all test subsets. These results further emphasize the importance of learning contextual objects, as it allows the model to effectively capture contextual information for target inference.
Initial Spatial Relation Features. We develop a diverse set of spatial relations between objects as initial features for modeling contextual relations. Table 5 presents an ablation study of these relation features. The first row of Table 5 shows an overall accuracy of  $44.42\%$  when the initial spatial relation features are not used. Then, we individually incorporate the relation features related to "Distance & Orientation", "Volume & Dimension", and "Perspective-related relations", improving the overall accuracy by  $2.15\%$ ,  $1.63\%$ , and  $3.35\%$ , respectively. Notably, the perspective-related relation feature achieves the greatest improvement, especially in the "View-dep." test subset (from  $40.75\%$  to  $46.46\%$ ), highlighting its effectiveness in reasoning textual descriptions associated with specific viewpoints. Finally, combining all these relation features in the last row of Table 5 yields the highest accuracy of  $47.90\%$ .
Weakly-Supervised Contextual Relation Learning. In Table 6, we conduct an ablation study on the learning of contextual relations. The first two rows of Table 6 show an overall accuracy improvement of  $3.48\%$  when directly using the initial spatial relation features. However, as shown in the third row of Table 6, the relation matching network alone fails to improve performance without the inclusion of the weakly-supervised losses in Equation 6. By incorporating weakly-supervised learning of contextual relations, we achieve an overall accuracy improvement of  $1.77\%$  and observe appreciable performance advancements across all test subsets, as depicted in the last row of Table 6. These results clearly demonstrate the necessity of the weakly-supervised learning in effectively modeling contextual relations.
Table 6: The ablation study of contextual relation learning.
Initial spatial relation features	Relation matching network	Weakly-supervised learning losses	Easy	Hard	View-dep.	View-indep.	Overall
49.85	39.20	40.75	45.78	44.42
✓	52.55	43.42	46.85	48.28	47.90
✓	✓	52.55	43.24	46.90	48.13	47.80
✓	✓	✓	53.91	45.39	48.19	50.07	49.57
(a) The lamp next to the red curtains.
(b) Choose the monitor on the desk that is close to the coats hanging up.
(c) Blue empty kid chair by green table right underneath the animals curtains
(d) The table is white with two monitors on it.
Figure 3: 3D visual grounding results with associated contextual objects and relations.
4.5 Visualization
In Figure 3, we visualize the visual grounding results of our method for various descriptions and 3D scenes. We use green bounding boxes to represent the detected target objects and red bounding boxes to indicate the contextual objects that exhibit relation matching scores greater than 0.4. The results illustrate the efficacy of our method in comprehending contextual information within the text to infer the referred target object. For example, in Figure 3(a), when given the description "The lamp next to the red curtains", our model successfully locates the correct lamp and the adjacent red curtains, thereby avoiding misclassification of the lamp near the bed. This explicit utilization of contextual information enables a more reliable inference of the target object.
In this paper, we presented a 3D visual grounding framework that focuses on learning contextual objects and relations. The proposed framework consists of three sequential modules: text-guided object detection, relation matching, and target identification networks. The innovative techniques of pseudo-label self-generation and weakly-supervised learning facilitate the learning of contextual objects and relations. We verify the effectiveness of contextual objects and relations learning through extensive experiments and achieve leading performances on multiple benchmarks, including Nr3D, Sr3D, and ScanRefer.
Despite the remarkable performance achieved, the generalization ability of our method is still limited since our model has been only trained on datasets with limited object semantics and corpus. The performance of our algorithms could be influenced by the types of devices used to capture the point clouds and the types of 3D scenes. In the future, we plan to further dig into the generalization ability of 3D visual grounding frameworks and extend our method to more practical and realistic scenarios.
Acknowledgements. This work is supported by the National Key R&D Program of China (No. 2022ZD0118501), the Beijing Natural Science Foundation (JQ21017, L223003, M22005, 4224091), the Natural Science Foundation of China (Grant No. 61972397, 62222206, 62036011, 62192782, 61721004, U2033210, 62372451, 62192785, 62372082, 62202469), the Major Projects of Guangdong Education Department for Foundation Research and Applied Research (2018KZDXM066), Guangdong Provincial University Innovation Team Project (2020KCXTD045).
[36] Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2017.
[37] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[38] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692, 2019.
[39] Ayush Jain, Nikolaos Gkanatsios, Ishita Mediratta, and Katerina Fragkiadaki. Bottom up top down detection transformers for language grounding in images and point clouds. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXVI, pages 417-433. Springer, 2022.
[40] Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2949-2958, 2021.
[41] Mingtao Feng, Zhen Li, Qi Li, Liang Zhang, XiangDong Zhang, Guangming Zhu, Hui Zhang, Yaonan Wang, and Ajmal Mian. Free-form description guided 3d visual graph network for object grounding in point cloud. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3722-3731, 2021.