SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Li Yang $^{1*}$ , Chunfeng Yuan $^{1\dagger*}$ , Ziqi Zhang $^{1}$ , Zhongang Qi $^{2}$ , Yan Xu $^{3}$ , Wei Liu $^{1}$ , Ying Shan $^{2}$ , Bing Li $^{1}$ , Weiping Yang $^{4}$ , Peng Li $^{5,6}$ , Yan Wang $^{5,6}$ , Weiming Hu $^{1,7,8}$
$^{1}$ State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA
$^{2}$ ARC Lab, Tencent PCG,  $^{3}$ The Chinese University of Hong Kong
4Education Management Information Center, Ministry of Education 5Alibaba Group
$^{6}$ Zhejiang Linkheer Science And Technology Co., Ltd.
<sup>7</sup>School of Artificial Intelligence, University of Chinese Academy of Sciences
$^{8}$ School of Information Science and Technology, ShanghaiTech University
3D visual grounding, the task of identifying visual objects in 3D scenes based on natural language inputs, plays a critical role in enabling machines to understand and engage with the real-world environment. However, this task is challenging due to the necessity to capture 3D contextual information to distinguish target objects from complex 3D scenes. The absence of annotations for contextual objects and relations further exacerbates the difficulties. In this paper, we propose a novel model, CORE-3DVG, to address these challenges by explicitly learning about contextual objects and relations. Our method accomplishes 3D visual grounding via three sequential modular networks, including a text-guided object detection network, a relation matching network, and a target identification network. During training, we introduce a pseudo-label self-generation strategy and a weakly-supervised method to facilitate the learning of contextual objects and relations, respectively. The proposed techniques allow the networks to focus more effectively on referred objects within 3D scenes by understanding their context better. We validate our model on the challenging Nr3D, Sr3D, and ScanRefer datasets and demonstrate state-of-the-art performance. Our code will be public at https://github.com/ yangli18/CORE-3DVG.
Teaching machines to interpret and interact with the real world through vision and language has been a long-standing pursuit in artificial intelligence research. An important and fundamental task towards this goal is visual grounding, which involves detecting the referred visual object based on natural language input. While previous works have largely focused on visual grounding within 2D images [1, 2, 3, 4, 5], there has been a growing interest in extending this task to point clouds of 3D scenes, giving rise to 3D visual grounding [6, 7].
In 3D scenes, objects exhibit diverse spatial distributions, and it is common for objects of the same category to coexist. To clearly describe the target object, language descriptions often provide the contextual information, i.e. the contextual objects as well as their relations to the target object within the environment. For instance, in Figure 1(a), the text describes the target chair by specifying its relative position to the door and table. The interpretation of such contextual information from textual descriptions to visual representations is crucial for 3D visual grounding.
*Equal contribution, †Corresponding author
Figure 1: (a) Language descriptions often provide the necessary contextual information to accurately depict the target objects. (b) Our proposed framework for 3D visual grounding.
Existing methods for 3D visual grounding typically involve matching a large set of object proposals with the textual description. These methods either use graph networks [6, 8, 9] or transformer architectures [10, 11, 12, 13, 14] to implicitly model contextual information between object proposals while performing cross-modal fusion with the text. Despite the effectiveness, their optimization objectives mainly focus on the final matching of object proposals with the textual description, without paying special attention to the alignment with the contextual information in the text. Consequently, important contextual cues may be overlooked during target inference, resulting in less reliable results.
In contrast, humans naturally possess a remarkable ability to infer the target object based on a language description. We can easily capture the relevant contextual information in the 3D scene to reason about the target indicated by the description in a two-step manner: Firstly, we locate all the objects mentioned in the text. Then we figure out the relations between these objects and compare them with the textual description, which enables us to ultimately identify the target referred to in the description. Motivated by this, we improve 3D visual grounding by mimicking this human reasoning process. Specifically, (1) we explicitly detect all mentioned contextual objects in addition to the target object, focusing the network on these objects to capture contextual information for target inference. As most current datasets do not have annotations for contextual objects, we propose a pseudo-label self-generation strategy, which leverages object semantics learned at the phrase level to generate learning labels for contextual objects. (2) We explicitly model the contextual relations between the candidate targets and the contextual objects as features of an adjacency matrix, which initially encode various spatial relations of objects. To facilitate the learning of contextual relations, we introduce a weakly-supervised learning loss that explicitly distinguishes related and unrelated relations for the target object.
The above ideas are embodied as a novel transformer-based framework that explicitly exploits Contextual Objects and RElations for 3D Visual Grounding (CORE-3DVG). Our framework, illustrated in Figure 1(b), comprises three modular networks that divide the 3D visual grounding process into three inference steps. Firstly, a text-guided object detection network performs joint detection of both the target and contextual objects. Next, a relation matching network models the representations of their contextual relations. Finally, the target identification network pinpoints the referred object based on the modeled contextual relations.
To summarize, our contributions are threefold:
- We propose to explicitly model and learn the objects and relations mentioned in the context to focus the network on contextual information for 3D visual grounding.
- We introduce a novel framework for 3D visual grounding, dubbed CORE-3DVG, which tackles the task of 3D visual grounding through three sequential modules. Our framework first detects all mentioned objects in the scene, then models their contextual relations, and finally performs context-based target inference.
- We evaluate our method on the challenging benchmarks including Nr3D, Sr3D, and ScanRefer. Our method outperforms the previous state-of-the-art counterparts by significant margins. Extensive experiments validate the efficacy of our proposed techniques.
2D/3D Visual Grounding. Visual grounding aims to locate target objects in 2D images or 3D point clouds based on the language input. Existing methods are generally categorized as one-stage methods [15, 16, 17, 4, 18] and two-stage methods [1, 19, 3, 20, 21, 22, 23, 24, 25, 26, 6, 7]. One-stage approaches fuse textual features with visual representations to generate target predictions in one shot. Two-stage methods follow a two-step detection and matching process: they first leverage the off-the-shelf detectors [27, 28, 29, 30] to generate object proposals, which are then matched with language input to select the best match as the estimated target.
The prevalent methods in 3D visual grounding adopt a two-stage scheme. Early works [6, 8, 9] use graph-based methods to model spatial relations among object proposals. With the rise of transformers [31], recent works [10, 12, 13, 14, 18] have adopted transformers for feature fusion and cross-modal fusion. Among them, SAT [12] incorporates 2D visual grounding to improve the performance of 3D visual grounding. Multi-view transformer [14] models different views in visual feature modeling.
Context Modeling for Visual Grounding. Modeling contextual information is important for visual grounding, especially for 3D visual grounding. While early works use graph neural networks or transformers to model contextual information, they fall short in encoding various spatial relations between objects. Recent 3DVG-transformer [11] and ViL3DRel [32] explicitly encode the objects' distances or orientations into their self-attention computation to better model 3D spatial relations. In contrast to prior works, our method focuses on the explicit learning of contextual objects and relations and exploiting this information for target inference.
Our method focuses on the learning of contextual objects and relations, and exploits such information to achieve reliable 3D visual grounding. Figure 2 illustrates the overall framework. Given a 3D scene's point cloud and a language description, we initially encode their features into  $F_{v} \in \mathbb{R}^{M \times C}$  and  $F_{l} \in \mathbb{R}^{L \times C}$ , respectively, where  $M$  and  $L$  represent the number of features and  $C$  denotes the feature dimension. Based on these encoded features, 1) A text-guided object detection network first detects semantically related objects in the 3D scene, including target and contextual objects. 2) Then, we construct the spatial relation features in a pairwise manner for these detected objects and input them into the relation matching network for modeling contextual relations. 3) Finally, utilizing the modeled contextual features, the target identification network can effectively match the detected objects with the textual information to find the referred object.
3.1.1 Text-Guided Object Detection Network
The object detection network, as illustrated in Figure 2(a), consists of  $N_{d}$  transformer decoder layers. The network is initiated with a set of  $K$  object queries  $[o_1, o_2, \dots, o_K]^{\mathrm{T}} \in \mathbb{R}^{K \times C}$ , which are generated from the point cloud features  $F_{v}$  based on their correlation with the textual features  $F_{l}$  (see the supplementary material for more details). In each decoder layer, the initial object queries are first input to a self-attention layer to enable feature interaction between candidate objects. The queries are then passed through two cascaded cross-attention layers to gather textual and visual features successively. Next, a feed-forward network (FFN) updates the object queries. For the resulting  $K$  object queries, two FFNs are applied to predict their object bounding boxes  $[\hat{b}_1, \hat{b}_2, \dots, \hat{b}_K]^{\mathrm{T}} \in \mathbb{R}^{K \times 6}$  and confidence scores  $[\hat{p}_1, \hat{p}_2, \dots, \hat{p}_K]^{\mathrm{T}} \in \mathbb{R}^{K \times 1}$ , respectively.
To avoid duplicate object detections, we employ non-maximum suppression (NMS) and retain the top  $T$  scoring detections. We represent the bounding boxes of the retained detections as  $[\bar{b}_1,\bar{b}_2,\dots ,\bar{b}_T]^{\mathrm{T}}\in \mathbb{R}^{T\times 4}$  and their corresponding object queries as  $[\bar{o}_1,\bar{o}_2,\dots ,\bar{o}_T]^{\mathrm{T}}\in \mathbb{R}^{T\times C}$ . These detections correspond to the most relevant objects detected for the language description, including both target and contextual objects. For the training of this detection network, we propose a pseudo-label self-generation strategy to facilitate learning of unlabeled contextual objects, which is detailed in Section 3.2.
Figure 2: The overview of our framework. Our framework comprises three modular networks that divide the 3D visual grounding process into three inference steps.
3.1.2 Relation Matching Network
The relation matching network is developed to model the target object's contextual relations that match the textual description. Specifically, for each pair of detected objects  $(\bar{b}_i,\bar{b}_j)$ , we devise multiple spatial relation features that consider different aspects, including distance & orientation, volume & dimension, and perspective (see the supplementary material for details). This results in  $T\times T$  spatial relation features  $[r_{i,j}^{s}]_{1\leq i,j\leq T}$  for the  $T$  detected objects, where  $r_{i,j}^{s}\in \mathbb{R}^{C_1}$  represents the relation feature with  $\bar{b}_i$  as the candidate target and  $\bar{b}_j$  as the contextual object. We perform linear projection on each feature  $r_{i,j}^{s}$  and its corresponding contextual object query  $\bar{o}_j$ , and then concatenate them as the initial contextual relation feature:
$$
r _ {i, j} ^ {0} = \operatorname {C o n c a t} \left(W _ {1} \cdot r _ {i, j} ^ {s}, W _ {2} \cdot \bar {o} _ {j}\right), \tag {1}
$$
where  $W_{1}\in \mathbb{R}^{C / 2\times C_{1}}$  and  $W_{2}\in \mathbb{R}^{C / 2\times C}$  are the linear projection weights.
As depicted in Figure 2(b), the relation matching network takes the contextual relation features  $[r_{i,j}^{0}]_{1\leq i,j\leq T}$  as input and applies  $N_{r}$  transformer decoder layers to model contextual relations that match the textual description. To achieve this, each decoder layer contains a self-attention layer for intra-modal information interaction among relation features, a cross-attention layer on textual features for cross-modal fusion, and an FFN to transform the relation features. The relation matching network outputs the updated relation features  $[r_{i,j}]_{1\leq i,j\leq T}$ , and we apply a two-layer FFN to generate their respective matching scores w.r.t. the textual information, denoted as  $[m_{i,j}]_{1\leq i,j\leq T}$ . A weakly-supervised learning approach is proposed to supervise these matching scores and thereby learn the contextual relations associated with the referred object (detailed in Section 3.3).
3.1.3 Target Identification Network
We propose the target identification network to find the referred object with fine-grained alignment to the textual description by leveraging the information of the detected objects and their contextual relations. As shown in Figure 2(c), with the detected objects as candidates, we input their object queries  $\left[\bar{o}_1,\bar{o}_2,\dots ,\bar{o}_T\right]^{\mathrm{T}}\in \mathbb{R}^{T\times C}$  into the target identification network. This network aligns the object queries with the textual features via  $N_{g}$  transformer decoder layers. Each decoder layer includes a self-attention layer and a cross-attention layer to enable intra-modal and cross-modal feature fusion, followed by an FFN layer to refine the object query representations. We take the output object queries from the last decoder layer and apply a two-layer FFN to estimate their confidence scores as the target object, denoted as  $[p_1^{\star},p_2^{\star},\dots ,p_T^{\star}]^{\mathrm{T}}\in \mathbb{R}^{T\times 1}$ .
To exploit contextual relations for target discrimination, we propose a context-aware self-attention mechanism incorporating relation features to better model contextual information among candidate objects. Specifically, the self-attention layer contains multiple attention heads to process the input object queries. In each attention head, the object queries are first linearly projected as the query, key,
and value embeddings, represented by  $Q, K, V \in \mathbb{R}^{T \times d_k}$ , respectively ( $d_k$  is the dimension of the embeddings). The conventional self-attention method computes the attention of the  $i$ -th object to the  $j$ -th object by correlating their query and key embeddings, i.e.  $\text{softmax}_j(Q(i)K(j)^{\mathrm{T}} / \sqrt{d_k})$ . Our self-attention mechanism extends this attention computation by considering the modeled contextual relations between objects. To accomplish this, we calculate an additional query embedding of the object queries as  $Q_r \in \mathbb{R}^{T \times d_k}$ , and perform linear projection on the contextual relation features  $[r_{i,j}]_{1 \leq i,j \leq T}$  to obtain their key and value embeddings as  $R_k, R_v \in \mathbb{R}^{T \times T \times d_k}$ , respectively. The attention of the  $i$ -th object to the  $j$ -th object is then computed as follows:
$$
\operatorname {a t t n} _ {i, j} = \operatorname {s o f t m a x} _ {j} \left(\frac {Q (i) K (j) ^ {\mathrm {T}} + Q _ {r} (i) R _ {k} (i , j) ^ {\mathrm {T}}}{\sqrt {2 \cdot d _ {k}}}\right). \tag {2}
$$
The resulting self-attention matrix allows each object to focus more on the objects that have important contextual relations between them. Moreover, we aggregate the value embeddings of both the object queries and their associated relation features for attention-based feature fusion:
$$
\operatorname {o u t} _ {i} = \sum_ {j} \operatorname {a t t n} _ {i, j} \cdot (V (j) + R _ {v} (i, j)). \tag {3}
$$
In this manner, each attention head of the self-attention layer achieves context-based feature aggregation, and the outputs from all attention heads are concatenated to update the input object queries. Consequently, the updated object queries can explicitly encode information about related contextual objects and relations. We incorporate the context-aware self-attention layer in each decoder layer of the target identification network, enabling a more effective comparison of modeled object queries with textual information to discern the referred object.
3.2 Pseudo-Label Self-Generation for Contextual Object Learning
Our method starts with a text-guided detection network to detect all the objects mentioned in the textual description. However, since the contextual objects associated with the referred targets are typically unlabeled, the object detection network lacks the requisite supervision for effective training. To address this issue, we propose a pseudo-label self-generation strategy that facilitates the learning of contextual objects within the network.
As shown in Figure 2(a), we first extract the noun phrases for physical objects from the input text and divide them into the labeled noun phrase (of the target object) and the unlabeled noun phrases (of the other contextual objects). We input these extracted noun phrases into the object detection network along with the same initial object queries and point cloud features. For the labeled noun phrase, the network produces  $K$  detected bounding boxes  $[\tilde{b}_1, \tilde{b}_2, \dots, \tilde{b}_K]^{\mathrm{T}} \in \mathbb{R}^{K \times 6}$  and confidence scores  $[\tilde{p}_1, \tilde{p}_2, \dots, \tilde{p}_K]^{\mathrm{T}} \in \mathbb{R}^{K \times 1}$ . Given the ground-truth bounding box of the target object as  $b_{\mathrm{gt}} \in \mathbb{R}^6$ , we find the best matching detection result  $\{\tilde{b}_{\eta}, \tilde{p}_{\eta}\}$  (with index  $\eta$ ) as the positive sample. Then we define the following losses to supervise the detection results of the labeled noun phrase:
$$
\mathcal {L} _ {\mathrm {n l o c}} = \mathcal {L} _ {\mathrm {b o x}} \left(b _ {\mathrm {g t}}, \tilde {b} _ {\eta}\right),
$$
$$
\mathcal {L} _ {\mathrm {n c l s}} = - \log (\tilde {p} _ {\eta}) - \sum_ {k \neq \eta} \lambda_ {\mathrm {n}} \cdot \log (1 - \tilde {p} _ {k}), \tag {4}
$$
where the loss function  $\mathcal{L}_{\mathrm{box}}$  consists of the GIoU loss [33] and the L1 loss, i.e.  $\mathcal{L}_{\mathrm{box}}(\cdot ,\cdot) = \lambda_{\mathrm{giou}}\mathcal{L}_{\mathrm{giou}}(\cdot ,\cdot) + \lambda_{\mathrm{L1}}\mathcal{L}_{\mathrm{L1}}(\cdot ,\cdot)$  ( $\lambda_{\mathrm{giou}}$  and  $\lambda_{\mathrm{L1}}$  are hyper-parameters to balance the two losses).
During training, based on the supervised learning for labeled noun phrases, we develop the network's ability to detect objects for various noun phrases. We thereby apply the detection network to the unlabeled noun phrases and utilize the output detection results to create pseudo labels for contextual objects. In particular, for each of the  $U$  unlabeled noun phrases, we process the detection results with NMS and select the  $H$  detections with the highest scores. We use  $b_{i,j}^{*}\in \mathbb{R}^{6}$  and  $p_{i,j}^{*}\in \mathbb{R}$  to denote the bounding box and unnormalized score of the  $j$ -th detection for the  $i$ -th unlabeled noun phrase. This gives  $U\times H$  object detections  $\{(b_{i,j}^{*},p_{i,j}^{*})\}_{1\leq i\leq U,1\leq j\leq H}$  to serve as the pseudo labels for contextual objects. As shown in Figure 2(a), we combine these pseudo labels with the target's ground-truth label  $b_{\mathrm{gt}}$  to supervise the network's detection results  $\{(\hat{b}_i,\hat{p}_i)\}$  for the entire sentence (see Section 3.1.1). Similar to DETR [34], we use the Hungarian algorithm to find the best matching detections for the combined labels, where  $\tau$  and  $\sigma_{i,j}$  denote the matching indexes for the target label
and each pseudo label, respectively. Then the supervision losses on the detection results are computed as:
$$
\mathcal {L} _ {\mathrm {l o c}} = \mathcal {L} _ {\mathrm {b o x}} \left(b _ {\mathrm {g t}}, \hat {b} _ {\tau}\right) + \sum_ {i = 1} ^ {U} \sum_ {j = 1} ^ {H} w _ {i, j} \cdot \mathcal {L} _ {\mathrm {b o x}} \left(b _ {i, j} ^ {*}, \hat {b} _ {\sigma_ {i, j}}\right), \tag {5}
$$
$$
\mathcal {L} _ {\mathrm {c l s}} = - \log (\hat {p} _ {\tau}) - \sum_ {i = 1} ^ {U} \sum_ {j = 1} ^ {H} w _ {i, j} \cdot \log (\hat {p} _ {\sigma_ {i, j}}) - \sum_ {\substack {k \neq \tau \\ k \notin \{\sigma_ {i, j} \}}} \lambda_ {n} \cdot \log (1 - \hat {p} _ {k}), \tag{5}
$$
where  $w_{i,j} = \frac{\exp(p_{i,j}^{*})}{\sum_{j=1}^{H}\exp(p_{i,j}^{*})} \cdot \text{sigmoid}(p_{i,j}^{*})$  denotes the learning weight for each pseudo label. We use  $w_{i,j}$  to encourage learning pseudo labels of higher scores under each unlabeled noun phrase while also considering the individual confidence of each pseudo label to mitigate the impact of learning unreliable pseudo labels. With Equation (5), the network learns to understand the object phrases within the input text for associated object detection. During inference, the network detects target and contextual objects based on the input text without the need to manually extract their noun phrases.
3.3 Weakly-Supervised Contextual Relation Learning
Our relation matching network outputs a score matrix  $[m_{i,j}]_{1\leq i,j\leq T}$  for all pairs of  $T$  detected objects, representing the matching scores between their contextual relations and the textual description. However, since there are no explicit labels for contextual relation learning, we propose a weakly-supervised method to find the contextual relations associated with the referred object. The overall optimization objectives for contextual relation learning are:
1. For the target object, there exists at least one contextual relation with a high matching score.
2. For the related contextual object, its contextual relations should have lower matching scores when paired with objects that are not the target.
To accomplish this, we first match the  $T$  detected objects with the ground truth target  $b_{\mathrm{gt}}$  and find the best matching detection as  $\bar{b}_{\alpha}$  (indexed by  $\alpha$ ), which the model shall later infer as the detected target. Then we take the  $\alpha$ -th row from the score matrix  $[m_{i,j}]_{1 \leq i,j \leq T}$ , obtaining  $[m_{\alpha,j}]_{j \neq \alpha}$  as the contextual relation matching scores between the detected target  $\bar{b}_{\alpha}$  and other objects. From these scores, we find the maximum score  $m_{\alpha,\beta}$  with column index  $\beta = \arg \max_j ([m_{\alpha,j}]_{j \neq \alpha})$ . We thereby consider the object  $\bar{b}_{\beta}$  as the most related contextual object for the target  $\bar{b}_{\alpha}$ . Following the optimization objectives 1 and 2, we devise the following losses to supervise the matching scores of the  $\alpha$ -th row and  $\beta$ -th column:
$$
\mathcal {L} _ {\text {r e l 1}} = \sum_ {\substack {j = 1 \\ j \neq \alpha}} ^ {T} \max (0, \Delta + m _ {\alpha , j} - m _ {\alpha , \beta}), \tag{6}
$$
$$
\mathcal {L} _ {\mathrm {r e l 2}} = \sum_ {i = 1} ^ {T} \max (0, \Delta + m _ {i, \beta} - m _ {\alpha , \beta}),
$$