SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yifan Zhang $^{1}$ , Qijian Zhang $^{1}$ , Junhui Hou $^{1*}$ , Yixuan Yuan $^{2*}$ , and Guoliang Xing $^{2}$ $^{1}$ City University of Hong Kong,  $^{2}$ The Chinese University of Hong Kong {yzhang3362-c, qijizhang3-c}@my.cityu.edu.hk; jh.hou@cityu.edu.hk; yxyuan@ee.cuhk.edu.hk; glxing@ie.cuhk.edu.hk
To achieve reliable and precise scene understanding, autonomous vehicles typically incorporate multiple sensing modalities to capitalize on their complementary attributes. However, existing cross-modal 3D detectors do not fully utilize the image domain information to address the bottleneck issues of the LiDAR-based detectors. This paper presents a new cross-modal 3D object detector, namely UPIDet, which aims to unleash the potential of the image branch from two aspects. First, UPIDet introduces a new 2D auxiliary task called normalized local coordinate map estimation. This approach enables the learning of local spatial-aware features from the image modality to supplement sparse point clouds. Second, we discover that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module. Extensive experiments and ablation studies validate the effectiveness of our method. Notably, we achieved the top rank in the highly competitive cyclist class of the KITTI benchmark at the time of submission. The source code is available at https://github.com/Eaphan/UPIDet.
In recent years, there has been increasing attention from both academia and industry towards 3D object detection, particularly in the context of autonomous driving scenarios [11, 57, 56, 58]. Two dominant data modalities - 3D point clouds and 2D RGB images - demonstrate complementary properties, with point clouds encoding accurate structure and depth cues but suffering from sparsity, incompleteness, and non-uniformity. On the other hand, RGB images convey rich semantic features and has already developed powerful learning architectures [10], but pose challenges in reliably modeling spatial structures. Despite the efforts made by previous studies [43, 40] devoted to cross-modal learning, how to reasonably mitigate the substantial gap between two modalities and utilize the information from the image domain to complement single-modal detectors remains an open and significant issue.
Currently, the performance of single-modal detectors is limited by the sparsity of observed point clouds. As shown in Figure 1 (b), the performance of detectors on objects with fewer points (LEVEL_2) is significantly lower than that on objects with denser points (LEVEL_1). And Fig.1 (a) shows the distribution of point clouds in a single object for the Waymo dataset. It can be observed that a substantial proportion of objects have less than 50 point clouds, making it challenging to deduce the 3D bounding boxes with seven degrees of freedom from very sparse point clouds, as discussed in detail in Sec.3. While the observed point cloud can be sparse due to heavy occlusion or low reflectivity material, their contour and appearance can still be clear in RGB images (as shown in the example in Fig.1 (c)). Despite previous work exploring the use of semantic image features to enhance features extracted from LiDAR [40, 12], they have not fundamentally improved the spatial
(b)
Figure 1: (a) Distribution of number of points in a single object in Waymo Dataset.(b) Performance of the Second on Waymo [38]. (c) Point cloud and image of a black car and part of their correspondence.
representation limited by the sparsity of the point cloud. There is a class of work that attempts to generate pseudo LiDAR points as a supplement to sparse point clouds but these models commonly rely on depth estimators trained on additional data [43].
On the other hand, the performance of LiDAR-based detection frameworks is limited by the inherent challenges of processing irregular and unordered point clouds, which makes it difficult for neural network backbones to learn effectively from 3D LiDAR data. Consequently, the representation capability of these backbone networks is still relatively insufficient, despite the performance boost achieved by previous cross-modal methods[12, 40] that incorporate discriminative 2D semantic information into the 3D detection pipeline. However, there is no evidence that these methods actually enhance the representation capability of the 3D LiDAR backbone network, which is also one of the most critical influencing factors.
This paper addresses the first issue mentioned above by leveraging local-spatial aware information from RGB images. As depicted in Fig. 1 (c), predicting the 3D bounding box of a reflective black car from the sparse observed point cloud is challenging. By incorporating image information, we can not only distinguish the foreground points but also determine the relative position of the point cloud within the object, given the known 3D-to-2D correspondence. To learn local spatial-aware feature representations, we introduce a new 2D auxiliary task called normalized local coordinate (NLC) map estimation, which provides the relative position of each pixel inside the object. Ultimately, we expect to improve the performance of the cross-modal detector by utilizing both the known global coordinates of points and their estimated relative position within an object.
On the other hand, the depth of point cloud serves as a complementary source to RGB signals, providing 3D geometric information that is resilient to lighting changes and aids in distinguishing between various objects. In light of this, we present a succinct point-to-pixel feature propagation module that allows 3D geometric features extracted from LiDAR point clouds to flow into the 2D image learning branch. Unlike the feature-level enhancements brought by existing pixel-to-point propagation, this method enhances the representation ability of 3D LiDAR backbone networks. Remarkably, this is an interesting observation because the back-propagated gradients from the 2D image branch built upon established 2D convolutional neural networks (CNNs) with impressive learning ability can effectively boost the 3D LiDAR branch.
We conduct extensive experiments on the prevailing KITTI [8] benchmark dataset. Compared with state-of-the-art single-modal and cross-modal 3D object detectors, our framework achieves remarkable performance improvement. Notably, our method ranks  $\mathbf{1}^{\mathrm{st}}$  on the cyclist class of KITTI 3D detection benchmark<sup>2</sup>. Conclusively, our main contributions are three-fold:
- We demonstrate that the representational capability of the point cloud backbone can be enhanced through the gradients backpropagated from the training objectives of the image branch, utilizing a succinct and effective point-to-pixel module.
- We reveal the importance of specific 2D auxiliary tasks used for training the 2D image learning branch which has not received enough attention in previous works, and introduce 2D NLC
2www.cvlabs.net/datasets/kitti/eval_object.php?obj_benchmark=3d
map estimation to facilitate learning spatial-aware features and implicitly boost the overall 3D detection performance.
- For the first time, we provide a rigorous analysis of the performance bottlenecks of single-modal detectors from the perspective of degrees of freedom (DOF) of the 3D bounding box.
LiDAR-based 3D Object Detection could be roughly divided into two categories. 1) Voxel-based 3D detectors typically voxelize the point clouds into grid-structure forms of a fixed size [61]. [47] introduced a more efficient sparse convolution to accelerate training and inference. 2) Point-based 3D detectors consume the raw 3D point clouds directly and generate predictions based on (downsampled) points. [32] applied a point-based feature extractor and generated high-quality proposals on foreground points. 3DSSD [49] adopted a new sampling strategy named F-FPS as a supplement to D-FPS to preserve enough interior points of foreground instances. It also built a one-stage anchor-free 3D object detector based on feasible representative points. [55] and [2] introduced semantic-aware down-sampling strategies to preserve foreground points as much as possible. [36] proposed to encode the point cloud by a fixed-radius near-neighbor graph.
Camera-based 3D Object Detection. Early works [25, 24] designed monocular 3D detectors by referring to 2D detectors [31, 39] and utilizing 2D-3D geometric constraints. Another way is to convert images to pseudo-lidar representations via monocular depth estimation and then resort to LiDAR-based methods [43]. [3] built dense 2D-3D correspondence by predicting normalized object coordinate map [42] and adopted uncertainty-driven PnP to estimate object location. Our proposed NLC map estimation differs in that it aims to extract local spatial features from image features and does not rely on labeled dense depth maps and estimated RoI boxes. Recently, multi-view detectors like BEVStereo [17] have also achieved promising performance, getting closer to LiDAR-based methods.
Cross-Modal 3D Object Detection can be roughly divided into four categories: proposal-level, result-level, point-level, and pseudo-lidar-based approaches. Proposal-level fusion methods [5, 14, 19] adopted a two-stage framework to fuse image features and point cloud features corresponding to the same anchor or proposal. For result-level fusion, [26] exploited the geometric and semantic consistencies of 2D detection results and 3D detection results to improve single-modal detectors. Obviously, both proposal-level and decision-level fusion strategies are coarse and do not make full use of correspondence between LiDAR points and images. Pseudo-LiDAR based methods [52, 45, 19] employed depth estimation/completion and converted the estimated depth map to pseudo-LiDAR points to complement raw sparse point clouds. However, such methods require extra depth map annotations that are high-cost. Regarding point-level fusion methods, [18] proposed the continuous fusion layer to retrieve corresponding image features of nearest 3D points for each grid in BEV feature map. Especially, [40] and [12] retrieved the semantic scores or features by projecting points to image plane. However, neither semantic segmentation nor 2D detection tasks can enforce the network to learn 3D-spatial-aware image features. By contrast, we introduce NLC Map estimation as an auxiliary task to promote the image branch to learn local spatial information to supplement the sparse point cloud. Besides, [1] proposed TransFusion, a transformer-based detector that performs fine-grained fusion with attention mechanisms on the BEV level. [27] and [20] estimated the depth of multi-view images and transformed the camera feature maps into the BEV space before fusing them with LiDAR BEV features.
This section introduces the definition of the NLC map and analyzes the performance bottlenecks of single-modal detectors in terms of degrees of freedom (DOF). The goal of the cross-modal 3D detector is to estimate the bounding box parameters, including the object dimensions  $(w,l,h)$ , center location  $(x_{c},y_{c},z_{c})$ , and orientation angle  $\theta$ . The input consists of an RGB image  $X\in \mathbb{R}^{3\times H\times W}$  where  $H$  and  $W$  denote the height and width of image, respectively. Additionally, the input includes a 3D LiDAR point cloud  $\pmb{p}_i|i = 1,\dots,N$ , where  $N$  is the number of points, and each point  $\pmb{p}_i\in \mathbb{R}^4$  is defined by its 3D location  $(x_p,y_p,z_p)$  in the LiDAR coordinate system and the reflectance value  $\rho$ .
Normalized Local Coordinate (NLC) System. We define the NLC system as follows: we take the center of an object as the origin, align the  $x$ -axis towards the head direction of its ground-truth (GT) bounding box, and then normalize the local coordinates with respect to the size of the GT bounding box [7, 34]. Figure 2 (a) shows an example of the NLC system for a typical car. With the geometry relationship between the input RGB image and point cloud, we can project the NLCs to the 2D
image plane and construct a 2D NLC map with three channels corresponding to the three spatial dimensions, as illustrated in Fig.2(b).
(0,0,0) (a)
(b)
Figure 2: Illustration of the (a) NLC system and (b) 2D NLC map.
Specifically, to build the correspondence between the 2D and 3D modalities, we project the observed points from the 3D coordinate system to the 2D coordinate system on the image plane:
$$
d _ {c} [ u v 1 ] ^ {T} = \boldsymbol {K} \left[ \begin{array}{l l} \boldsymbol {R} & \boldsymbol {T} \end{array} \right] \left[ x _ {p} y _ {p} z _ {p} 1 \right] ^ {T}, \tag {1}
$$
where  $u, v, d_c$  denote the corresponding coordinates and depth on the image plane,  $R \in \mathbb{R}^{3 \times 3}$  and  $T \in \mathbb{R}^{3 \times 1}$  denote the rotation matrix and translation matrix of the LiDAR relative to the camera, and  $K \in \mathbb{R}^{3 \times 3}$  is the camera intrinsic matrix.
Given the point cloud and the bounding box of an object, LiDAR coordinates of foreground points can be transformed into the NLC system by proper translation, rotation, and scaling, i.e.,
$$
\left[ \begin{array}{l} x _ {p} ^ {N L C} \\ y _ {p} ^ {N L C} \\ z _ {p} ^ {N L C} \end{array} \right] = \left[ \begin{array}{c c c} 1 / l & 0 & 0 \\ 0 & 1 / w & 0 \\ 0 & 0 & 1 / h \end{array} \right] \cdot \left[ \begin{array}{l} x _ {p} ^ {L C S} \\ y _ {p} ^ {L C S} \\ z _ {p} ^ {L C S} \end{array} \right] + \left[ \begin{array}{l} 0. 5 \\ 0. 5 \\ 0. 5 \end{array} \right] = \left[ \begin{array}{c c c} 1 / l & 0 & 0 \\ 0 & 1 / w & 0 \\ 0 & 0 & 1 / h \end{array} \right] \cdot \left[ \begin{array}{c c c} \cos \theta & - \sin \theta & 0 \\ \sin \theta & \cos \theta & 0 \\ 0 & 0 & 1 \end{array} \right] \cdot \left[ \begin{array}{l} x _ {p} - x _ {c} \\ y _ {p} - y _ {c} \\ z _ {p} - z _ {c} \end{array} \right] + \left[ \begin{array}{l} 0. 5 \\ 0. 5 \\ 0. 5 \end{array} \right], \tag {2}
$$
where  $(x_p^{NLC},y_p^{NLC},z_p^{NLC})$  and  $(x_p^{LCS},y_p^{LCS},z_p^{LCS})$  denote the coordinates of points in NLC system and local coordinate system, respectively.
Discussion. Suppose that the bounding box is unknown, but the global coordinates and NLCs of points are known. We can build a set of equations to solve for the parameters of the box, namely  $x_{c}, y_{c}, z_{c}, w, l, h, \theta$ . To solve for all seven parameters, we need at least seven points with known global coordinates and NLCs to construct the equations. However, as shown in Fig. 1, the number of points for some objects is too small to estimate the 3D box with 7 degrees of freedom. Moreover, it is challenging to infer NLCs of points with only LiDAR data for objects far away or with low reflectivity, as the point clouds are sparse. However, RGB images can still provide visible contours and appearances for these cases. Therefore, we propose to estimate the NLCs of observed points based on enhanced image features. We can then retrieve the estimated NLCs based on the 2D-3D correspondence. Finally, we expect the proposed cross-modal detector to achieve higher detection accuracy with estimated NLCs and known global coordinates.
Overview. Architecturally, the processing pipeline for cross-modal 3D object detection includes an image branch and a point cloud branch, which learn feature representations from 2D RGB images and 3D LiDAR point clouds, respectively. Most existing methods only incorporate pixel-wise semantic cues from the 2D image branch into the 3D point cloud branch for feature fusion. However, we observe that point-based networks typically show insufficient learning ability due to the essential difficulties in processing irregular and unordered point cloud data modality [53]. Therefore, we propose to boost the expressive power of the point cloud backbone network with the assistance of the 2D image branch. Our ultimate goal is not to design a more powerful point cloud backbone network structure, but to make full use of the available resources in this cross-modal application scenario.
As shown in Fig.3, in addition to pixel-to-point feature propagation explored by mainstream point-level fusion methods[18, 12], we employ point-to-pixel propagation allowing features to flow inversely from the point cloud branch to the image branch. We achieve bidirectional feature propagation in a multi-stage fashion. In this way, not only can image features propagated via the pixel-to-point module provide additional information, but gradients backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone. We also employ auxiliary tasks to train the pipeline, aiming to enforce the network to learn rich semantic and spatial
Figure 3: Flowchart of the proposed UPIDet for cross-modal 3D object detection. It contains two branches separately extracting features from input 2D RGB images and 3D point clouds. UPIDet is mainly featured with (1) the point-to-pixel module that enables gradient backpropagation from 2D branch to 3D branch and (2) the supervision task of NLC map estimation for promoting the image branch to exploit local spatial features.
representations. In particular, we propose NLC map estimation to promote the image branch to learn spatial-aware features, providing a necessary complement to the sparse spatial representation extracted from point clouds, especially for distant or highly occluded cases. Appendix A.1 provides detailed network structures of the image and point cloud backbones.
4.1 Bidirectional Feature Propagation
The proposed bidirectional feature propagation consists of a pixel-to-point module and a point-to-pixel module, which bridges the two learning branches that operate on RGB images and LiDAR point clouds. Functionally, the point-to-pixel module applies grid-level interpolation on point-wise features to produce a 2D feature map, while the pixel-to-point module retrieves the corresponding 2D image features by projecting 3D points to the 2D image plane.
In general, we partition the overall 2D and 3D branches into the same number of stages. This allows us to perform bidirectional feature propagation at each stage between the corresponding layers of the image and point cloud learning networks. Without loss of generality, we focus on a certain stage where we can acquire a 2D feature map  $\mathbf{F} \in \mathbb{R}^{C_{2D} \times H' \times W'}$  with  $C_{2D}$  channels and dimensions  $H' \times W'$  from the image branch and a set of  $C_{3D}$ -dimensional embedding vectors  $\mathbf{G} = \{g_i \in \mathbb{R}^{C_{3D}}\}_{i=1}^{N'}$  of the corresponding  $N'$  3D points.
Point-to-Pixel Module. As illustrated in Fig. 4, we design the point-to-pixel module to propagate geometric information from the 3D point cloud branch into the 2D image branch. Formally, we
Figure 4: Illustration of the proposed point-to-pixel module.
aim to construct a 2D feature map  $\pmb{F}^{P2I} \in \mathbb{R}^{C_{3D} \times H' \times W'}$  by querying from the acquired point-wise embeddings  $\pmb{G}$ . More specifically, for each pixel position  $(r, c)$  over the targeted 2D grid of  $\pmb{F}^{P2I}$  with dimensions  $H' \times W'$ , we collect the corresponding geometric features of points that are projected inside the range of this pixel and then perform feature aggregation by avg-pooling  $\mathrm{AVG}(\cdot)$ :
$$
\boldsymbol {F} _ {r, c} ^ {P 2 I} = \operatorname {A V G} \left(\boldsymbol {g} _ {j} | j = 1, \dots , n\right), \tag {3}
$$
where  $n$  counts the number of points that can be projected inside the range of the pixel located at the  $r^{th}$  row and the  $c^{th}$  column. For empty pixel positions where no points are projected inside, we define its feature value as 0.
After that, instead of directly feeding the "interpolated" 2D feature map  $\pmb{F}^{P2I}$  into the subsequent 2D convolutional stage, we introduce a few additional refinement procedures for feature smoothness as described below:
$$
\boldsymbol {F} ^ {f u s e} = \operatorname {C O N V} (\operatorname {C A T} [ \operatorname {C O N V} (\boldsymbol {F} ^ {P 2 I}), \boldsymbol {F} ]), \tag {4}
$$
where  $\mathrm{CAT}[\cdot, \cdot]$  and  $\mathrm{CONV}(\cdot)$  stand for feature channel concatenation and the convolution operation, respectively. The resulting  $F^{fuse}$  is further fed into the subsequent layers of the 2D image branch.
Pixel-to-Point Module. This module propagates semantic information from the 2D image branch into the 3D point cloud branch. We project the corresponding  $N'$  3D points onto the 2D image plane, obtaining their planar coordinates denoted as  $\mathbf{X} = \{\pmb{x}_i\in \mathbb{R}^2\}_{i = 1}^{N'}$ . Considering that these 2D coordinates are continuously and irregularly distributed over the projection plane, we apply bilinear interpolation to compute exact image features at projected positions, deducing a set of point-wise embeddings  $\pmb{F}^{I2P} = \{\pmb {F}(\pmb {x}_i)\in \mathbb{R}^{C_{2D}}|i = 1,\dots,N'\}$ . Similar to our practice in the point-to-pixel module, the interpolated 3D point-wise features  $\pmb{F}^{I2P}$  need to be refined by the following procedures:
$$
\boldsymbol {G} ^ {f u s e} = \operatorname {M L P} \left(\operatorname {C A T} \left[ \operatorname {M L P} \left(\boldsymbol {F} ^ {I 2 P}; \boldsymbol {\theta} _ {1}\right), \boldsymbol {G} \right]; \boldsymbol {\theta} _ {2}\right), \tag {5}
$$
where  $\mathrm{MLP}(\cdot ;\pmb {\theta})$  denotes shared multi-layer perceptrons parameterized by learnable weights  $\pmb{\theta}$ , and the resulting  $G^{fuse}$  further passes through the subsequent layers of the 3D point cloud branch.
Remark. Unlike some prior work that introduces complicated bidirectional fusion mechanisms [16, 21, 23] to seek more interaction between the two modalities and finally obtain more comprehensive features, we explore a concise point-to-pixel propagation strategy from the perspective of directly improving the representation capability of the 3D LiDAR backbone network. We believe that a concise but effective point-to-pixel propagation strategy will better demonstrate that with feasible 2D auxiliary tasks, the gradients backpropagated from the training objectives of the image branch can boost the representation ability of the point cloud backbone.
4.2 Auxiliary Tasks for Training
The semantic and spatial-aware information is essential for determining the category and boundary of objects. To promote the network to uncover such information from input data, we leverage auxiliary tasks when training the whole pipeline. For the point cloud branch, following SA-SSD [9], we introduce 3D semantic segmentation and center estimation to learn structure-aware features. For the image branch, we propose NLC map estimation in addition to 2D semantic segmentation.
NLC Map Estimation. Vision-based semantic information is useful for networks to distinguish foreground from background. However, existing detectors face a performance bottleneck due to limited localization accuracy in distant or highly occluded cases where the spatial structure is incomplete due to the sparsity of point clouds. To address this, we use NLC maps as supervision to learn the relative position of each pixel inside the object from the image. This auxiliary task can drive the image branch to learn local spatial-aware features that complement the sparse spatial representation extracted from point clouds. Besides, it can augment the representation ability of the point cloud branch.
Remark. The proposed NLC map estimation shares the same objective with pseudo-LiDAR-based methods, which aim to enhance the spatial representation limited by the sparsity of point clouds. However, compared to learning a pseudo-LiDAR representation via depth completion, our NLC map estimation has several advantages: 1) the local NLC is easier to learn than the global depth owing to its scale-invariant property at different distances; 2) our NLC map estimation can be naturally incorporated and trained with the detection pipeline end-to-end, while it is non-trivial to