SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Jinghua Hou $^{1*}$ , Zhe Liu $^{1*}$ , Dingkang Liang $^{1}$ , Zhikang Zou $^{2}$ , Xiaqing Ye $^{2}$ , Xiang Bai $^{1\dagger}$
$^{1}$ Huazhong University of Science & Technology
$^{2}$ Baidu Inc.
{jhhou,zheliu1994,dkliang,xbai}@hust.edu.cn
{zhikangzou001}@gmail.com
{yexiaoqing}@baidu.com
Effectively utilizing temporal information to improve 3D detection performance is vital for autonomous driving vehicles. Existing methods either conduct temporal fusion based on the dense BEV features or sparse 3D proposal features. However, the former does not pay more attention to foreground objects, leading to more computation costs and sub-optimal performance. The latter implements time-consuming operations to generate sparse 3D proposal features, and the performance is limited by the quality of 3D proposals. In this paper, we propose a simple and effective Query-based Temporal Fusion Network (QTNet). The main idea is to exploit the object queries in previous frames to enhance the representation of current object queries by the proposed Motion-guided Temporal Modeling (MTM) module, which utilizes the spatial position information of object queries along the temporal dimension to construct their relevance between adjacent frames reliably. Experimental results show our proposed QTNet outperforms BEV-based or proposal-based manners on the nuScenes dataset. Besides, the MTM is a plug-and-play module, which can be integrated into some advanced LiDAR-only or multi-modality 3D detectors and even brings new SOTA performance with negligible computation cost and latency on the nuScenes dataset. These experiments powerfully illustrate the superiority and generalization of our method. The code is available at https://github.com/AlmoonYsl/QTNet.
3D object detection is the fundamental task of autonomous driving, which consumes data from sensors (e.g., LiDAR, Camera) to localize and recognize objects in the 3D world. Recent studies have explored utilizing temporal information to improve the detection performance. According to different manners of temporal fusion, we divide them into two categories, including BEV-based (Figure 1(a)) and proposal-based (Figure 1(b)).
BEV-based methods [16, 49, 55, 43, 57] conduct temporal fusion on the dense bird's-eye-view (BEV) feature by applying an affine transformation to align BEV features in previous frames to enhance the current BEV features. However, not all pixel-wise BEV features are beneficial for boosting the detection performance of foreground objects. To alleviate this problem, the proposal-based approaches [46, 34, 5] usually adopt a region proposal network (RPN) to obtain many 3D proposals
(a) BEV-based
(b) Proposal-based
(c) Query-based (Ours)
(d) Comparison of different paradigms on the same baseline
Figure 1: Temporal fusion architecture comparison. (a) BEV-based: conduct temporal fusion on the dense BEV features. (b) Proposal-based: first extract proposal by 3D RoI pooling operation and then implement temporal fusion on the proposal-level features. (c) Query-based (Ours): directly adopt the query-level features to achieve temporal fusion. (d) We compare the performance, computational cost, and latency of different paradigms on the same baseline (TransFusion-L [2]). Computation cost and latency are tested on a single NVIDIA RTX 4090 GPU with 1 batch size. For BEV-based and proposal-based paradigm, we reproduce the representative long-term fusion of MGTANet [16] and MPPNet [5] on the baseline, respectively.
and then achieve temporal fusion by focusing on foreground object features extracted by the time-consuming 3D RoI-like operations [37, 36, 9]. However, the effectiveness of temporal fusion is bounded by the quality of the generated 3D proposals.
We argue that an effective and lightweight temporal fusion module is more meaningful to autonomous driving technology with high real-time and detection accuracy requirements. Towards this goal, we rethink the format of feature representation and the temporal modeling paradigm in temporal fusion for 3D object detection task, and propose a new query-based temporal fusion method called QTNet. Different from the BEV-based and proposal-based feature representation in Figure 1(a) and Figure 1(b), we adopt a query-based mechanism shown in Figure 1(c) that possesses two merits. On the one hand, the query-based representation is sparse and can effectively aggregate the foreground object information by attention operation [39]. On the other hand, the query-based format is more efficient due to getting rid of complex 3D RoI operations and is less sensitive to 3D object of size and orientation than proposal-based representation.
However, the temporal modeling based on query representation is non-trivial to work well. A natural manner is to compute the feature similarity between the queries of the current and previous frame by cross attention operation. However, the feature measurement is less reliable since 3D objects with similar geometric structures between adjacent frames are difficult to distinguish, leading to failed matching. Fortunately, we observe that 3D objects usually follow the physical law of motion in the real-world 3D space, which means that the same 3D object between adjacent frames does not shift too much in a short time.
Based on the above observations, we further adopt a Motion-guided Temporal Modeling (MTM) module based on the sparse query representation. Specifically, we first align objects from previous frame to current frame and then generate the attention map by explicit position information of objects. Finally, we aggregate the temporal features by the attention map to enhance the current query features for the subsequent refinement and produce the final detection results.
Besides, as shown in Figure 1(d), we compare our query-based paradigm with the BEV-based and the proposal-based paradigm in terms of FLOPs, latency, and performance. We can observe that our QTNet possesses distinct advantages in performance, computation cost, and latency. Overall, our contributions are summarized as follows: (i) We propose a new temporal fusion method called QTNet for 3D object detection based on sparse query-based feature representation, which is more effective and efficient than BEV-based and proposal-based manners. (ii) We propose the MTM, which can be plugged into LiDAR-only or multi-modality 3D detectors and boost their performance with negligible computation cost and latency.
3D object detection The current 3D object detection detectors can be roughly categorized into three categories: LiDAR-based, camera-based, and multi-modality. For LiDAR-based, some methods [37, 56, 54, 52] directly consume pnoit clouds and predict the 3d detection results. Other methods [17, 27, 36, 50, 2, 18] first quantify point clouds into regular grid structures (e.g., voxels, pillars) and then utilize a 3D backbone to extract features. These methods map 3D grids to bird-eye view (BEV) representation by height compression and utilize 2D CNN to extract features further for 3D object detection with efficiency. For camera-based, some methods [21, 12, 20, 19] lift the image features into 3D space and then generate BEV features. Other methods [23, 42] follow the DETR [4] paradigm, and they utilize queries to interact with the multi-view image features and predict the 3D detection results. LiDAR-based methods have precise geometric information with high performance but lack dense texture information. In contrast, camera-based methods have rich texture information but lack explicit 3D geometric information. Obviously, LiDAR and camera information are complementary. Therefore, multi-modality methods [14, 24, 51, 8, 2, 28, 22, 47, 15, 26, 25] utilize multi-modal fusion to improve detection performance.
3D multi-object tracking Due to the lack of rich texture information for point clouds, LiDAR-based tracking methods [41, 50, 31, 33] usually utilize the motion of objects for the association. For example, CenterPoint [50] estimates the velocity by an additional regression head and utilizes L2 distance to associate objects in the previous and current frame. SimpleTrack [33] explores settings of tracking on different datasets through a series of experiments. With the development of camera-based detectors, some methods [53, 32] have begun to explore conducting 3D multi-object tracking on the camera domain since there is rich appearance information in images. MUTR3D [53] uses queries to represent objects and achieves tracking by attention [39]. PF-Track [32] utilizes a temporal transformer to conduct temporal fusion and integrates past and future reasoning for tracked objects.
3D object detection with temporal fusion In the real world, temporal information is beneficial to perceive the localization and direction of movement for 3D objects. Towards this goal, some researchers exploit temporal information to boost the performance of 3D detection task. There are two sets of mainstream, including BEV-based [49, 21, 16, 13] and proposal-based [46, 34, 5] temporal fusion. For BEV-based paradigm, the typical approach MGTANet [16] adopts an SM-VFE module to encode LiDAR point features and then a motion-guided deformable module to align and aggregate multi-frame BEV features for temporal fusion. However, this BEV-based paradigm takes both the foreground and background into consideration, leading to additional computation costs and sub-optimal performance. For proposal-based paradigm, MPPNet [5] utilizes RoI grid pooling [9] to extract proposal features and generate trajectories for temporal fusion. However, this proposal-based paradigm usually adopts a multi-stage pipeline and some time-consuming 3D RoI pooling operation, which makes the whole pipeline complex and brings unacceptable runtime. In summary, the above methods have the disadvantage of sub-optimal performance or high latency and computation cost. To address these problems, in this paper, we propose an efficient temporal fusion method QTNet based on query representation, which achieves better performance and notably reduces the computation cost and latency.
In this section, we introduce the proposed Query-based Temporal Fusion Network (QTNet), which is presented in Figure 2. QTNet consists of a DETR-like 3D detector, a memory bank, and our proposed Motion-guided Temporal Modeling (MTM) module. In contrast to the prior arts, including
Figure 2: The overall architecture of the proposed QTNet. We first feed sensor data to the DETR-like 3D detector and generate current frame queries. We utilize a memory bank to store historical queries for query-based temporal fusion, which follows the FIFO rule. In the temporal fusion, we fetch these historical queries from the memory bank and then feed them to our motion-guided temporal modeling module to fuse queries in different frames. Finally, the fused queries are utilized to refine current detection results.
BEV-based [16, 5, 49, 57] and proposal-based [34, 5], we conduct the temporal fusion on sparse query representation. Specifically, we first feed LiDAR point clouds to DETR-like 3D detectors [2, 47, 15] and generate sparse query features. Note that our method can also support multi-modal data as input (e.g., images and LiDAR point clouds). Thus, the input of images is optional. Then, we reserve the historical sparse query features by a memory bank to reduce repeated computation costs. Next, we take the query features of the previous frame and current frame into our MTM module and produce the enhanced query features for the final 3D detection. Next, we will present the details of the main components in QTNet.
3.1 Memory Bank
Different from the existing temporal fusion methods [16, 49, 57] that require much repeated computation for the feature extraction from the previous frames, we utilize a memory bank to preserve the historical information for the subsequent temporal fusion. Here, we define a memory bank  $S$  with the size of  $N \times K$ , where  $N$  means the number of used historical frames and  $K$  is the number of queries. Our memory bank follows the first-in and first-out (FIFO) rule to achieve query feature updates. In more detail, we first conduct temporal fusion in the  $t$  frame and reserve  $N$  historical frames. Then, we fetch queries  $[Q_{t-1}, Q_{t-2}, \ldots, Q_{t-N}]$  in historical frames. Next, we can fuse these historical queries with the current queries  $Q_t$  so as to obtain the enhanced current queries  $Q_t'$ . Finally, we push current queries  $Q_t$  into  $S$  and pop the oldest queries  $Q_{t-N}$ .
3.2 Motion-guided Temporal Modeling Module
The Motion-guided Temporal Modeling (MTM) module is applied to enhance the current query features by combining the feature information from the history frames. Specifically, we first obtain the current object queries  $Q_{t}$  and their corresponding positions  $C_t$  (can also be regarded as the center of 3D objects) from the DETR-like 3D detectors. In this paper, we adopt Transfusion-L [2] as the default 3D detector. Then, we feed current queries  $Q_{t}$ , previous fused queries  $Q_{t - 1}^{\prime}$ , and their corresponding positions  $C_t$  and  $C_{t - 1}$  to MTM for generating current fused queries  $Q_{t}^{\prime}$ . In more detail, we present the structure of MTM in Figure 3, which includes three steps: the position alignment of objects between the previous and the current frames, the attention map generation, and the feature aggregation. For the process of this alignment, we first align objects from previous frame to current frame by the motion of objects and the ego vehicle. Given the ego pose matrix  $R_w^{t - 1}$  and  $R_w^t$  in  $t - 1$
Figure 3: The details of MTM. In the MTM, the center of queries in different frames is used to produce the attention map of queries, which is then applied to enhance the current queries.
frame and  $t$  frame, we transform objects from the world coordinate to the ego coordinate. Then, the aligned transformation matrix  $R_{t - 1}^{t}$ , which transforms objects from  $t - 1$  frame to  $t$  frame, can be calculated as:
$$
R _ {t - 1} ^ {t} = R _ {w} ^ {t} \cdot \operatorname {i n v} \left(R _ {w} ^ {t - 1}\right), \tag {1}
$$
where the inv denotes the matrix inversion operation. Then, we can align the center of objects in  $t - 1$  frame to  $t$  frame by their velocity  $V_{t - 1}$  and the aligned transformation matrix  $R_{t - 1}^{t}$  with the related time interval  $\Delta t$  between  $t$  and  $t - 1$  frame:
$$
C _ {t - 1} ^ {\prime} = \left(C _ {t - 1} + V _ {t - 1} \cdot \Delta t\right) \cdot \left(R _ {t - 1} ^ {t}\right) ^ {T} \tag {2}
$$
For the attention map generation, we first define the number of  $Q_{t}$  is  $N_{t}$  and  $Q_{t - 1}$  is  $N_{t - 1}$ . Then, we utilize the L2 center distance of objects between adjacent frames to build the cost matrix  $O_{t - 1}^{t}\in \mathbb{R}^{N_{t}\times N_{t - 1}}$ :
$$
O _ {t - 1} ^ {t} = \mathrm {L} _ {2} \left(C _ {t} - C _ {t - 1} ^ {\prime}\right), \tag {3}
$$
where the  $\mathrm{L}_2$  denotes the euclidean distance. To avoid the occurrence of impossible matching situations for better temporal modeling, we introduce the L2 velocity error distribution of each category in 0.5 seconds as our distance cost threshold  $\gamma$ , which is similar to CenterPoint [50]. Besides, these associated objects with different categories are masked. We formulate the above process as follows:
$$
M = \left\{ \begin{array}{l l} 0, & O _ {t - 1} ^ {t} \leq \gamma \quad a n d \quad s _ {t} = s _ {t - 1} \\ 1 e ^ {8}, & O _ {t - 1} ^ {t} > \gamma \quad o r \quad s _ {t} \neq s _ {t - 1} \end{array} , \right. \tag {4}
$$
where  $s_t$  and  $s_{t-1}$  denote the category of objects in  $t$  and  $t-1$  frames, respectively. After adopting the attention mask  $M$  with the row-wise softmax function on the cost matrix, we generate the attention map  $A$ , which is derived from the explicit position information:
$$
A = \operatorname {s o f t m a x} (- 1 \cdot O _ {t - 1} ^ {t} - M), \tag {5}
$$
For feature aggregation, we utilize the attention map to aggregate historical features. Given the current queries  $Q_{t}$ , previous fused queries  $Q_{t-1}^{\prime}$ , and attention map  $A$  in Figure 3, the aggregated temporal features  $F_{t-1}$  can be calculated as:
$$
F _ {t - 1} = \phi_ {2} \left(A \cdot \phi_ {1} \left(Q _ {t - 1} ^ {\prime}\right)\right), \tag {6}
$$
where the  $\phi_{1}$  and  $\phi_{2}$  denote two linear layers. After aggregation, we can combine  $F_{t - 1}$  and  $Q_{t}$  to generate the current fused queries  $Q_{t}^{\prime}$ :
$$
Q _ {t} ^ {\prime} = \operatorname {N o r m} \left(Q _ {t} + \text {D r o p o u t} (\operatorname {F F N} \left(\operatorname {N o r m} \left(Q _ {t} + \text {D r o p o u t} \left(F _ {t - 1}\right)\right)\right)\right), \tag {7}
$$
where Dropout, Norm, and FFN denote the dropout operation [38], the layer normalization [1], and the feed-forward network, respectively. Finally, we utilize  $Q_{t}^{\prime}$  to refine the predicted results of the current frame through two simple FFNs, which are utilized to estimate the confidence of the classification branch and the residual of the regression branch for further refinement. Besides, we follow [40, 35] to utilize the IoU branch to rectify the confidence score of prediction results in the current frame.
Figure 4: Decouple temporal fusion. The decoupling manner conducts temporal fusion on classification and regression branches separately for better detection performance. Here, the weights of classification and regression branches are not shared.
3.3 Decouple Strategy
We observe that there is an imbalance in the classification and regression learning for temporal fusion. This phenomenon is also mentioned in VISTA [10] for 3D detection task. However, this decoupled manner brings more computation costs and latency, which is unacceptable for most BEV-based and proposal-based methods. In contrast, benefiting from our efficient design of temporal fusion, the computation cost and latency of QTNet are negligible compared with the whole 3D detection network. Therefore, we decouple the classification and regression branches as shown in Figure 4(b) instead of the coupling manner in Figure 4(a) that is usually employed in the previous temporal fusion methods [49, 5, 16]. Specifically, we separately implement the temporal fusion for the classification branch and the regression branch so as to achieve better detection performance.
Dataset We evaluate our method on the nuScenes dataset [3], which is a large-scale autonomous driving dataset. It contains 700, 150, and 150 scenes for training, validation, and testing. Each scene is roughly 20 seconds long and annotated at  $2\mathrm{Hz}$ , and provides point clouds acquired by 32-beam LiDAR and surrounding images acquired by 6 cameras. Note that each sample consists of one key frame and nine sweep frames. Besides, there are roughly 1.6M objects for 10 categories (barrier, bicycle, bus, car, motorcycle, pedestrian, trailer, truck, construction vehicle, and traffic cone) on the 3D object detection task. For the evaluation metric, Mean Average Precision (mAP) [11] and nuScenes detection score (NDS) [3] are used to evaluate the performance of 3D detectors on the nuScenes dataset.
Following the settings of nuScenes dataset, we concatenate point clouds of one key frame and nine sweeps as the input of a sample, which is roughly 0.5 seconds long. Each point is represented as  $(x,y,z,r,\Delta t)$ , where the  $(x,y,z)$ ,  $r$ , and  $\Delta t$  denote the 3D coordinate of a point, the reflectance, and the time offset from the key frame, respectively.
Our method is implemented on 3D object detectors with LiDAR or multi-modality (LiDAR and images). For LiDAR, we take the TransFusion-L [2] as our LiDAR-only baseline. For multimodality, we take the TransFusion [2] and DeepInteraction [47] as our multi-modality baseline. The voxelization range is set to  $[-54m, 54m]$  for both  $X$  and  $Y$  axes and  $[-5m, 3m]$  for  $Z$  axis. The voxel size is set to  $(0.075m, 0.075m, 0.1m)$ . We set the number of queries as 200 for training and testing. For temporal fusion, we utilize 2 or 3 historical frames. Note that we do not adopt any test time augmentation (TTA), multiple models ensemble, or future frames during inference.
Table 1: Comparison with state-of-the-art methods on the nuScenes validation set. The L and C represent LiDAR and camera, respectively. The column of Frames denotes the number of key frame. * denotes our reproduced results. † denotes future information is used.
Method	Year	Modality	Frames	Backbones	mAP↑	NDS↑
MVP [51]	NeurIPS 2021	LC	1	DLA34 & VoxelNet	67.1	70.8
AutoAlignV2 [8]	ECCV 2022	LC	1	CSPNet & VoxelNet	67.1	71.2
TransFusion [2]	CVPR 2022	LC	1	R50 & VoxelNet	67.5	71.3
BEVFusion [22]	NeurIPS 2022	LC	1	Swin-Tiny & VoxelNet	67.9	71.0
DeepInteraction [47]	NeurIPS 2022	LC	1	R50 & VoxelNet	69.9	72.6
BEVFusion [28]	ICRA 2023	LC	1	Swin-Tiny & VoxelNet	68.5	71.4
MSMDFusion [15]	CVPR 2023	LC	1	R50 & VoxelNet	69.3	72.1
CMT [44]	ICCV 2023	LC	1	V2-99 & VoxelNet	70.3	72.9
TransFusion* [2]	CVPR 2022	LC	1	R50 & VoxelNet	67.1	70.7
+QTNet	-	LC	4	R50 & VoxelNet	68.5	71.6
DeepInteraction* [47]	NeurIPS 2022	LC	1	R50 & VoxelNet	69.9	72.6
+QTNet	-	LC	4	R50 & VoxelNet	70.3	73.1
CenterPoint [50]	CVPR 2021	L	1	VoxelNet	59.6	66.8
TransFusion-L [2]	CVPR 2022	L	1	VoxelNet	65.1	70.1
INT [43]	ECCV 2022	L	10	VoxelNet	61.8	67.3
LidarMultiNet [48]	AAAI 2023	L	1	VoxelNet	63.8	69.5
VoxelNeXt [7]	CVPR 2023	L	1	VoxelNet	60.0	67.1
LargeKernel3D [6]	CVPR 2023	L	1	Voxel-LargeKernel3D	63.3	69.1
LinK [30]	CVPR 2023	L	1	Voxel-LinK	63.6	69.5
MGTANet [16]	AAAI 2023	L	3	VoxelNet	62.9	68.7
MGTANet† [16]	AAAI 2023	L	3	VoxelNet	64.8	70.6
TransFusion-L* [2]	CVPR 2022	L	1	VoxelNet	65.0	70.0
+QTNet	-	L	3	VoxelNet	66.3	70.8
+QTNet	-	L	4	VoxelNet	66.5	70.9
Table 2: Comparison with state-of-the-art methods on the nuScenes test set. † denotes future information is used.
Method	Year	Modality	Frames	Backbones	mAP↑	NDS↑
CenterPoint [50]	CVPR 2021	L	1	VoxelNet	60.3	67.3
TransFusion-L [2]	CVPR 2022	L	1	VoxelNet	65.5	70.2
VISTA [10]	CVPR 2022	L	1	VoxelNet	63.7	70.4
LidarMultiNet [48]	AAAI 2023	L	1	VoxelNet	67.0	71.6
VoxelNeXt [7]	CVPR 2023	L	1	VoxelNet	64.5	70.0
LargeKernel3D [6]	CVPR 2023	L	1	Voxel-LargeKernel3D	65.3	70.5
LinK [30]	CVPR 2023	L	1	Voxel-LinK	66.3	71.0
3DVID† [49]	TPAMI 2021	L	3	VoxelNet	65.4	71.4
MGTANet† [16]	AAAI 2023	L	3	VoxelNet	65.4	71.2
QTNet	-	L	3	VoxelNet	68.2	72.0
QTNet	-	L	4	VoxelNet	68.4	72.2
Our training process is divided into two stages. In the first stage, we train the DETR-like 3D detectors (TransFusion-L, TransFusion, and DeepInteraction) with their default settings. In the second stage, we generate the memory bank of query features and prediction results. Then, we train our QTNet for 10 epochs without GT-Sampling [45] and CBGS [58] on four NVIDIA RTX 4090 GPUs. To optimize our network, we adopt the AdamW [29] optimizer with a one-cycle learning rate policy, and the batch size is set to 16.
4.3 Comparison to the state of the art
We compare the proposed QTNet with previous state-of-the-art 3D detectors on the nuScenes [3] validation and test set. As shown in Table 1, QTNet shows consistent performance improvement when our module is integrated into the LiDAR-only detector Transfusion-L [2] or multi-modality methods Transfusion [2] and DeepInteraction [47]. Compared with the single frame LiDAR-only
Table 3: Comparison of computation cost and latency on the nuScenes validation set. * denotes our reproduced results. The FLOPs and Latency are tested on a single NVIDIA RTX 4090 GPU with the batch size of 1.
Method	Modality	mAP (%)↑	NDS (%)↑	FLOPs (G)↓	Latency (ms)↓	Params (M)
TransFusion* [2]	LC	67.1	70.7	445.9	201.2	37.0
+QTNet	LC	68.5	71.6	446.4	207.7	37.7
DeepInteraction* [47]	LC	69.9	72.6	499.2	355.0	57.8
+QTNet	LC	70.3	73.1	499.7	361.5	58.5
TransFusion-L* [2]	L	65.0	70.0	90.7	138.2	8.3
+QTNet	L	66.5	70.9	90.8	144.7	8.6