SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Compared with other paradigms on the TransFusion-L. * denotes the fine-tuned results. Note that when introducing MGTANet into TransFusion-L, we need to fine-tune the detection head of TransFusion-L, which is harmful to the final detection performance. Therefore, for a fair comparison, we choose the same fine-tuned TransFusion-L on different paradigms.
Method	Paradigm	mAP	NDS	FLOPs (G)	Latency (ms)	Params (M)
TransFusion-L* [2]	-	63.1	67.8	90.7	138.2	8.3
+MGTANet [16]	BEV	64.0 (+0.9)	68.1 (+0.3)	+193.0	+22.1	+5.5
+MPPNet [5]	Proposal	63.4 (+0.3)	68.2 (+0.4)	+131.3	+127.9	+7.0
+QTNet	Query	64.7 (+1.6)	69.0 (+1.2)	+0.1	+4.5	+0.3
baseline TransFusion-L, QTNet brings  $1.3\%$  (or  $1.5\%$ ) mAP and  $0.8\%$  (or  $0.9\%$ ) NDS improvements with 3 (or 4) frames, which achieves a new SOTA performance for LiDAR-only detectors. Besides, QTNet even outperforms MGTANet†, a manner of using the future frame information, by  $1.5\%$  mAP and  $0.2\%$  NDS. Note that QTNet does not utilize any future information since future information can not be fetched in the realistic scene of autonomous driving. Besides, compared with the single frame multi-modality baseline TransFusion, QTNet also produces  $1.4\%$  mAP and  $0.9\%$  NDS improvement. Based on the previous SOTA multi-modality method DeepInteraction, QTNet brings  $0.4\%$  mAP and  $0.5\%$  NDS improvement, achieving a new SOTA result for multi-modality 3D object detection.
On the nuScenes test benchmark, we take TransFusion-L as our baseline. As shown in Table 2, QTNet outperforms the advanced temporal fusion method MGTANet† by  $2.8\%$  mAP and  $0.8\%$  NDS for 3 frames as input, which illustrates the superiority of QTNet. QTNet achieves  $68.4\%$  mAP and  $72.2\%$  NDS with 4 frames, setting a new SOTA performance. Besides, we present the computation cost and latency by integrating our QTNet into different baselines in Table 3. It can be seen that our method brings consistent performance improvement with negligible computation cost and latency.
Query-based v.s. Other paradigms For a fair comparison, we integrate the representative long-term fusion of MGTANet[16] and MPPNet[5] into our baseline TransFusion-L[2] and keep the same number of frames as 3. As shown in Table 4, we present the results of QTNet and other methods in terms of the performance and computation cost. QTNet outperforms the BEV-based method MGTANet and the proposal-based method MPPNet  $0.9\%$  NDS and  $0.8\%$  NDS, respectively. For computation cost and latency, QTNet only brings a tiny increment of 0.1 GFLOPs and 4.5 ms based on the baseline of TransFusion-L. However, MGTANet even additionally produces 193.0 GFLOPs and 22.1 ms, and MPPNet brings 131.3 GFLOPs and 127.9 ms on the same baseline. These experiments effectively demonstrate that QTNet possesses superior performance and less computation cost and latency when compared with other paradigms. Besides, QTNet is lightweight, which only brings tiny extra parameters of 0.3 M.
Ablation studies on our QTNet We adopt the TransFusion-L [2] as our baseline (I) under the case of 3 frames as input and conduct ablation studies on our QTNet as shown in Table 5(a). Here, we provide an available manner to model the relationship of adjacent frames for temporal fusion by the cross attention operation [39] (II) for comparison. It can be observed that there is only a minor performance improvement with the mAP of  $0.2\%$  over baseline. In contrast, our proposed MTM
Table 5: Ablation studies on the temporal fusion, evaluated on the nuScenes validation set.
(a) Ablation studies on components of QTNet.
#	Cross Attention	MTM	IoU	Decouple	mAP	NDS
I	65.0	70.0
II	✓	65.2	70.0
III	✓	66.2	70.5
IV	✓	✓	66.3	70.6
V	✓	✓	✓	66.4	70.8
(b) Ablation studies on the number of frames
Frames	mAP	NDS	FLOPs (G)	Latency (ms)
1	65.0	70.0	90.65	138.2
2	66.1	70.6	+0.07	+3.1
3	66.4	70.8	+0.10	+4.5
4	66.5	70.9	+0.14	+6.5
5	66.4	70.9	+0.24	+7.7
Figure 5: Attention map comparison. For better visualization, we select some predictions with high confidence score and calculate the attention map. The row and column of attention map represent objects in  $t$  and  $t - 1$  frame, respectively. The value of attention map is higher and the relevance is stronger. The attention map (a) generated by cross attention is ambiguous. The attention map (b) generated by our MTM is more discriminative than cross attention.
(III) obtains  $1.2\%$  mAP and  $0.5\%$  NDS performance gains over baseline. Besides, we apply the IoU branch [35, 40] to rectify the confidence score of prediction results (IV), which can further boost the detection performance. Finally, we decouple the classification and regression of temporal fusion (V), which further brings  $0.2\%$  NDS performance gains. The above experiments demonstrate the effectiveness of our proposed components in our QTNet.
Different Number of Frames We explore the impact of detection performance on the different number of frames in the temporal fusion, whose results are shown in Table 5(b). We can find that as the number of frames increases from 1 to 4, the performance gain is gradually strengthening. When the number of frames reaches 5, the performance starts to saturate. Besides, the computation cost and latency slightly increase as the number of frames increases. However, thanks to our efficient design, the extra computation cost and latency are much slight compared with our whole network.
Why MTM is better? Intuitively, the cross attention operation [39] can directly construct the relevance among query features. However, objects with similar geometric structures between adjacent frames are difficult to distinguish in LiDAR point clouds, which makes the construction of the relationship between two frames less reliable. On the contrary, MTM builds an explicit motion modeling for temporal fusion by taking advantage of the fact that the same 3D objects between adjacent frames do not shift too much in a short time. To further illustrate the effectiveness of our MTM, we present the visualization for the manners of cross attention and our MTM as shown in Figure 5. Here, for convenience, we select three objects from predictions with high scores in  $t$  and  $t - 1$  frames. The attention map generated by cross attention is ambiguous, which leads to sub-optimal performance for temporal fusion. On the contrary, the attention map generated by our MTM is more discriminative. Therefore, our MTM can establish more reliable relevance among queries than cross attention, which proves the rationality and superiority of our proposed MTM.
4.5 Limitation
Our method can be integrated into some advanced LiDAR-only or multi-modality DETR-like 3D detectors and brings performance improvements with negligible computation cost and latency. However, the main limitation is that the cost and latency of our method increase as the number of frames
increases. In the future, we will explore recurrent architecture for 3D object detection with temporal fusion.
In this paper, we propose a query-based temporal fusion network named QTNet for 3D object detection. Based on our query-based feature representation, we introduce a lightweight motion-guided temporal modeling MTM module to aggregate temporal features more efficiently and effectively. Extensive experiments demonstrate the superiority of our proposed QTNet by comparing it with the mainstream BEV-based and proposal-based paradigms. Besides, QTNet even achieves new state-of-the-art on the nuScenes dataset for LiDAR-only and multi-modality 3D detectors with negligible computation cost and latency compared with the baseline network.
Acknowledgement This work was supported in part by the National Science Fund for Distinguished Young Scholars of China (Grant No. 62225603), in part by the Hubei Key R&D Program (Grant No. 2022BAA078), and in part by the Taihu Lake Innovation Fund for Future Technology (HUST: 2023-A-1).
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In arXiv preprint arXiv:1607.06450, 2016.
[2] Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, 2020.
[5] Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Cheung, Hang Xu, and Hongsheng Li. Mppnet: Multi-frame feature intertwining with proxy points for 3d temporal object detection. In European Conference on Computer Vision, 2022.
[6] Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels in 3d cnns. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[7] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[8] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Autoalignv2: Deformable feature aggregation for dynamic multi-modal 3d object detection. In European Conference on Computer Vision, 2022.
[9] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI Conference on Artificial Intelligence, 2021.
[10] Shengheng Deng, Zhihao Liang, Lin Sun, and Kui Jia. Vista: Boosting 3d object detection via dual cross-view spatial attention. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[11] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (voc) challenge. In International Journal of Computer Vision, 2010.
[12] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. In arXiv preprint arXiv:2112.11790, 2021.
[52] Dingyuan Zhang, Dingkang Liang, Zhikang Zou, Jingyu Li, Xiaoqing Ye, Zhe Liu, Xiao Tan, and Xiang Bai. A simple vision transformer for weakly semi-supervised 3d object detection. In IEEE International Conference on Computer Vision, 2023.
[53] Tianyuan Zhang, Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao. Muitr3d: A multi-camera tracking framework via 3d-to-2d queries. In IEEE Conference on Computer Vision and Pattern Recognition Workshop, 2022.
[54] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[55] Zhishuai Zhang, Jiyang Gao, Junhua Mao, Yukai Liu, Dragomir Anguelov, and Congcong Li. Stinet: Spatio-temporal-interactive network for pedestrian detection and trajectory prediction. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[56] Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. Se-ssd: Self-ensembling single-stage object detector from point cloud. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[57] Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In European Conference on Computer Vision, 2022.
[58] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. In arXiv preprint arXiv:1908.09492, 2019.
A.1 Performance on the Camera-based method
Although we design our motion-guided temporal modeling (MTM) module based on the LiDAR domain, we also explore the performance of MTM on camera-based methods. Thus, we integrate the MTM into the advanced camera-based detector CAPE [42] with two frames as input for temporal fusion on the nuScenes [3] validation set. As shown in Table 6, our MTM can also boost the performance of the camera-based method, which effectively demonstrates the generality of our method.
Table 6: Performance of camera-based method with MTM. The C represents camera. * denotes our reproduced results. All models are trained by four NVIDIA RTX 4090 GPUs with 24 epochs and without CBGS [58]. The batch size is set to 4.
Method	Year	Modality	Frames	Resolution	Backbone	mAP	NDS	mATE	mASE	mAOE	mAVE	mAAE
CAPE* [42]	CVPR 2023	C	1	704 × 256	R50	27.5	35.9	0.794	0.286	0.642	0.847	0.215
+MTM	-	C	2	704 × 256	R50	31.6	43.8	0.752	0.277	0.558	0.438	0.182
CAPE* [42]	CVPR 2023	C	1	800 × 320	V2-99	39.7	46.3	0.693	0.270	0.438	0.747	0.206
+MTM	-	C	2	800 × 320	V2-99	43.9	53.6	0.656	0.266	0.380	0.350	0.183
A.2 Performance breakdown for each category
We report the detailed performance of QTNet for each category on the nuScenes [3] testing benchmark, as shown in Table 7. Compared with our LiDAR-only baseline TransFusion-L [2], QTNet brings consistent improvements on most categories, especially on the construction vehicle  $(+7.0\%)$  AP), motorcycle  $(+6.6\%)$  AP), and bicycle  $(+6.3\%)$  AP).
Table 7: Comparison with state-of-the-art methods on the nuScenes testing set for each category. The L and C represent LiDAR and camera, respectively. C.V., Ped., M.C., B.C., T.C., and B.R. represent construction vehicle, pedestrian, motorcycle, bicycle, traffic cone, and barrier, respectively. The column of Frames denotes the number of key frame.  $\dagger$  denotes future information is used.
Method	Modality	Frames	mAP	NDS	Car	Truck	Bus	Trailer	C.V.	Ped.	M.C.	B.C.	T.C.	B.R.
CenterPoint [50]	L	1	60.3	67.3	85.2	53.5	63.6	56.0	20.0	84.6	59.5	30.7	78.4	71.1
TransFusion-L [2]	L	1	65.5	70.2	86.2	56.7	66.3	58.8	28.2	86.1	68.3	44.2	82.0	78.2
VISTA [10]	L	1	63.7	70.4	84.7	54.2	64.0	55.0	29.1	83.6	71.0	45.2	78.6	71.8
LidarMultiNet [48]	L	1	67.0	71.6	86.9	57.4	64.7	61.0	31.5	87.2	75.3	47.6	85.1	73.5
VoxelNeXt [7]	L	1	64.5	70.0	84.6	53.0	64.7	55.8	28.7	85.8	73.2	45.7	79.0	74.6
LargeKernel3D [6]	L	1	65.3	70.5	85.9	55.3	66.2	60.2	26.8	85.6	72.5	46.6	80.0	74.3
LinK [30]	L	1	66.3	71.0	86.1	55.7	65.7	62.1	30.9	85.8	73.5	47.5	80.4	75.1
3DVID† [49]	L	3	65.4	71.4	87.5	56.9	63.5	60.2	32.1	82.1	74.6	45.9	78.8	69.3
MGTANet† [16]	L	3	65.4	71.2	87.7	56.9	64.6	59.0	28.5	86.4	72.7	47.9	83.8	65.9
QTNet	L	3	68.2	72.0	86.5	57.2	68.3	63.0	34.3	88.1	74.9	49.7	82.7	77.0
QTNet	L	4	68.4	72.2	86.6	57.7	68.3	62.9	35.2	88.2	74.9	50.5	82.8	77.3
Besides, we report the detailed performance of QTNet for each category on the nuScenes [3] validation benchmark, as shown in Table 8.
Table 8: Comparison with different baselines on the nuScenes validation set for each category. * denotes our reproduced results.
Method	Modality	Frames	mAP	NDS	Car	Truck	Bus	Trailer	C.V.	Ped.	M.C.	B.C.	T.C.	B.R.
TransFusion* [2]	LC	1	67.1	70.7	87.7	61.6	75.9	42.4	26.5	88.0	75.2	63.8	77.3	72.2
+QTNet	LC	4	68.5	71.6	87.8	63.0	76.6	43.1	27.7	89.3	77.5	68.8	78.3	72.5
DeepInteraction* [47]	LC	1	69.9	72.6	88.5	64.4	79.2	44.5	30.1	88.9	79.0	67.8	80.0	76.4
+QTNet	LC	4	70.3	73.1	88.4	64.7	79.0	44.8	29.4	89.4	80.5	70.6	79.7	76.1
BEVFusion* [28]	LC	1	69.9	72.6	88.5	64.4	79.2	44.5	30.1	88.9	79.0	67.8	80.0	76.4
+QTNet	LC	4	70.3	73.1	88.4	64.7	79.1	44.8	29.4	89.4	80.5	70.6	79.7	76.1
TransFusion-L* [2]	L	1	65.0	70.0	86.7	60.4	75.3	41.6	24.6	86.8	71.8	56.5	74.4	71.8
+QTNet	L	3	66.3	70.8	87.1	61.1	75.5	43.0	25.7	87.8	75.2	61.2	75.6	71.4
+QTNet	L	4	66.5	70.9	87.2	61.5	75.8	43.0	25.7	87.8	75.5	61.5	75.4	71.4
A.3 Visualization
To illustrate the superiority of our QTNet, we visualize the results of TransFusion-L [2] on the nuScenes [3] validation set for comparison. As shown in Figure 6, QTNet can detect the hard-detected objects for TransFusion-L and boost the detection performance thanks to our proposed temporal fusion module MTM. As shown in Figure 7, QTNet successfully corrects the angle error of objects for TransFusion-L thanks to our proposed temporal fusion module MTM. Besides, as shown in Figure 8, we compare TransFusion-L and QTNet along the temporal dimension for better presentation. It can be seen that the object on the lower left, which is moving away from the ego vehicle, is not detected in  $t$  frame by TransFusion-L. However, QTNet can still capture the object in  $t$  frame, benefiting from our effective temporal fusion.
(b)
Figure 6: Comparison of LiDAR-only baseline TransFusion-L (a) and QTNet (b) on the nuScenes validation set. Blue and green boxes are the prediction and ground truth boxes. It can be seen that TransFusion-L fails to detect the hard-detected objects. However, thanks to the temporal information, QTNet detects these objects successfully.
Figure 7: Comparison of LiDAR-only baseline TransFusion-L (a) and QTNet (b) about orientation of objects. Thanks to the temporal information, QTNet successfully corrected the orientation error.
Figure 8: Comparison of LiDAR-only baseline TransFusion-L (a) and QTNet (b) along the temporal dimension. The ego vehicle is moving from bottom to top.
A.4 Discussions of potential societal impacts
Effectively utilizing temporal information is vital for autonomous driving. QTNet improves 3D detection performance with negligible computation cost and latency by a lightweight temporal fusion module MTM, which can utilize temporal information to improve the safety of autonomous driving in the real world. However, temporal fusion usually requires sensor synchronization in time, which puts forward higher requirements for the hardware of autonomous driving.