Query-based Temporal Fusion with Explicit Motion for 3D Object Detection
Authors: Jinghua Hou
Zhe Liu
dingkang liang
Zhikang Zou
Xiaoqing Ye
Xiang Bai
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D Object Detection
Temporal
LiDAR-only
Multi-modality
Autonomous Driving

Abstract: Effectively utilizing temporal information to improve 3D detection performance is vital for autonomous driving vehicles. Existing methods either conduct temporal fusion based on the dense BEV features or sparse 3D proposal features. However, the former does not pay more attention to foreground objects, leading to more computation costs and sub-optimal performance. The latter implements time-consuming operations to generate sparse 3D proposal features, and the performance is limited by the quality of 3D proposals. In this paper, we propose a simple and effective Query-based Temporal Fusion Network (QTNet). The main idea is to exploit the object queries in previous frames to enhance the representation of current object queries by the proposed Motion-guided Temporal Modeling (MTM) module, which utilizes the spatial position information of object queries along the temporal dimension to construct their relevance between adjacent frames reliably. Experimental results show our proposed QTNet outperforms BEV-based or proposal-based manners on the nuScenes dataset. Besides, the MTM is a plug-and-play module, which can be integrated into some advanced LiDAR-only or multi-modality 3D detectors and even brings new SOTA performance with negligible computation cost and latency on the nuScenes dataset. These experiments powerfully illustrate the superiority and generalization of our method. The code is available at https://github.com/AlmoonYsl/QTNet.

Query-based Temporal Fusion with Explicit Motion for 3D Object Detection
Jinghua Hou Zhe Liu Dingkang Liang' Zhikang Zou? Xiaoqing Ye Xiang Bai
Huazhong University of Science & Technology
Baidu Inc.
2hhoo,,hheliu194 dkliang, Xhuit .ew.ustteuu....
(2hhikanggzauu001@@@maa... (yexfaooinng@bbaidu.com
Abstract
Effectively utilizing temporal information to improve 3D detection performance is vital for autonomous driving vehicles. Existing methods either conduct tem- poral fusion based on the dense BEV features or sparse proposal features. However, the former does not pay more attention to foreground objects. leading more computation costs and sub-optimal performance. The latter implements time-consuming operations generate sparse 3D proposal features, and the perfor- mance is limited by the quality of 3D proposals. In this paper, we propose a simple and effective Query-based Temporal Fusion Network (QTNet). The main idea is to exploit the object queries previous frames to enhance the representation of current object queries by the proposed dotion-ggudde Temporal Modeling (MTM) module, which utilizes the spatial position information object queries along the temporal dimension to construct their relevance between adjacent frames liably. Experimental results show our proposed QTNet outperforms BEV-based proposal- based manners the nuScenes dataset. Besides, the MTM is a plug- and-play module, which can integrated into some advanced LiDAR-only or multi-modality 3D detectors and even brings SOTA performance with negli- gible computation cost and latency on the nuScenes dataset. These experiments powerfully illustrate the superiority and generalization of our method. The code is available at https: //g thub. com/
Introduction
3D object detection is the fundamental task of autonomous driving, which consumes data from sensors (e.g., LiDAR, Camera) localize and recognize objects in the 3D world. Recent studies have explored utilizing temporal information to improve the detection performance. According to different manners temporal fusion, divide them into two categories, including BEV-based (Figure ((a)) and proposal-based (Figure 1(b)).
BEV-based methods [16, 49, 55, 43, 57] conduct temporal fusion the dense bird's seee (BEV) feature by applying an affine transformation to align BEV features in previous frames to enhance the current BEV features. However, not all pixel-wise BEV features beneficial for boosting the detection performance of foreground objects. To alleviate this problem, the proposal-based approaches [46, 34, 5] usually adopt region proposal network (RPN) obtain many 3D proposals
"Equal contribution. t Corresponning author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

nEV-taoo Temppre Tuokan
Strpest
Preercai for
Scatore
Point Clands
Proposal Festure
BEY featurs
Quary Featurs
(a) BEV-hased
(b) Proposal-based
(c) Query-based (Ours)
04.8
EEV-baved
350
Proporal baied
301-bout
B Dutty Liand
B Query-aoeed
Quiry booe
66.3
102 114
69.2
200
$3.6
612
300
Tramns
Framas
Framres
(d) Comparison of different paradigms on the same baseline
Figure I: Temporal fusion architecture comparison. (a) BEV-based: conduct temporal fusion on the dense BEV features. (b) Proposal- based:: first extract proposal by 3D Rol pooling operation and then implement temporal fusion on the proposal-tevel features. (c) Query-based (Ours): directly adopt the query-level features achieve temporal fusion. (d) We compare the performance, computational cost, and latency of different paradigms on the same baseline (TransFusion-L. [2]). Computation cost and latency are tested on single NVIDIA RTX 4090 GPU with batch size. For BEV-based and
proposal-bases paradigm, we reproduce the representative long-term fusion of MGTANet [16] and MPPNet [5] the baseline, respectively.
and then achieve temporal fusion by focusing foreground object features extracted by the time- consuming 3D Rol-like operations [37, 36, 9]. However, the effectiveness of temporal fusion is bounded by the quality of the generated 3D proposals.
argue that an effective and lightweight temporal fusion module more meaningful to autonomous driving technology with high real-time and detection accuracy requirements. Towards this goal, we rethink the format feature representation and the temporal modeling paradigm temporal fusion for 3D object detection task. and propose
query- based temporal fusion method called QTNet. Different from the BEV-based and proposal-base based feature representation Figure ((a) and Figure 1(b), adopt query-basee mechanism shown in Figure I(c) that possesses two merits. On the one hand, the query-based representation is sparse and can effectively aggregate the foreground object information by attention operation [39]. On the other hand, the query-based format is more efficient due to getting rid of complex 3D Rol operations and less sensitive to 3D object of size and orientation than proposal- prosssa--based representation.
However, the temporal modeling based on query representation non-trivial to work well. natural manner is to compute the feature similarity between the queries of the current and previous frame by cross attention operation. However, the feature measurement is less reliable since 3D objects with similar geometric structures between adjacent frames difficult to distinguish, leading to failed matching. Fortunately, we observe that 3D objects usually follow the physical law of motion in the real-world 3D space, which means that the same 3D object between adjacent frames does not shift too much a short time.
Based on the above observations, we further adopt Motion Motion-guided Temporal Modeling (MTM) module based on the sparse query representation. Specifically, we first align objects from previous frame to current frame and then generate the attention map by explicit position information of objects. Finally, we aggregate the temporal features by the attention map to enhance the current query features for the subsequent refinement and produce the final detection results.

---

Besides, as shown in Figure I(d), we compare our query-based paradigm with the BEV-based and the proposal-bare based paradigm terms of FLOPs, latency, and performance. We can observe that our QTNet possesses distinct advantages in performance, computation cost, and latency. Overall, our contributions are summarized as follows: (i) We propose a new temporal fusion method called QTNe for 3D object detection based on sparse query-based feature representation, which is more effective and efficient than BEV-based and proposal-basee manners. (ii) We propose the MTM, which can be plugged into LiDyR-only or multi-modality 3D detectors and boost their performance with negligible computation cost and latency.
2 Related work
3D object detection
The current 3D object detection detectors can be roughly categorized into three
categories: LiDAR-based, camera-based, and multi-modality, For MIDAR-basede some methods [37,
56, 54, 52] directly consume pnoit clouds and predict the 3d detection results. Other methods [17, 27, 36, 50, 2, 18] first quantify point clouds into regular grid structures (e.g., voxels, pillars) and then utilize 3D backbone to extract features. These methods map 3D grids to bird-eye view (BEV) representation by height compression and utilize 2D CNN to extract features further for 3D object detection with efficiency. For camera- basedd some methods [21, 12, 20, 19] lift the image features into 3D space and then generate BEV features. Other methods [23, 42] follow the DETR [4] paradigm, and they utilize queries to interact with the multi-view image features and predict the 3D detection results. LiDAR-bases methods have precise geometric information with high performance but lack dense texture information. In contrast, baser methods have rich texture information but lack explicit 3D geometric information. Obviously, LiDAR and camera information are complementary. Therefore, multi-modality methods [14, 24, 51, 8, 2, 28, 22, 47, 15, 26, 25] utilize multi-modal fusion to improve detection performance.
3D multi-object tracking Due to the lack of rich texture information for point clouds, LiDARR based tracking methods [41, 50, 31, 33] usually utilize the motion of objects for the association. For example, CenterPoint [50] estimates the velocity by an additional regression head and utilizes L2 distance to associate objects in the previous and current frame. Simple [33] explores settings of tracking on different datasets through series of experiments. With the development of camera- based detectors, some methods [53, 32] have begun to explore conducting 3D multi-object tracking on the camera domain since there is rich appearance information in images. MUTR3D [53] uses queries to represent objects and achieves tracking by attention [39]. PF- Track [32] utilizes temporal transformer conduct temporal fusion and integrates past and future reasoning for tracked objects.
3D object detection with temporal fusion In the real world, temporal information is beneficial to perceive the localization and direction of movement for 3D objects. Towards this goal, some researchers exploit temporal information to boost the performance of 3D detection task. There two sets of mainstream, including BEV-based [49, 21, 16, 13] and proposal-based [46, 34, 5] temporal fusion. For BEV-based paradigm, the typical approach MGTANet [16] adopts an SM-VFE module to encode LiDAR point features and then a motion-guiidee aggregate muulti-frame BEV features for temporal fusion. However, this BEV-based paradigm takes both the foreground and background into consideration, leading to additional computation costs and sub-optimal performance. For proposal- based paradigm, MPPNet [5] utilizes Rol grid pooling [9] to extract proposal features and generate trajectories for temporal fusion. However, this proposal- based paradigm usually adopts a multi-stage pipeline and some time-come coine-conuumnn 3D Rol pooling operation, which makes the whole pipeline complex and brings unacceptable runtime. In summary, the above methods have the disadvantage of sub-optimal performance or high latency and computation cost. To address these problems, in this paper, we propose an efficient temporal fusion method QTNeet based on query representation, which achieves better performance and notably reduces the computation cost and latency.
deformable module align and
3 Method
In this section, we introduce the proposed Query-based Temporal Fusion Network (QTNet), which presented in Figure? 2. QTNet consists of a DETR-like 3D detector, a memory bank, and our proposed Motion-guided Temporal Modeling (MTM) module. In contrast to the prior arts, including
3

---

Updated Memory Bank
M Motion-guided Tomporal Modeling Modulc
Pop
Push
Results
DETR
Refinement
Point Clouds
1
Share
Weights
Optional
Fetch
N
Images
Memory Bank
Query Fcature
t Refined Results
Figure 2: The overall architecture of the proposed QTNet. We first feed sensor data the DETR- like 3D detector and generate current frame queries. We utilize a memory bank to store historical queries for query-based temporal fusion, which follows the FIFO rule. In the temporal fusion, we fetch these historical queries from the memory bank and then feed them to our motion motion-guided guided temporal modeling module fuse queries in different frames. Finally, the fused queries are utilized to refine current detection results.
BEV-based [16, 5, 49, 57] and proposal-based [34, 5J, conduct the temporal fusion sparse query representation. Specifically, we first feed LiDAR point clouds to DETR-like 3D detectors [2, 47, 15] and generate sparse query features. Note that method can also support multi-modal data as input (e.g., images and LiDAR point clouds). Thus, the input of images is optional. Then, we reserve the historical sparse query features by memory bank to reduce repeated computation costs. Next, take the query features of the previous frame and current frame into our MTM module and produce the enhanced query features for the final 3D detection. Next, will present the details of the main components in QTNet.
3.1 Memory Bank
Different from the existing temporal fusion methods [16, 49, 57] that require much repeated com- putation for the feature extraction from the previous frames, we utilize a memory bank to preserve the historical information for the subsequent temporal fusion. Here, we define memory bank S with the size of N x K, where means the number of used historical frames and K is the number of queries. Our memory bank follows the first-in and first-out (FIFO) rule to achieve query feature updates. In more detail, we first conduct temporal fusion in the t frame and reserve N historical frames. Then, we fetch queries 1Q--1, Qr-2, 2........... ...Q:-Nt x] in historical frames. Next, we can fuse these historical queries with the current queries Qt
obtain the enhanced current queries Q Finally, we push current queries Q; into 5 and pop the oldest queries Qt-N-
3.2 Motion-guided Temporal Modeling Module
The Notion-guided Temporal Modeling (MTM) module is applied to enhance the current query features by combining the feature information from the history frames. Specifically, we first obtain the current object queries Q and their corresponding positions Ct (can also be regarded as the center of 3D objects) from the DETR-like 3D detectors. In this paper, we adopt Transfusion-L [2] as the default. 3D detector. Then, we feed current queries Qt, previous fused queries Q, and their corresponding positions C2 and Ct-1 to MTM for generating current fused queries Q In more detail, we present the structure MTM in Figure 3, which includes three steps: the position alignment of objects between the previous and the current frames, the attention map generation, and the feature aggregation. For the process of this alignment, we first align objects from previous frame to current frame by the motion of objects and the ego vehicle. Given the ego pose matrix R(-1 and R in 1

---

Qt-1
Lincar
Add & Norm
Food Forward
Add & Norm
F t-1
Linear
Ct-1
Pr
C
Alignment
Multiplication
Attention Map Generation
Figure 3: The details of MTM. In the MTM, the center of queries different frames is used to produce the attention map of queries, which is then applied to enhance the current queries.
frame and t frame, we transform objects from the world coordinate to the ego coordinate. Then, the aligned transformation matrix Rt R which transforms objects from t- 1 frame frame, can be calculated as:
R = R inv(R
(1)
where the inv denotes the matrix inversion operation. Then, we can align the center of objects in t frame frame by their velocity V and the aligned transformation matrix R; with the related time interval AL between / and - frame:
c-1 = (C,-1 + V At) -RRG-1)
For the attention map generation, we first define the number of Qe is N and Qi-1 is N Then, utilize the L2 center distance of objects between adjacent frames build the cost matrix O R.N.xN--1:
L
(3)
where the La denotes the euclidean distance. To avoid the occurrence of impossible matching
situations for better temporal modeling, we introduce the L2 velocity error distribution of each category in 0.5 seconds as distance cost threshold 7, which is similar CenterPoint [50]. Besides, these associated objects with different categories are masked. We formulate the above process follows:
0. 0 S and = LEs.
M M- 11E.
81-1
(4)
0 > or S1 S;-1
where s, and st-1 denote the category of objects in f and t- 1 frames, respectively. After adopting the attention mask M with the row-wise softmax function on the cost matrix, we generate the attention map A, which is derived from the explicit position information:
= sofumax -1 0 M),
For feature aggregation, utilize the attention map to aggregate historical features. Given the current queries Q., previous fused queries Qi- and attention map A in Figure 3, the aggregated temporal features Ft-1 can be calculated as:
F. Φ2(A
(1())))) -.
(6)
where the and denote two linear layers. After aggregation, we can combine F- and Q, to generate the current fused queries Q
Q' Norm(Q (Q, Dropont(FFNNNormm)) + Dropout(Ff-1)))),
where Dropout, Norm, and FFN denote the dropout operation [38], the layer normalization [1]. and the the current frame through two simple FFNs, which are utilized to estimate the confidence of the classification branch and the residual of the regression branch for further refinement. Besides, follow [40, 35] to utilize the IoU branch to rectify the confidence score of prediction results in the current frame.
network, respectively. Finally, we utilize Q to refine the predicted results of
5

---

FFN
N
Classification
Classification
M
M
M
M
FFN
Regression
Regression
(a) Couple
(b) Decouple
Figure 4: Decouple temporal fusion. The decoupling manner conducts temporal fusion on classi- fication and regression branches separately for better detection performance. Here, the weights of classification and regression branches are not shared.
3.3 Decouple Strategy
We observe that there is imbalance in the classification and regression learning for temporal fusion. This phenomenon is also mentioned in VISTA [10] for 3D detection task. However, this decoupled manner brings more computation costs and latency, which is unacceptable for most BEV-based and proposal-basee methods. In contrast, benefiting from our efficient design of temporal fusion, the computation cost and latency of QTNet are negligible compared with the whole 3D detection network. Therefore, we decouple the classification and regression branches shown in Figure 4(b) instead coupling manner in Figure 4(a) that usually employed the previous temporal fusion methods [49, 5, 16]. Specifically, we separately implement the temporal fusion for the classification branch and the regression branch so as to achieve better detection performance.
4 Experiments
4.1 Experimental setup
Dataset We evaluate our method on the nuScenes dataset [3], which a large-scale autonomous driving dataset. It contains 700, 150, and 150 scenes for training. validation, and testing. Each scene roughly 20 seconds long and annotated at 2Hz, and provides point clouds acquired by 32-beam LiDAR and surrounding images acquired by 6 cameras. Note that each sample consists of one key frame and nine sweep frames. Besides, there are roughly 1.6M objects for 10 categories (barrier, bicycle, bus, motorcycle, pedestrian, trailer, truck, construction vehicle, and traffic cone) the 3D object detection task. For the evaluation metric, Mean Average Precision (mAP) [11] and nuScenes detection score (NDS) [3] are used to evaluate the performance of 3D detectors the nuScenes dataset.
4.2 Implementation details
Following the settings of nuScenes dataset, we concatenate point clouds of one key frame and nine sweeps as the input of sample, which is roughly 0.5 seconds long. Each point represented as (z, y, 2, At), where the (I, y, and At denote the 3D coordinate of point, the reflectance. and the time offset from the key frame, respectively.
Our method is implemented on 3D object detectors with LiDAR or multi-modality (LiDAR and images). For LiDAR, we take the TransFusion-L [2] as our LiDAR-only baseline. For multi- modality, we take the TransFusion [2] and DeepInteraction [47] as our multi-modality baseline. The voxelization range set [-54m,,5mm for both and Y axes and [-5m, 5mm for Z axis. The voxel size is set to (0.075m, 0.075m, 0m1.). We set the number of queries as 200 for training and testing. For temporal fusion, we utilize
historical frames. Note that we do not adopt any test time augmentation (TTA), multiple models ensemble, or future frames during inference.

---

Table 1: Comparison with state-of-the-arr methods on the nuScenes validation set. The L and C represent LiDAR and camera, respectively. The column of Frames denotes the number of key frame. denotes our reproduced results. i denotes future information is used.
Method
Year
Modality
Frames
Backhones
mAPt
NDS1
MVP [51]
NeurIPS 2021
LC
1
DLA34 & VoxelNet
67.1
70,8
AutoAlignV2 [8]
ECCV 2022
I.C
CSPNet VoxelNet
67.1
71.2
TransFusion [2]
CVPR 2022
LC
R50 & VoxelNet
71.3
BEVFusion [22]
NeurIPS 2022
LC
1
Swin-Timg & VoxelNet
67.9
71.0
DeepInteraction [47]
NeurIPS 2022
LC
1
R50 & VoxelNet
69.9
72.6
BEVFusion [28]
ICRA 2023
LC
I
Swin-Tine & VoxeINet
68.5
71.4
MSMDFuuion [15]
CVPR 2023
I.C
R50 & VoxelNet
69.3
72.1
CMT [44]
ICCV 2023
LC
V2-99 & VoxelNet
70.3
72.9
TransFusion" [2]
CVPR 2022
I.C.
1
R50 & VoxelNet
67.1
70.7
+QTNet
LC
4
R50 & VoxelNet
68.5
71.6
DeepInteraction" [47] +QTNet
NeurIPS 2022
LC
R50 & VoxelNet 69.9
72.6
LC
R50 & VoxelNet 70.3
73.1
CenterPoint [50]
CVPR 2021
I.
1
VoxeINet
59.6
66.8
TransFusion-L [2]
CVPR 2022
L
1
VoxeINet
65.1
70.1
[43]
ECCV 2022
L
10
VoxeINet
61.8
67.3
LidarMultiNet [48]
AAAI 2023
1
VoxeINet
63.8
69.5
VoxeINeXt [7]
CVPR 2023
L
VoxelNet
60.0
67.1
LargeKerneI3D [6]
CVPR 2023
L.
1
Voxel- Loxel-LargeKerne333
69.1
Link [30]
CVPR 2023
L
1
Voxel-LinK
63.6
69.5
MGTANet [16]
AAAI 2023
L
3
VoxeINet
62,9
68.7
MGTANet* [16]
AAAI 2023
I.
3
VoxeINet
64.8
70.6
TransFusion-L' [2]
CVPR 2022
L
1
VoxeINet
65.0
70.0
+QTNet
L
3
VoxeINet
66.3
70.8
+QTNet
L
4
VoxeINet
66.5
70.9
Table 2: Comparison with state-of-the-art methods on the nuScenes test set. information used.
denotes future
Method
Year
Modality
Frames
Backbones
mAPt
NDSf
CenterPoint [50]
CVPR 2021
I.
VoxelNet
60.3
67.3
TransFusion-L,
CVPR 2022
I.
VoxeINet
65.5
70.2
VISTA [10]
CVPR 2022
L
VoxeINet
63.7
70.4
LidarMultiNet [48]
AAAI 2023
L
VexeINet
67.0
71.6
VoxeINeXt [7]
CVPR 2023
L
VoxeINet
64.5
70.0
LargeKerneI3D [6]
CVPR 2023
L
Voxel-LargeKernel3D
65.3
70.5
Link [30]
CVPR 2023
L
Voxel-LinK
66.3
71.0
3DVIDI [49]
TPAMI 2021
L
VoxeINet
65.4
71.4
MGTANett [16]
AAAI 2023
L
VoxeINet
65.4
71.2
QTNet
L
VoxeINet
68.2
72.0
QTNett
L
VoxeINet
68.4
72.2
Our training process is divided into two stages. In the first stage, train the DETR-like 3D detectors (TransFusion-L. TransFusion, and DeepInteraction) with their default settings. In the second stage, we generate the memory bank of query features and prediction results. Then, we train our QTNe for 10 epochs without GT-Sampling [45] and CBGS [58] on four NVIDIA RTX 4090 GPUs. To optimize our network, we adopt the AdamW [29] optimizer with one-cycle learning rate policy, and the batch size is set to 16.
4.3 Comparison to the state of the art
compare the proposed QTNe with previous state-of-the-art 3D detectors on nuScenes [3] validation and test set. As shown in Table 1, QTNet shows consistent performance improvement when our module integrated into the LiDAR-only detector Transfusion-Le [2] or multi-modality methods Transfusion [2] and DeepInteraction [47]. Compared with the single frame LiDAR-only

---

Table 3: Comparison of computation cost and latency on the nuScenes validation set. denotes our reproduced results. The FLOPs and Latency are tested on a single NVIDIA RTX 4090 GPU with the batch size of
Method
Modality
mAP (%)1
NDS (%)
FLOPP (G)
Latency (ms))-
Params (M)
TransFusion" [2]
LC
67.1
70.7
445.9
201.2
37.0
+QTNet
LC
68.5
71.6
446.4
207.7
37.7
DeepInteraction [47]
LCC
69.9
72.6
499.2
355.0
57.8
+QUNet
I.C.
70.3
73.1
499.7
361.5
58.5
TransFusion-L." [2]
L
65.0
70.0
90.7
138.2
8.3
+QTNet
I.
66.5
70.9
90.8
144.7
8.6
Table 4: Compared with other paradigms on the TransFusion-L. denotes the fine-tuned results. Note that when introducing MGTANet into TransFusion-L. we need to fine-tune the detection head of TransFusion-L, which is harmful to the final detection performance. Therefore, for fair comparison, we choose the same fine-tuned TransFusion-L. on different paradigms.
Method
Paradigm
mAP
NDS
FLOPs (G)
Latency (ms)
Params (M)
TransFusion--L' [2]
63.1
67.8
90.7
138.2
8.3
+MGTANet [16]
BEV
(+0.9)
68.1 (+0.3)
+193.0
+22.1
+5.5
+MPPNet [5]
Proposal
63.4 (40.3)
68.2 (+0.4)
+131.3
+127.9
+7.0
+QTNet
Query
64.7 (+1.6)
69.0 (+1.2)
+0.1
+4.5
+0.3
baseline TransFusion-L. QTNet brings 1.3% (or 1.5%) mAP and 0.8% (or 0.9%) NDS improvements with 3 (or 4) frames, which achieves new SOTA performance for LiDAR-only detectors. Besides, QTNet even outperforms MGTANett, a manner of using the future frame information, by 1.5% mAP and 0.2% NDS. Note that QTNet does not utilize any future information since future information not be fetched the realistic scene of autonomous driving. Besides, compared with the single frame multi-modality baseline TransFusion, QTNet also produces 1.4% mAP and 0.9% NDS improvement. Based on the previous SOTA multi-modality method DeepInteraction, QTNet brings 0.4% mAP and 0.5% NDS improvement, achieving new SOTA result for multi-modality 3D object detection.
On the nuScenes test benchmark, we take TransFusion-L as our baseline. As shown in Table QTNet outperforms the advanced temporal fusion method MGTANet by 2.8% mAP and 0.8% NDS for 3 frames as input, which illustrates the superiority of QTNet. QTNet achieves 68.4 38.4% mAP and 72.2% NDS with 4 frames, setting cost and latency by integrating our QTNet into different baselines in Table 3. It can be seen that our method brings consistent performance improvement with negligible computation cost and latency.
SOTA performance. Besides, present the computation
4.4 Ablation studies
Query-based v.s. Other paradigms For a fair comparison, we integrate the representative long- term fusion of MGTANe[[16] and MPPNet[5] into our baseline Trans[usion-L[2] and keep the same number of frames as 3. As shown in Table 4, we present the results of QTNet and other methods in terms of the performance and computation cost. QTNet outperforms the BEV-based method method MPPNet 0.9% NDS and 0.8% NDS, respectively. For
MGTANet and the proposal-base computation cost and latency, QTNet only brings tiny increment of 0.1 GFLOPs and 4.5 ms based on the baseline of TransFusion-L. However, MGTANet even additionally produces 193.0 GFLOPs and 22.1 ms, and MPPNet brings 33.3 GFLOPs and 127.9 effectively demonstrate that QTNet possesses superior performance and less computation cost and latency when compared with other paradigms. Besides, QTNet is lightweight, which only brings tiny extra parameters of 0.3 M.
the same baseline. These experiments
Ablation studies on our QTNet We adopt the TransFusion-L: [2] as our baseline (I) under the case 3 frames as input and conduct ablation studies on our QTNet as shown Table 5(a). Here, we provide an available manner to model the relationship of adjacent frames for temporal fusion by the cross attention operation [39] (II for comparison. It can be observed that there is only a minor performance improvement with the mAP of 0.2% over baseline. In contrast, proposed MTM

---

Table 5: Ablation studies on the temporal fusion, evaluated on the nuScenes validation set.
(a) Ablation studies components of QTNet.
Ablation studies the number of frumes
Cross Attention
MTM
IoU
Decouple
mAP
NDS
Frames
NDS
FLOPs (G)
Latency
65.0
70.0
65.0
70.0
90.65
138.2
65.2
70.0
2
66.1
70.6
+0.07
+3.1
v
70.8
+0.10
+4.5
66.3
70.6
66.5
70.9
+0.14
+6.5
66.4
70.8
66.4
70.9
+0.24
+7.7
ID:2
ID:0
ID:0
ID:1
10:2
1D:2
ID:0
Dd:1
(a)
(b)
Figure 5: Attention map comparison. For better visualization, we select some predictions with high confidence score and calculate the attention map. The and column of attention map represent objects and t frame, respectively. The value of attention map is higher and the relevance is stronger. The attention map (a) generated by cross attention is ambiguous. The attention map (b) generated by our MTM more discriminative than cross attention.
(III obtains 1.2% mAP and 0.5% NDS performance gains over baseline. Besides, we apply the loU branch [35, 40] to rectify the confidence score of prediction results (IV), which can further boost the detection performance. Finally, we decouple the classification and regression of temporal fusion (V), which further brings 0.2% NDS performance gains. The above experiments demonstrate the effectiveness of our proposed components in our QTNet.
Different Number of Frames We explore the impact of detection performance on the different number of frames in the temporal fusion, whose results are shown in Table 5(b). We can find that as the number of frames increases from to 4, the performance gain gradually strengthening. When the number of frames reaches 5, the performance starts saturate. Besides, the computation cost and latency slightly increase as the number frames increases. However, thanks to our efficient design, the extra computation cost and latency much slight compared with whole network.
Why MTM is better? Intuitively, the cross attention operation [39] can directly construct the relevance among query features. However, objects with similar geometric structures between adjacent frames difficult distinguish in LiDAR point clouds, which makes the construction of the relationship between two frames less reliable. On the contrary, MTM builds an explicit motion modeling for temporal fusion by taking advantage of the fact that the same 3D objects between adjacent frames do not shift too much in a short time. To further illustrate the effectiveness of our MTM, we present the visualization for the manners of cross attention and our MTM as shown in Figure 5. Here, for convenience, we select three objects from predictions with high scores in and frames. The attention map generated by cross attention ambiguous, which leads sub-optimal performance for temporal fusion. On the contrary, the attention map generated by our MTM is more discriminative. Therefore, our MTM can establish more reliable relevance among queries than cross attention, which proves the rationality and superiority of our proposed MTM.
4.5 Limitation
Our method can be integrated into some advanced LiDAR-only or multi-modality DETR-like 3D detectors and brings performance improvements with negligible computation cost and latency. How- the main limitation is that the cost and latency of our method increase the number of frames

---

increases. In the future, we will explore recurrent architecture for 3D object detection with temporal fusion.
Conclusion
In this paper, wC propose a query-based temporal fusion network named QTNet for 3D object detection. Based on our query-based feature representation, introduce lightweight motion-guided temporal modeling MTM module to aggregate temporal features more efficiently and effectively. Extensive experiments demonstrate the superiority of our proposed QTNet by comparing it with the mainstream BEV-based and proposal-basee -based paradigms. Besides, QTNet even achieves new state-of- the-art the nuScenes dataset for LiDAR-only and multi-modality 3D detectors with negligible computation cost and latency compared with the baseline network.
Acknowledgement This work was supported in part by the National Science Fund for Distinguished Young Scholars of China (Grant No. 62225603), in part by the Hubei Key R&D Program (Grant No. 2022BAA078), and in part by the Taihu Lake Innovation Fund for Future Technology (HUST: 2023-A-1).
References
[1] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey Hinton. Layer normalization. In arXiv preprint Sakiv:667.006555 2016.
Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongho Fu, and Chiew-Lan Tai. Trans- fusion: Robust lidar-camera fusion for 3d object detection with transformers. IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[3] Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[4] Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference Computer Vision, 2020.
[5] Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Cheung, Hang Xu, and Hongsheng Li. Mppnet: Multi- frame feature intertwining with proxy points for 3d temporal object detection. In European Conference on Computer Vision, 2022.
[6] Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels 3d cnns. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[7] Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet object detection and tracking. IEEE Conference on Computer Vision and Puttern Recognition, 2023.
[8] Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang. Qinhong Jiang, and Feng Zhao. Autoaligny2: Deformable feature aggregation for dynamic multi-modal 3d object detection. In European Conference on Computer Vision, 2022.
[9] Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI Conference Artificial Intelligence, 2021.
[10] Shengheng Deng, Zhihao Liang, Lin Sun, and Kui Jia. Vista: Boosting 3d object detection via dual cross-view spatial attention. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[11] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. In International Journal of Computer Vixion, 2010.
[12] Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du Bevdet: hilh-peeformance multi-camera 3d object detection in bird-eye-view. In arXiv preprint ar 2021.
10

---

[13] Rui Huang. Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A Ross, Thomas Funkhouser. and Alireza Fathi. An Istm approach to temporal 3d object detection in lidar point clouds. In European Conference on Computer Vision, 2020.
[14] Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In European Conference on Computer Vision, 2020.
[15] Yang Jiao, Zequn Jic, Shaoxiang Chen, Jingjing Chen, Xiaolin Wei, Lin Ma, and Yu-Gang Jiang. Msmdfu- sion: Fusing lidar and camera multiple scales with multi-depth seeds for 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognitiin, 2023.
[16] Junho Koh, Junhyung Lee, Youngwoo Lee, Jackyum Kim, and Jun Won Choi. Mgtanet: Encoding sequential lidar points using long short-term motion-guided temporal attention for 3d object detection. In AAAI Conference on Artificial Intelligence, 2023.
[17] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[18] Jingyu Li, Zhe Liu, Jinghua Hou, and Dingkang Liang. Dds3d: Dense pseudo-labels with dynamic threshold for semi- superviuee 3d object detection. In IEEE International Conference on Robotics and Automation, 2023.
[19] Yinhao Li. Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. In AAAI Conference on Artificial Intelligence, 2023.
[20] Yinhao Li, Zheng Ge. Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi. Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In AAAI Conference on Artificial Intelligence. 2023.
[21] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie. Chonghao Sima. Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning sird's-eye-view representation from multi-camera images via spatiotemporal trans- formers. In European Conference on Computer Vision, 2022.
[22] Tingting Liang. Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang. and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. In Advances in Neural Information Processing Systems, 2022.
[23] Yingfei Liu. Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for nulti-view 3d object detection. In European Conference Computer Vision 2022.
[24] Zhe Liu, Tengteng Huang, Bingling Li, Xiwu Chen, Xi Wang, and Xiang Bai. Epnet++: Cascade bi- directional fusion for multi-modal 3d object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[25] Zhe Liu, Xiaoqing Yc, Xiao Tan, Errui Ding, and Xiang Bai. Stereodistill: Pick the cream from lidar for distilling stereo-based 3d object detection. In AAAI Conference Artificial Intelligence, 2023.
[26] Zhe Liu, Xiaoqing Ye, Zhikang Zou, Xinwei He, Xiao Tan, Errui Ding, Jingdong Wang, and Xiang Bai. Multi-moda 3d object detection by box matching. arXiv preprint arXiv:2305. 07713, 2023.
[27] Zhe Liu, Xin Zhao, Tengteng Huang, Ruolan Hu, Yu Zhou, and Xiang Bai. Tanet: Robust 3d object detection from point clouds with triple attention. In AAAI Conference on Artificial Intelligence, 2020.
[28] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfu- sion: Multi-task multi-sensor fusion with unified bird's-eye s-yee view representation. In IEEE International Conference on Robotics and Automation, 2023.
[29] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In arXiv preprint arXiv:1711.0510/ 2017.
[30] Tao Lu, Xiang Ding, Haisong Liu, Gangshan Wu, and Limin Wang. Link: Lincar kernel for lidar-based 3d perception. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[31] Chenxu Luo, Xiaodong Yang, and Alan Yuille. Exploring simple 3d multi-object tracking for autonomous driving. In IEEE International Conference Computer Vision, 2021.
11

---

[32] Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang. Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking. In IEEE Conference on Computer Vixion and Puttern Recognition, 2023.
[33] Ziqi Pang, Zhichao Li, and Naiyan Wang. Simpletrack: Understanding and rethinking 3d multi-object tracking. In arXiv preprint arXiv:211
2021.
[34] Charles R Qi. Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard 3d object detection from point cloud sequences. In IEEE Conference on Computer Vision and Puttern Recognition, 2021.
[35] Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and tieh-performance pillar- based 3d object detection, In European Conference on Computer Vision, 2022.
[36] Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-renm: Point-voxel feature set abstraction for 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2020.
[37] Shaoshuai Shi, Xiaogang Wang. and Hongsheng Li. Pointrean: object proposal generation and detection from point cloud. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
[38] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. In Journal of Machine Learning Research, 2014.
[39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, 2017.
[40] Haiyang Wang. Chen Shi, Shaoshuai Shi, Meng Lei. Sen Wang. Di He, Bernt Schiele, and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. In IEEE Conference Computer Vision and Pattern Recognition, 2023.
[41] Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. 3D Multi-Object Tracking: Baseline and New Evaluation Metrics. In IEEE/RSJ International Conference on Intelligent Robots and Systems, 2020.
[42] Kaixin Xiong, Shi Gong, Xiaoqing Ye, Xiao Tan. Ji Wan, Errui Ding. Jingdong Wang. and Xiang Bai. Cape: Camera view position emhodding for multi-view 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[43] Jianyun Xu, Zhenwei Miao, Da Zhang, Hongyu Pan, Kaixuan Liu, Peihan Hao, Jun Zhu, Zhengyang Sun, Hongmin Li, and Xin Zhan. Int: Towards infinite-frames 3d detection with efficient framework. In European Conference on Computer Vision, 2022.
[44] Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang. Cross modal transformer via coordinates encoding for 3d object dectection. IEFE International Conference on Computer Vision, 2023.
[45] Yan Yan, Yuxing Mao, and Bo Li. Second: Spansely embedded convolutional detection. In Sensors, 2018.
[46] Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3d-man: 3d multi-frame attention network for object detection. In IEEE Conference Computer Vision and Puttern Recognition, 2021.
[47] Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. In Advances in Neural Information Processing Systems, 2022.
[48] Dongqiangzi Ye, Zixiang Zhou, Weijia Chen, Yufei Xie. Yu Wang, Panqu Wang, and Hassan Foroosh. Lidarmultinet: Towards unified multi-task network for lidar perception. In AAAI Conference on Artificial Intelligence, 2023.
[49] Junbo Yin, Jianbing Shen. Xin Gao, David Crandall, and Ruigang Yang. Graph neural network and spatiotemporal transformer attention for 3d video object detection from point clouds. In IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[50] Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In IEEE Conference on Computer Vision and Pattern Recognition, 2021.
[51] Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühh. Multimodal virtual point 3d detection. In Advances in Neural Information Processing Systems, 2021.
12

---

[52] Dingyuan Zhang, Dingkang Liang, Zhikang Zou, Jingyu Li. Xiaoqing Ye. Zhe Liu, Xiao Tan, and Xiang Bai. A simple vision transformer for weakly semi-supervised 3d object detection. In IEEE International Conference on Computer Vision, 2023.
[53] Tianyuan Zhang. Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao. Mutr3d: A multi-camera tracking framework 3d-to-2d queries. IEEE Conference on Computer Vision and Pattern Recognition Workshop, 2022.
[54] Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, Yulan Guo. Not points are equal: Learning highly efficient point-based based detectors for 3d lidar point clouds. In JEEE Conference on Computer Vision and Pattern Recognition, 2022.
[55] Zhishuai Zhang, Jiyang Gao, Junhua Mao, Yukai Liu, Dragomir Anguelov, and Congcong Li. Stinet: apatio-temporralinntcacatet network for pedestrian detection and trajectory prediction. In IEEE Conference Computer Vision and Pattern Recognition, 2020.
[56] Wu Zheng, Weiliang Tang, Li Jiang. and Chi- Wing Fu. Se-ssd: Self-ensemblic single-stage object detector from point cloud. In IEEE Conference Computer Vision and Pattern Recognition, 2021.
[57] Zixiang Zhou, Xiangehen Zhao, Yu Wang. Panqu Wang, and Hassan Foroosh, Centerformer: Center-based transformer for 3d object detection. In European Conference Computer Vision, 2022.
[58] Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. In arXiv preprint arXiv: 1908. 190,0099,, 2019.
13

---

A Appendix
A.1 Performance the Camera-based method
Although we design our motion- motion--gided temporal modeling (MTM) module based on the LiDAR domain, we also explore the performance of MTM on camera- based methods. Thus, we integrate the MTM into the advanced camera- based detector CAPE [42] with two frames as input for temporal fusion on the nuScenes [3] validation set. As shown Table our MTM can also boost the performance of the camera-based method, which effectively demonstrates the generality of our method.
Table Performance of camera- based method with MTM. The represents camera. denotes our reproduced results. All models are trained by four NVIDIA RTX 4090 GPUs with 24 epochs and without CBGS [58]. The batch size is set to
Method
Year
Modality
Frames
Resolution
Backbone
mAOE
CAPE" [42]
CVPR 2023
C
1
704 256
R50
27.5
35.9
0.794
0.286
0.642
0.847
0.215
+MTM
704 256
R50
31.6
43.8
0.752
0.277
0.558
0.438
0.182
CAPE' [42]
CVPR 2023
C
1
800
320
V2-99
| 39.7
46.3
0.693
0.270
04488
0.747
0.206
+MTM
C
800 320
V2-99
43.9
53.6
0.656
0,266
0.380
0,350
0.183
A.2
Performance breakdown for each category
We report the detailed performance of QTNet for each category on the nuScenes [3] testing benchmark, as shown in Table 7. Compared with our LiDAR-only baseline TransFusion-L [2]. QTNet brings consistent improvements on most categories, especially on the construction vehicle (+7.0% AP), motorcycle (+6.6% AP), and bicycle (+6.3% AP).
Table 7: Comparison with state-of-the-art methods on the nuScenes testing set for each category. The and C represent LiDAR and camera, respectively. C.V., Ped., M.C., B.C., T.C., and B.R. represent construction vehiele, pedestrian, motorcycle, bicycle, traffic cone, and barrier, respectively. The column of Frames denotes the number of key frame. denotes future information used.
Method
Modality
Frames
mAP
NDS
Car
Truck
Bus
Trailer
C.V.
Pod.
M.C.
B.C.
T.C.
B.R.
CenterPoint
TransFusion-L [2]
65.5
70.2
86.2
56.7
66.3
58.8
28.2
86.1
68.3
44.2
82.0
78.2
VISTA [10]
63.7
70.4
84.7
54.2
64.0
55.0
29.1
83.6
71.0
45.2
78.6
71.8
LädarMultiNet [48]
67.0
71.6
86.9
57.4
64.7
61.0
31.5
87.2
75.3
47.6
85.1
73.5
VoxeINeXi [7]
64.5
70.0
84.6
53.0
64.7
55.8
28.7
85.8
73.2
45.7
79.0
74.6
LargeKerneIDD [6]
65.3
70.5
85.9
55.3
66.2
60.2
26.8
85.6
72.5
46.6
80.0
74.3
LinK [30]
66.3
71.0
86.1
55.7
65.7
62.1
30.9
85.8
73.5
47.5
80.4
75.1
3DVID+ [49]
3
65.4
71.4
87.5
56.9
63.5
60.2
32.1
82.1
74.6
45.9
78.8
69.3
MGTANet?
65.4
71.2
87.7
56.9
64.6
59.0
28.5
86.4
72.7
47.9
83.8
65.9
QTNet
68.2
72.0
86.5
57.2
68.3
63.0
34.3
88.1
74.9
49.7
82.7
77.0
QTNet
68.4
72.2
86.6
57.7
68.3
62.9
35,2
88.2
74.9
50,5
82.8
77.3
Besides, we report the detailed performance of QTNet for each category on the nuScenes [3] validation benchmark, shown in Table 8.
Table 8: Comparison with different baselines on the nuScenes validation set for each category. denotes reproduced results.
Method
Modality
Frames
mAP
NDS
Car
Truck
Trailer
C.V.
Ped.
M.C.
B.C.
T.C.
TransFusion" [2]
LC
67.1
70.7
87.7
61.6
75.9
42.4
26.5
88.0
75.2
63.8
77.3
72.2
+QTNet
68.5
71.6
87.8
63.0
89.3
77.5
68.8
78.3
DeepInteraction [47]
LC
69.9
72.6
88.5
64.4
79.2
44.5
30.1
88.9
79.0
67.8
80.0
76.4
+QTNet
LC
70,3
73.1
88.4
64.7
79.0
44.8
29.4
89.4
80.5
70.6
79.7
76.1
BEVFusion [28]
69.9
72.6
88.5
64.4
79.2
44.5
88.9
79.0
67.8
80.0
*QTNer
L.C
70.3
73.1
88.4
64.7
79.0
44.8
29.4
89.4
80.5
70.6
79.7
76.1
Trans-osion-L L"
65.0
70.0
86.7
60.4
75.3
41.6
24.6
86.8
71.8
56.5
74.4
71.8
+QTNet
66.3
70.8
87.1
61.1
75.5
43.0
25.7
87.8
75.2
61.2
75.6
71.4
*eTNet
66.5
70.9
87.2
61.5
75.8
43,0
25.7
87.8
75.5
61.5
75.4
71.4
14

---

A.3 Visualization
To illustrate the superiority of our QTNer, we visualize the results of TransFusion-L. [2] on the nuScenes [3] validation set for comparison. As shown in Figure 6, QTNet detect the hard- detected objects for TransFusion-L and boost the detection performance thanks to our proposed temporal fusion module MTM. As shown in Figure 7, QTNet successfully correct the angle error of objects for TransFusion-L thanks to our proposed temporal fusion module MTM. Besides, as shown in Figure 8, we compare TransFusion-L. and QTNet along the temporal dimension for better presentation. It can be seen that the object on the lower left, which is moving away from the ego vehicle, is not detected in t frame by TransFusion-L. However, QTNet can still capture the object in frame, benefiting from our effective temporal fusion.
(a)
(b)
Figure 6: Comparison of LiDAR-only baseline TransFusion-L (a) and QTNe (b) the nuScenes validation set. Blue and green boxes are the prediction and ground truth boxes. It can be seen that TransFusion-L fails to detect the hard-detected objects. However, thanks to the temporal information, QTNeet detects these objects successfully.
JHR
(a)
(b)
Figure 7: Comparison of LiDAR-om onlyy baseline TransFusion-L (a) and QTNet (b) about orientation of objects. Thanks to the temporal information, QTNet successfully corrected the orientation error.
15

---

(b)
Figure 8: Comparison of LiDAR-only baseline TransFusion-L (a) and QTNet (b) along the temporal dimension. The ego vehicle is moving from bottom to top.
A.4 Discussions of potential societal impacts
Effectively utilizing temporal information is vital for autonomous driving. QTNet improves 3D detection performance with negligible computation cost and latency by lightweight temporal fusion module MTM, which can utilize temporal information to improve the safety of autonomous driving in the real world. However, temporal fusion usually requires sensor synchronization in time, which puts forward higher requirements for the hardware of autonomous driving.
16


Reviews:
[{"review_id": "3Q1AEED5eu", "reviewer": "Reviewer_T8P4", "summary": "This paper introduces an approach that leverages object queries as a form of temporal memory. By establishing an attention map between queries from current and previous frames, the method effectively captures and encodes explicit motion information using velocity prediction and pose transformation. The proposed approach, referred to as QTNet, achieves superior performance compared to BEV-based or proposal-based techniques when evaluated on the nuScenes dataset. Furthermore, the Motion-guided Temporal Modeling Module (MTM) module can be seamlessly integrated as a plug-in component into other existing methods.", "questions": "Please see the Weaknesses", "limitations": "None", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "1. This article is well-written and can be easily reproduced based on its content.\n2. The visualization of the attention map reveals the reasons behind the superior performance of the proposed method.\n3. Despite its simplicity, this approach achieves significant performance improvements.\n4. Decouple Strategy is interesting.", "weaknesses": "1. In line 126, the memory bank stores $Q_{t - 1}$, however in line 154, you actually use previous fused queries $Q_{t - 1}'$.\n2. This article lacks innovation as it employs a manually defined attention map based on simple projections of velocity and pose, and the memory bank consists of straightforward query storage. Additionally, the performance of this approach is reliant on velocity prediction and tends to degrade in congested scenes.\n3. There should be more exploration and comparison of different paradigms. Currently, each paradigm is represented by only one method, lacking universality. The author could validate their approach using a simpler paradigm, such as employing the straightforward BEV feature query method used in BEVfusion. This kind of comparison would be more reliable and credible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your patient and detailed review. We try to address your comments below.\n### W1: In line 126, the memory bank stores $Q_{t-1}$ , however in line 154, you actually use previous fused queries ${Q^{'}}_{t-1}$.\nSorry for the misunderstanding. In fact, we only store historical queries $[Q_{t-1}, Q_{t-2}, ...Q_{t-N}]$ in the memory bank and then fused these historical queries to generate the ${Q^{'}}_{t-1}$ for enhancing current queries $Q_t$. We will make it clear in this original paper.\n\n### W2: Manually defined attention map based on simple projections of velocity and pose, and the performance of this approach is reliant on velocity prediction and tends to degrade in congested scenes.\nThanks. We summary your concerns as follows:\n\n**About manually defined attention map:** The defined attention map is simple, effective, and well-designed. Actually, 3D objects obey the physical law of motion in the real-world 3D space (LiDAR domain), which means that the same 3D object between adjacent frames does not shift too much in a short time. Therefore, utilizing this prior information is important to define the attention map among object queries, which makes our temporal model know where to pay attention, resulting in better performance.\n\n**About the reliance on velocity prediction:** To achieve promising detection performance, velocity prediction of objects is the critical information for almost all temporal-based 3D detectors (e.g. MPPNet, MSF, MGTANet) rather than only our method. \n\n**About the performance on congested scenes:** Since nuScence dataset does not provide the settings in the congested scene, we simply divide it into two scenes, which contains the crowded scenes (the number of objects is greater than 80) and the uncrowded scenes (the number of objects is less than 80). The corresponding results are shown in table. We can observe that our approach can obtain more obvious performance improvement in crowded scenes than that of uncrowded scenes, which illustrates the effectiveness of our proposed method.\n\n|  Method | Scenes | mAP | NDS  |\n|  ----   | :----: | :----: | :----: |\n|  TransFusion-L | uncrowded | 67.0 | 69.9 |\n|  QTNet | uncrowded | 67.9 | 70.7  |\n|  TransFusion-L | crowded | 64.4 | 68.7 |\n|  QTNet | crowded | 65.8 | 69.7 |\n\n### W3: There should be more exploration and comparison of different paradigms. \n\n**More exploration and comparison of different paradigms:** Thanks. The BEV-based paradigm conducts the temporal fusion on the dense BEV features, which may bring a lot of unnecessary computation cost on background. The proposal-based paradigm needs time-consuming operations to generate sparse 3D proposal features, and the performance highly depends on the quality of 3D proposals. In contrast, our query-based paradigm is based a sparse feature representation and can effectively aggregate the foreground object information, which is effective and efficient. Besides, our method can get rid of complex 3D RoI operations and less sensitive to 3D object of size and orientation than proposal-based representation.\n\n**More models for comparison:** Actually, we have selected one of the advanced and representative temporal fusion methods for each paradigm as comparison. To further illustrate the universality of our method, we select a new proposal-based method MSF, an improved version for MPPNet, as comparison. We can observe that QTNet consistently outperforms MSF in terms of performance and efficiency.\n\n|  Method | mAP | NDS | FLOPs (G) | Latency (ms) |\n|  ----   | :----: | :----: | :----: | :----: |\n|  baseline | 63.1 | 67.8 | 90.7 | 138.2 |\n|  +MPPNet | 63.4 | 68.2 | +131.3 | +127.9 |\n|  +MSF | 62.9 | 67.9 | +198.3 | +81.5 |\n|  +QTNet | 64.7 | 69.0 | +0.1 | +4.5 |\n\n**Validate on the BEVFusion:** We integrate our temporal fusion into BEVFusion by utilizing our method on the queries of BEVFusion. The experimental results show QTNet brings further performance improvement to BEVFusion with a small latency.\n\n|  Method | Modality | mAP | NDS | Latency (ms) |\n|  ----   | :----: | :----: | :----: | :----: |\n|  BEVFusion | LC | 69.6 | 72.1 | 965.5 |\n|  +QTNet | LC | 70.1 | 72.5 | +6.5 |"}, {"review_id": "95NqBz7n7N", "reviewer": "Reviewer_ycEE", "summary": "In this paper, the authors propose a simple and effective Query-based Temporal Fusion Network (QTNet). The main idea is to exploit the object queries in previous frames to enhance the current object queries by the proposed Motion-guided Temporal Modeling (MTM)  module,  Experimental results show the proposed QTNet outperforms BEV-based or proposal-based manners on the nuScenes dataset.", "questions": "See Weakness.", "limitations": "See Weakness.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed QTNet can be plugged into LiDAR-only or multi-modality 3D detectors.\n2. QTNet can boost 3D detector's performance with negligible computation cost and latency.\n", "weaknesses": "1) The innovation of the method is limited. The fusion of temporal features using motion information has similar ideas in both the proposal-based method （MPPNet，MSF [1]）and the query-based method （MOTRV2 [2]）. For example, MOTRV2 greatly improves the positioning and tracking performance of the 2D transformer-based tracker by fusing the timing query features and passing the location of the timing query.\n2) The comparison with proposal-based methods is not comprehensive.  \n\n      a) The fusion strategy of MPPNet only uses the historical proposal information that can match the current moment, that is, forms the trajectory, and discards other frames. This means that MPPnet cannot use historical information to restore frames missed at the current moment. The author can use motion information to transfer the unmatched past proposals to the current frame, like query-based startegy, and then verify the performance of MPPNet in this way, which can more fairly verify the performance of query-based strategy and proposal-based strategy.  \n\n     b) The comparison with SOTA methods, such as MSF, is missing. MSF also uses the motion-guided feature fusion strategy and achieve higher performance and efficiency than MPPNet.  \n\n3. The ablation experiments of MTM are not sufficient enough to verify the superiority of MTM design. The author should provide a simple baseline that just transfer all motion-aligned past queries to the current frame with an NMS to remove redundant temporal queries, and then send them to the decoder  with the current query together. Compared with this baseline, we can know more clearly whether it is the gain brought by the MTM attention design or just temporal queries.\n\n[1] MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences  \n[2] MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your patient and detailed review. We try to address your comments below.\n\n**Difference of using motion information with MPPNet and MSF:** Thanks. The main idea of our QTNet is different from MPPNet and MSF. MPPNet utilizes the velocity prediction for motion compensation in tracking, and then generate trajectory for temporal fusion. MSF utilizes the motion for propagating the current detection boxes to history frames for sampling point clouds. However, our method utilize the motion to establish the attention map among queries and fuse these queries by transformer, which is a more efficient way for temporal fusion.\n\n**Difference from MOTRv2:** MOTRv2 propagates the historical queries as the track queries in the current frame. Then it concatenates these queries as the $Q$ of transformer decoder, and the dense image features as the $K$ and $V$ of transformer decoder. In other words, MOTRv2 is a sparse-to-dense fusion strategy while our proposed MTM conducts the temporal fusion among sparse queries, which can be regarded as a sparse-to-sparse fusion strategy. Besides, MOTRv2 can work well by building the relationship in camera domain due to the rich appearance information in 2D images. However, objects with similar geometric structures between adjacent frames are difficult to distinguish in LiDAR domain, which makes the association between two frames less reliable. Therefore, we design the MTM operation to solve this problem. \n\n**Comparison with proposal-based methods:** Thanks. We propagate the historical detection results to the current frame to verify the performance MPPNet. As shown in table, there is a large performance degradation. The behind reason is that propagation operation may leads to a lot of false positive predictions and harm the final prediction performance.\n\n|  Method | Propagate | mAP | NDS |\n|  ----   |  :----:  | :----: | :----: |\n|  baseline |  | 63.1 | 67.8 |\n|  +MPPNet |  | 63.4 | 68.2 |\n|  +MPPNet | &#10003; | 61.5 | 67.3 |\n\nBesides, as shown in table, we compare MSF with MPPNet on the same baseline on the nuScenes dataset. Although MSF has lower latency than MPPNet, MSF produces worse performance. The main reason is that MSF only utilizes the velocity to move current boxes to previous frames and does not take into account the turning angle, which limits MSF generalize to long-sequence.\n\n|  Method | mAP | NDS | FLOPs (G) | Latency (ms) |\n|  ----   |  :----:  | :----: | :----: | :----: |\n|  baseline | 63.1 | 67.8 | 90.7 | 138.2 |\n|  +MPPNet | 63.4 | 68.2 | +131.3 | +127.9 |\n|  +MSF | 62.9 | 67.9 | +198.3 | +81.5 |\n|  +QTNet | 64.7 | 69.0 | +0.1 | +4.5 |\n\n**Comparison with simple baseline that just transfer all motion-aligned past queries to the current frame:** Thanks. We transfer the past queries to the current frame and send them to the decoder ('+Propagate' in table). We find that this temporal fusion manner does not bring obvious performance improvement. The main reason is that there lacks obvious feature distinction in LiDAR domain, which leads to having a difficulty in learning attention among object queries. However, MTM makes the learning process easier by our proposed explicit geometric constraint.\n\n|  Method | mAP | NDS |\n|  ----   | :----: | :----: |\n|  TransFusion-L | 65.0 | 70.0 |\n|  +Propagate | 65.1 | 70.0 |\n|  +MTM | 66.2 | 70.5 |"}, {"review_id": "4PGcNUhEFD", "reviewer": "Reviewer_BRpP", "summary": "This paper proposes a new strategy of fusing temporal information for camera-lidar based 3D object detectors. The main method is to design a plug-and-play module, which uses predicted velocity and vehicle ego information to compute the correspondence matrix among queries explicitly.  This module can be integrated into some advanced LiDAR-only or multi-modality 3D detectors.\n", "questions": "1. Could you please provide your baseline(DeepInteraction or Transfusion-L) multi-frame results, just aligning the previous frame to the current frame? I hope to know how much gain is brought by MTM, rather than multi-frame alignment enhancing feature representation.\n2. Could you please provide some visual samples? MTM should alleviate the problem of direction misdetection.", "limitations": "As mentioned before, using explicit geometric constraint violates the simplicity of DETR. Thus, I do not think the proposed strategy will be widely adopted by future work, although the Query-based alignment strategy shows benefits compared with the BEV-based and Proposal-based methods. Anyway, I hope the authors can provide me a discussion comparing the proposed Query-based strategy and directly conducting Attention among current and historical queries. ", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper provides a clear and well-structured overview of the approach, making it easy for readers to understand and follow.\n2. The proposed method brings competitive performance with negligible computation cost and latency on the nuScenes dataset.", "weaknesses": "1. The experiments are not very sufficient. For detectors of the DETR architecture, I think using explicit geometric constraint to match queries may not be necessary. Maybe directly conducting Attention operation among current queries and historical queries are enough to bring promising performance, just like StreamPETR. However, the authors do not analyze this issue. \n2. The method proposed in this paper does not bring enough benefits, This method works on the nuScenes dataset. It may be that  nuScenes has a large proportion of static data. I hope to verify it on other data sets. That will affect whether I am willing to improve the score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your patient and detailed review. We try to address your comments below.\n\n**Analyze the attention and MTM:** Good suggestion. As shown in table, We replace the MTM with attention operation and find that there is no noticeable performance gain. The main reason is that objects with similar geometric structures between adjacent frames are difficult to distinguish in LiDAR domain, which makes the association between two frames less reliable. However, StreamPETR can work well by building the relationship of queries between two frames in camera domain due to the rich appearance information in 2D images.\n\n|  Method | mAP | NDS |\n|  ----   | :----: | :----: |\n|  TransFusion-L | 65.0 | 70.0 |\n|  +StreamPETR | 65.2 | 70.0 |\n|  +MTM | 66.2 | 70.5 |\n\n**The method proposed in this paper does not bring enough benefits:** Thanks. We provide the results of our MTM on the Waymo dataset in table. Here, we select the representative DETR-like method ConQueR (CVPR 2023) as our baseline. Due to the limited time and computation resources, we only train all models on  training set with only 20% sequences to keep temporal consistency. As shown, our method even brings more obvious performance improvement on Waymo dataset, which illustrates its effectiveness.\n\n|  Method | Veh | Ped | Cyc | L2 mAPH |\n|  ----   | :----: | :----: | :----: | :----: |\n|  ConQueR | 58.7 | 62.7 | 49.3 | 56.9 |\n|  +QTNet | 59.5 | 63.3 | 54.1 | 59.0 |\n\n**Provide your baseline multi-frame results, just aligning the previous frame to the current frame:** Thanks. We align the previous queries to the current frame and send them to the decoder for comparison. As shown in table, directly transfering the past queries to current frame only brings 0.1% mAP improvement. In contrast, our MTM brings more promising performance improvement with 1.2% mAP.\n\n|  Method | mAP | NDS |\n|  ----   | :----: | :----: |\n|  TransFusion-L | 65.0 | 70.0 |\n|  +Propagate | 65.1 | 70.0 |\n|  +MTM | 66.2 | 70.5 |\n\n**Provide some visual samples:** Thanks. In fact, we have provided the visualization of detection results in the supplement materials. Besides, following your valuable suggestion, we highlight some visual samples for direction misdetection in the uploaded pdf of our rebuttal."}, {"review_id": "orJP8mj3zr", "reviewer": "Reviewer_DMZS", "summary": "The paper introduced a Query-based Temporal Fusion Network. It faciliate the object queries in previous frame to enhance the\ncurrent object queries by aproposed Motion-guided Temporal Modeling module, which utilizes both spatial and motion information to construct a cost matrix for efficient temporal fusion. The proposed method significantly improve upon the query-based baseline while incurring a negligible runtime cost.", "questions": "It is better to publish the results on NuScenes leaderboard and validate its method on Waymo Open Dataset.", "limitations": "The author addressed the limitations.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper introduces a novel temporal fusion paradigm that directly utilizes query-based features for achieving temporal fusion.\n2. The proposed framework significantly improves upon the query-based baseline method while incurring a negligible runtime cost.\n3. The paper is well-written, and the clarity of the figure enhances the understanding of the proposed concepts.", "weaknesses": "1. The novelty of this work is limited. This work is built on top of existing DETR-based detector and perform temporal fusion on query-based features. The idea is quite similar to CenterFormer. What makes different is that the query-based features are generated from DETR detector instead of sampling from heat map in CenterFormer. The memory bank is also not new.\n2. As the method use TransFusion as backbone model, it would be worth exploring the use of intermediate BEV features as the key and value for temporal fusion.  These features are expected to offer a richer context and can be also cached in the memory bank.\n3. Using the velocity \\times time to align the features across frames is not able to generalize to long-sequence as the object motion is not always constant and the turning is not considered.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your patient and detailed review. We try to address your comments below.\n\n**Difference from CenterFormer:** Thanks. In fact, our method is different from CenterFormer. CenterFormer conducts the temporal fusion between the current queries and historical BEV features, which is a sparse-to-dense strategy. But our method conducts the temporal fusion between current and historical queries, which is a sparse-to-sparse strategy. For the memory bank, we do not regard it as our contribution since it is only used to store historical information for avoiding redundant computation.\n\n**It would be worth exploring the use of intermediate BEV features as the key and value for temporal fusion:** Thanks. As shown in Table, we conduct the temporal fusion by achieving the interaction between sparse object queries (as Q) and dense BEV features (as K or V). We find that our method brings more improvement than directly conducting the temporal fusion between queries and BEV features. We think that establishing the relationship between current queries and historical BEV features is difficult since there are many backgrounds in BEV features, which hinders the instance-level temporal fusion. Besides, object queries have already aggregated context information from BEV features. Therefore, we think our proposed sparse-to-sparse temporal fusion is reasonable.\n\n|  Method | Q | K/V | mAP | NDS |\n|  ----   | :----: | :----: | :----: | :----: |\n|  TransFusion-L | - | - | 65.0 | 70.0 |\n|  +QTNet | Query Features | BEV Features | 65.3 | 70.1 |\n|  +QTNet | Query Features | Query Features | 66.2 | 70.6 |\n\n**Using motion to align the features across frames is not able to generalize to long-sequence as the object motion is not always constant and the turning is not considered:** Actually, our method achieves temporal fusion between two adjacent frames in a progressive temporal fusion mechanism (Please refer to the structure figure in the uploaded pdf of our rebuttal). Besides, we compute the mean error of velocity (mAVE) between adjacent frames as $0.24 m/s$, which is relatively small. Thus, we argue that our method is applicable to the long-sequence cases.\n\n**Publish the results on NuScenes leaderboard and validate on Waymo:** Thanks for your valuable suggestion. We publish the results on NuScenes leaderboard, which is included in the uploaded pdf. Besides, as shown in table, we integrate our QTNet into ConQueR on the Waymo dataset. Due to the limited time and computation resources, we only train all models on the 20% sequences (keep temporal consistency) of training set and validate on the full validation set. We find that there is an obvious performance improvement, illustrating the effectiveness of our method.\n\n|  Method | Veh | Ped | Cyc | L2 mAPH |\n|  ----   | :----: | :----: | :----: | :----: |\n|  ConQueR | 58.7 | 62.7 | 49.3 | 56.9 |\n|  +QTNet | 59.5 | 63.3 | 54.1 | 59.0 |"}, {"review_id": "K9OzegtRxg", "reviewer": "Reviewer_LNvD", "summary": "This paper proposes a small component that can be added on-top of single-frame 3D object detectors and perform (late) temporal fusion. This comes at a tiny computational cost and provides some performance improvements.", "questions": "Q1) How is $C_t$ obtained? Is it a fix anchor (like in AnchorDETR), a learnable anchor, or computed from $Q_t$ (like in DETR)?\n\nQ2) What are the details of the transformer architecture? Is it DETR, AnchorDETR, ConditionalDET, DeformableDETR, or something else?\n\nQ3) Is it correct that at each time-step that one would run the model, the MTM is run N+1 times (as shown in Figure 2)? That would not be as efficient as a fully recurrent model. Is this correct, and was a recurrent model considered?\n\nQ4) I do not understand what the cost matrix is, what it is used for, or how it is computed. How is this achieved?\n\nQ5) What is $C_t$? In l130 it seems as if it is a matrix of shape $\\mathbb{R}^{K\\times 3}$ (or possibly the transpose of that). In equation 3, however, the l2 vector norm is computed, which makes it seem as if $C_t\\in\\mathbb{R}^3$.\n\nQ6) What is the motivation for going as low as 200 queries? Image-based detectors seem to often adopt 300 queries (e.g., ConditionalDETR) whereas 3D detectors adopt 600 to 1200 (SpatialDETR). BEVFormer adopts 40000 queries, though it uses a sparse attention mechanism.\n\nQ7) What is the standard deviation between experiments? Please remember to not seed dataloading or weight initialization, and to retrain also the TransFusion/DeepInteraction backbone.\n\nQ8) Regarding cross-attention versus MTM qualitative results, could the same be achieved by temperature scaling the cross-attention? Or perhaps by performing cross-attention with positional encodings based on $C_{t-1}'$ instead of $C_{t-1}$ (i.e., correct the position with the predicted velocity of that object)?\n\nQ9) Why is relying on a DETR-like design a limitation (\"the main limitation\")?", "limitations": "The authors provide some discussion of limitations. Though, it is unclear to me why the main limitation is the reliance of a DETR-like design.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 4, "strengths": "S1) The background, as described in this paper in the introduction and related work sections, is apt yet concise.\n\nS2) The authors identify and tackle an important problem.\n\nS3) The idea -- to aggregate queries over time -- should be appreciated for its simplicity and is different from the approach taken by many prior works, e.g., BEVFormer.\n\nS4) The proposed approach can be added on-top of prior methods to obtain performance improvements (probably, see W4) at an insignificant computational cost. This is demonstrated on NuScenes with TransFusion and DeepInteraction.", "weaknesses": "W1) There are a few missing references in related work, for instance\n- SpatialDETR from ECCV2022.\n- The pioneering work DETR3D from conference on robotics learning.\n- TrackFormer, a simple yet elegant approach to multi-object tracking that is completely query-based.\n\nW2) One of the core motivations of the proposed approach is that BEV-based methods propagate information about the /background/ through time, which is unecessary (Section 1 second paragraph and Section 2 last paragraph). This hypothesis is not directly tested. I would have expected an experiment with BEV-based fusion in which the background was removed, to show that this is indeed unecessary.\n\nW3) Section 3.2, which constitutes the core of the proposed approach, is very difficult to understand (or possibly incorrect!). See questions Q1 to Q5 below.\n\nW4) The performance improvements are rather small (0.5 to 0.9 NDS). There is no analysis of standard deviation between different trainings, so it is difficult to appreciate whether these improvements are statistically significant.\n\nW5) There is no analysis of in what scenarios that the proposed approach provide performance improvements. While perhaps not strictly necessary, the performance improvements of the proposed approach are rather small and it is impossible to know whether this is because scenarios where temporal fusion is necessary are scarce in NuScenes or whether the proposed approach does not help so much in such scenarios. A more thorough analysis would be helpful to shed some light on the actual performance of the proposed approach.\n\nSome nitpicks)\n- The equations have poor formatting, always starting with a colon and not ending with a comma/period. Moreover, operators are written as variables, e.g., $and$ instead of $\\text{and}$.\n- $R$ is perhaps not the best choice for the transformation from world-coordinates to ego-coordinates, because $R$ is often used for rotations. In contrast, we also have a translation.\n- Equation 2 seems incorrect to me in that it behaves as if $R_{t-1}^t$ was in homogenous coordinates but I suspect that $C_{t-1}$ is not. Moreover, I think it is more common to let $C_{t-1}'\\in\\mathbb{R}^{3}$ than to put $C_{t-1}'\\in\\mathbb{R}^{1\\times 3}$ in order to have it left of the transformation matrix.\n- I think \"mathcal\" is usually used for sets or operators. It is a bit confusing to see it for the mask and attention matrices.\n- It is both clearer and more concise to write $A\\in\\mathbb{R}^{N_t\\times N_{t-1}}$ than \"matrix with shape of $N_t\\times N_{t-1}$\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your patient and detailed review. We try to address your comments below.\n\n**Missing references:** Sorry for missing these references. We will cite them into our submitted paper.\n\n**I would have expected an experiment with BEV-based fusion in which the background was removed:** Thanks. We conduct this experiment on BEV-based fusion method MGTANet and then removing all background by the provided ground truth. As shown in table, we surprisingly find that there is a significant performance improvement. However, it is very difficult to thoroughly distinguish foreground and background in practical situations.\n\n|  Method | mAP | NDS |\n|  ----   | :----: | :----: |\n|  MGTANet | 64.0 | 68.1 |\n|  +Remove Background | 94.6 | 82.5 |\n\n**There is no analysis of standard deviation between different trainings:** Thanks. We re-train our method for 3 times as shown in table. The corresponding standard deviation of NDS is $3.3e^{-5}$, which illustrates the stability of our method.\n\n|  # | mAP | NDS |\n|  ----   | :----: | :----: |\n|  1 | 66.47 | 70.86 |\n|  2 | 66.49 | 70.86 |\n|  3 | 66.46 | 70.86 |\n\n\n**There is no analysis of in what scenarios that the proposed approach provide performance improvements:** Valuable suggestion! To illustrate what scenarios that the proposed approach provide performance improvements, we provide the detailed results (mAP) for the categories of Motorcycle and Bicycle (fast-moving small objects), Traffic Cone and Barrier (static objects). We find that our method mainly improves the performance of fast moving small objects. For static objects, our method can maintain a good performance. For the results of more moving objects such as Car and Truck (please refer to the details in our supplemental materials), our method also brings promising performance improvement. We will add this analysis in the final version.\n\n|  Method | Motorcycle | Bicycle | Traffic Cone | Barrier |\n|  ----   | :----: | :----: | :----: | :----: |\n|  TransFusion-L | 71.8 | 56.5 | 74.4 | 71.8 |\n|  +QTNet | 75.5 | 61.5 | 75.4 | 71.4 |\n\n\n**Some errors in equations:** Thanks for pointing out these errors. We will revise them in the final version.\n\n\n### Some Questions\n**Q1:** The $C_t$ is computed from $Q_t$ by a DETR-like detection head.\n\n**Q2:** Sorry for making you misunderstand. Actually, the mentioned DETR in submitted paper is a DETR-like 3D detector. In particular, we select the TransFusion-L as our default DETR-like detector.\n\n**Q3:** In fact, MTM runs $N$ times rather $N+1$ times. Theoretically, although our method is not more efficient compared with a recurrent model, our MTM is lightweight and only brings a small amount of time overhead compared with the whole network. As shown in table, we can clearly observe that when adopting 5 frames as inputs, it only brings 7.7 ms additional time overhead.\n\n|  Frames | mAP | NDS | FLOPS (G) | Latency (ms) |\n|  ---- | :----: | :----: | :----: | :----: |\n|  1    | 65.0 | 70.0 | 90.65 | 138.2 |\n|  2    | 66.1 | 70.6 | +0.07 | +3.1 |\n|  3    | 66.4 | 70.8 | +0.10 | +4.5 |\n|  4    | 66.5 | 70.9 | +0.14 | +6.5 |\n|  5    | 66.4 | 70.9 | +0.24 | +7.7|\n\n**Q4:** The cost matrix is computed by the L2 distance of objects' center in previous and current frames. It is used to generate the attention map among objects for establishing relationship.\n\n**Q5:** \n\n(i) $C_{t}\\in \\mathbb{R}^{N\\times 3}$ means the centers of $N$ objects.\n\n(ii) Given the current $C_{t}\\in \\mathbb{R}^{N\\times 3}$ and previous $C_{t-1}\\in \\mathbb{R}^{M\\times 3}$, the L2 norms actually means the L2 distance between $C_{t}$ and $C_{t-1}$ so as to obtain the cost matrix $L\\in \\mathbb{R}^{N\\times M}$. We will revise them in final version.\n\n**Q6:** In fact, for a fair comparison, we keep the same value of 200 queries with The LiDAR-based (TransFusion-L) or Multi-modality DETR detectors (e.g. TransFusion, DeepInteraction, BEVFusion). Besides, for highly sparse 3D point cloud, setting 200 queries is usually sufficient to detect most objects\n\n**Q7:**As shown in table, we re-train our methods for 3 times based on TransFusion, and the corresponding standard deviation of NDS is $3.3e^{-5}$. We will make it clear in the final version.\n\n|  # | mAP | NDS |\n|  ----   | :----: | :----: |\n|  1 | 66.47 | 70.86 |\n|  2 | 66.49 | 70.86 |\n|  3 | 66.46 | 70.86 |\n\n**Q8:** Sorry for making you misunderstanding. In fact, we have implemented the cross-attention with the positional encoding based on the correct position (${C^{'}}_{t-1}$) and provide the  corresponding results in Table 5 of the original submitted paper. Here, we further simplify the table and show the results in the following table.\n\n|  # | Cross Attention | MTM | mAP | NDS |\n|  ---- | :----: | :----: | :----: | :----: |\n|  1 |  | | 65.0 | 70.0 |\n|  2 | &#10003; | | 65.2 | 70.0 |\n|  3 |  | &#10003; | 66.2 | 70.5 |\n\n**Q9:** To our best knowledge, DETR-like approaches are still developing in the 3D domain. In this paper, we mainly verify the superiority of our method for DETR-like 3D detectors. The effectiveness of 3D detectors for other paradigms remains to be explored in future research. We will make further clarification on limitations."}]

Author rebuttal:
We upload a pdf, which contains the visualization about orientation, results on the nuScenes leaderboard, and the illustration about our progressive temporal fusion mechanism.

Meta-review: The paper received mixed reviews. ycEE requests more comparisons with proposal-based methods and more ablation studies, which are adequately provided in the author rebuttal. T8P4 also requests more comparisons, which are provided in the rebuttal and authors are encouraged to include them in the revised version. BRpP questions the significance of the benefits and further analysis of the method, which are provided in the rebuttal and indicated by the reviewer as sufficient. DMZS states similarity to CenterFormer, for which the author response satisfactorily outlines the distinction and the reviewer maintains a rating to accept. LNvD requests additional analysis of standard deviations and scenarios that lead to improvements, which are addressed by the rebuttal and which the authors may include in the revised version. Overall, the meta-reviewer agrees that the concerns raised by the reviews have been comprehensively addressed by the rebuttal and the paper may be accepted.
Decision: Accept (poster)