Title: Query-based Temporal Fusion with Explicit Motion for 3D Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D Object Detection

Temporal

LiDAR-only

Multi-modality

Autonomous Driving
Abstract: Effectively utilizing temporal information to improve 3D detection performance is vital for autonomous driving vehicles. Existing methods either conduct temporal fusion based on the dense BEV features or sparse 3D proposal features. However, the former does not pay more attention to foreground objects, leading to more computation costs and sub-optimal performance. The latter implements time-consuming operations to generate sparse 3D proposal features, and the performance is limited by the quality of 3D proposals. In this paper, we propose a simple and effective Query-based Temporal Fusion Network (QTNet). The main idea is to exploit the object queries in previous frames to enhance the representation of current object queries by the proposed Motion-guided Temporal Modeling (MTM) module, which utilizes the spatial position information of object queries along the temporal dimension to construct their relevance between adjacent frames reliably. Experimental results show our proposed QTNet outperforms BEV-based or proposal-based manners on the nuScenes dataset. Besides, the MTM is a plug-and-play module, which can be integrated into some advanced LiDAR-only or multi-modality 3D detectors and even brings new SOTA performance with negligible computation cost and latency on the nuScenes dataset. These experiments powerfully illustrate the superiority and generalization of our method. The code is available at https://github.com/AlmoonYsl/QTNet.

Query-based Temporal Fusion with Explicit Motion for 3D Object Detection

Abstract

Effectively utilizing temporal information to improve 3D detection performance is vital for autonomous driving vehicles. Existing methods either conduct temporal fusion based on the dense BEV features or sparse 3D proposal features. However, the former does not pay more attention to foreground objects, leading to more computation costs and sub-optimal performance. The latter implements time-consuming operations to generate sparse 3D proposal features, and the performance is limited by the quality of 3D proposals. In this paper, we propose a simple and effective Q uery-based T emporal Fusion Net work (QTNet). The main idea is to exploit the object queries in previous frames to enhance the representation of current object queries by the proposed M otion-guided T emporal M odeling (MTM) module, which utilizes the spatial position information of object queries along the temporal dimension to construct their relevance between adjacent frames reliably. Experimental results show our proposed QTNet outperforms BEV-based or proposal-based manners on the nuScenes dataset. Besides, the MTM is a plugand-play module, which can be integrated into some advanced LiDAR-only or multi-modality 3D detectors and even brings new SOTA performance with negligible computation cost and latency on the nuScenes dataset. These experiments powerfully illustrate the superiority and generalization of our method. The code is available at https://github.com/AlmoonYsl/QTNet .

1 Introduction

3D object detection is the fundamental task of autonomous driving, which consumes data from sensors (e.g., LiDAR, Camera) to localize and recognize objects in the 3D world. Recent studies have explored utilizing temporal information to improve the detection performance. According to different manners of temporal fusion, we divide them into two categories, including BEV-based (Figure 1(a)) and proposal-based (Figure 1(b)).

BEV-based methods [16, 49, 55, 43, 57] conduct temporal fusion on the dense bird's-eye-view (BEV) feature by applying an affine transformation to align BEV features in previous frames to enhance the current BEV features. However, not all pixel-wise BEV features are beneficial for boosting the detection performance of foreground objects. To alleviate this problem, the proposal-based approaches [46, 34, 5] usually adopt a region proposal network (RPN) to obtain many 3D proposals

BEV-based Temporal Fusion

t

BEV feature

t - k

t - n

t

(a) BEV-based

Proposal-based Temporal Fusion

t

Pooling

Pooling

Pooling

t - k

t - n

t

(b) Proposal-based

Query-based Temporal Fusion

Query Feature

TopK

BEV feature

TopK

TopK

t - n

t

t - k

t - 1

(c) Query-based (Ours)

50

100

150

200

250

300

350

400

1

2

3

4

FLOPs (G)

Frames

BEV-based

Proposal-based

Query-based

100

150

200

250

300

350

1

2

3

4

Latency (ms)

Frames

BEV-based

Proposal-based

Query-based

63

63.2

63.4

63.6

63.8

64

64.2

64.4

64.6

64.8

65

1

2

3

4

mAP

Frames

BEV-based

Proposal-based

Query-based

(d) Comparison of different paradigms on the same baseline

一

O

I

and then achieve temporal fusion by focusing on foreground object features extracted by the timeconsuming 3D RoI-like operations [37, 36, 9]. However, the effectiveness of temporal fusion is bounded by the quality of the generated 3D proposals.

We argue that an effective and lightweight temporal fusion module is more meaningful to autonomous driving technology with high real-time and detection accuracy requirements. Towards this goal, we rethink the format of feature representation and the temporal modeling paradigm in temporal fusion for 3D object detection task, and propose a new query-based temporal fusion method called QTNet. Different from the BEV-based and proposal-based feature representation in Figure 1(a) and Figure 1(b), we adopt a query-based mechanism shown in Figure 1(c) that possesses two merits. On the one hand, the query-based representation is sparse and can effectively aggregate the foreground object information by attention operation [39]. On the other hand, the query-based format is more efficient due to getting rid of complex 3D RoI operations and is less sensitive to 3D object of size and orientation than proposal-based representation.

However, the temporal modeling based on query representation is non-trivial to work well. A natural manner is to compute the feature similarity between the queries of the current and previous frame by cross attention operation. However, the feature measurement is less reliable since 3D objects with similar geometric structures between adjacent frames are difficult to distinguish, leading to failed matching. Fortunately, we observe that 3D objects usually follow the physical law of motion in the real-world 3D space, which means that the same 3D object between adjacent frames does not shift too much in a short time.

Based on the above observations, we further adopt a Motion-guided Temporal Modeling (MTM) module based on the sparse query representation. Specifically, we first align objects from previous frame to current frame and then generate the attention map by explicit position information of objects. Finally, we aggregate the temporal features by the attention map to enhance the current query features for the subsequent refinement and produce the final detection results.

Besides, as shown in Figure 1(d), we compare our query-based paradigm with the BEV-based and the proposal-based paradigm in terms of FLOPs, latency, and performance. We can observe that our QTNet possesses distinct advantages in performance, computation cost, and latency. Overall, our contributions are summarized as follows: (i) We propose a new temporal fusion method called QTNet for 3D object detection based on sparse query-based feature representation, which is more effective and efficient than BEV-based and proposal-based manners. (ii) We propose the MTM, which can be plugged into LiDAR-only or multi-modality 3D detectors and boost their performance with negligible computation cost and latency.

2 Related work

3D object detection The current 3D object detection detectors can be roughly categorized into three categories: LiDAR-based, camera-based, and multi-modality. For LiDAR-based, some methods [37, 56, 54, 52] directly consume pnoit clouds and predict the 3d detection results. Other methods [17, 27, 36, 50, 2, 18] first quantify point clouds into regular grid structures (e.g., voxels, pillars) and then utilize a 3D backbone to extract features. These methods map 3D grids to bird-eye view (BEV) representation by height compression and utilize 2D CNN to extract features further for 3D object detection with efficiency. For camera-based, some methods [21, 12, 20, 19] lift the image features into 3D space and then generate BEV features. Other methods [23, 42] follow the DETR [4] paradigm, and they utilize queries to interact with the multi-view image features and predict the 3D detection results. LiDAR-based methods have precise geometric information with high performance but lack dense texture information. In contrast, camera-based methods have rich texture information but lack explicit 3D geometric information. Obviously, LiDAR and camera information are complementary. Therefore, multi-modality methods [14, 24, 51, 8, 2, 28, 22, 47, 15, 26, 25] utilize multi-modal fusion to improve detection performance.

3D multi-object tracking Due to the lack of rich texture information for point clouds, LiDARbased tracking methods [41, 50, 31, 33] usually utilize the motion of objects for the association. For example, CenterPoint [50] estimates the velocity by an additional regression head and utilizes L2 distance to associate objects in the previous and current frame. SimpleTrack [33] explores settings of tracking on different datasets through a series of experiments. With the development of camerabased detectors, some methods [53, 32] have begun to explore conducting 3D multi-object tracking on the camera domain since there is rich appearance information in images. MUTR3D [53] uses queries to represent objects and achieves tracking by attention [39]. PF-Track [32] utilizes a temporal transformer to conduct temporal fusion and integrates past and future reasoning for tracked objects.

3D object detection with temporal fusion In the real world, temporal information is beneficial to perceive the localization and direction of movement for 3D objects. Towards this goal, some researchers exploit temporal information to boost the performance of 3D detection task. There are two sets of mainstream, including BEV-based [49, 21, 16, 13] and proposal-based [46, 34, 5] temporal fusion. For BEV-based paradigm, the typical approach MGTANet [16] adopts an SM-VFE module to encode LiDAR point features and then a motion-guided deformable module to align and aggregate multi-frame BEV features for temporal fusion. However, this BEV-based paradigm takes both the foreground and background into consideration, leading to additional computation costs and sub-optimal performance. For proposal-based paradigm, MPPNet [5] utilizes RoI grid pooling [9] to extract proposal features and generate trajectories for temporal fusion. However, this proposal-based paradigm usually adopts a multi-stage pipeline and some time-consuming 3D RoI pooling operation, which makes the whole pipeline complex and brings unacceptable runtime. In summary, the above methods have the disadvantage of sub-optimal performance or high latency and computation cost. To address these problems, in this paper, we propose an efficient temporal fusion method QTNet based on query representation, which achieves better performance and notably reduces the computation cost and latency.

3 Method

In this section, we introduce the proposed Q uery-based T emporal Fusion Net work (QTNet), which is presented in Figure 2. QTNet consists of a DETR-like 3D detector, a memory bank, and our proposed Motion-guided Temporal Modeling (MTM) module. In contrast to the prior arts, including

M

Motion-guided Temporal Modeling Module

DETR

Push

···

Fetch

Pop

Query Feature

Memory Bank

GLYPH(cmap:d835)

···

···

···

M

M

M

Refinement

Images

Optional

Refined Results

Updated Memory Bank

Share

Weights

Results

···

···

···

GLYPH(cmap:d835)-1

GLYPH(cmap:d835)

-GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

BEV-based [16, 5, 49, 57] and proposal-based [34, 5], we conduct the temporal fusion on sparse query representation. Specifically, we first feed LiDAR point clouds to DETR-like 3D detectors [2, 47, 15] and generate sparse query features. Note that our method can also support multi-modal data as input (e.g., images and LiDAR point clouds). Thus, the input of images is optional. Then, we reserve the historical sparse query features by a memory bank to reduce repeated computation costs. Next, we take the query features of the previous frame and current frame into our MTM module and produce the enhanced query features for the final 3D detection. Next, we will present the details of the main components in QTNet.

3.1 Memory Bank

Different from the existing temporal fusion methods [16, 49, 57] that require much repeated computation for the feature extraction from the previous frames, we utilize a memory bank to preserve the historical information for the subsequent temporal fusion. Here, we define a memory bank S with the size of N × K , where N means the number of used historical frames and K is the number of queries. Our memory bank follows the first-in and first-out (FIFO) rule to achieve query feature updates. In more detail, we first conduct temporal fusion in the t frame and reserve N historical frames. Then, we fetch queries [ Q t -1 , Q t -2 , ...Q t -N ] in historical frames. Next, we can fuse these historical queries with the current queries Q t so as to obtain the enhanced current queries Q ′ t . Finally, we push current queries Q t into S and pop the oldest queries Q t -N .

3.2 Motion-guided Temporal Modeling Module

The Motion-guided Temporal Modeling (MTM) module is applied to enhance the current query features by combining the feature information from the history frames. Specifically, we first obtain the current object queries Q t and their corresponding positions C t (can also be regarded as the center of 3D objects) from the DETR-like 3D detectors. In this paper, we adopt Transfusion-L [2] as the default 3D detector. Then, we feed current queries Q t , previous fused queries Q ′ t -1 , and their corresponding positions C t and C t -1 to MTM for generating current fused queries Q ′ t . In more detail, we present the structure of MTM in Figure 3, which includes three steps: the position alignment of objects between the previous and the current frames, the attention map generation, and the feature aggregation. For the process of this alignment, we first align objects from previous frame to current frame by the motion of objects and the ego vehicle. Given the ego pose matrix R t -1 w and R t w in t -1

A Alignment

Multiplication

G Attention Map Generation

A

Linear

Linear

Add & Norm

Feed Forward

Add & Norm

GLYPH(cmap:d835)

GLYPH(cmap:d835)

'

G

GLYPH(cmap:d835)

GLYPH(cmap:d835)-1

'

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)-1

GLYPH(cmap:d835)

GLYPH(cmap:d835)-1

frame and t frame, we transform objects from the world coordinate to the ego coordinate. Then, the aligned transformation matrix R t t -1 , which transforms objects from t -1 frame to t frame, can be calculated as:

where the inv denotes the matrix inversion operation. Then, we can align the center of objects in t -1 frame to t frame by their velocity V t -1 and the aligned transformation matrix R t t -1 with the related time interval ∆ t between t and t -1 frame:

For the attention map generation, we first define the number of Q t is N t and Q t -1 is N t -1 . Then, we utilize the L2 center distance of objects between adjacent frames to build the cost matrix O t t -1 ∈ R N t × N t -1 :

where the L 2 denotes the euclidean distance. To avoid the occurrence of impossible matching situations for better temporal modeling, we introduce the L2 velocity error distribution of each category in 0 . 5 seconds as our distance cost threshold γ , which is similar to CenterPoint [50]. Besides, these associated objects with different categories are masked. We formulate the above process as follows:

̸

where s t and s t -1 denote the category of objects in t and t -1 frames, respectively. After adopting the attention mask M with the row-wise softmax function on the cost matrix, we generate the attention map A , which is derived from the explicit position information:

For feature aggregation, we utilize the attention map to aggregate historical features. Given the current queries Q t , previous fused queries Q ′ t -1 , and attention map A in Figure 3, the aggregated temporal features F t -1 can be calculated as:

where the ϕ 1 and ϕ 2 denote two linear layers. After aggregation, we can combine F t -1 and Q t to generate the current fused queries Q ′ t :

where Dropout , Norm , and FFN denote the dropout operation [38], the layer normalization [1], and the feed-forward network, respectively. Finally, we utilize Q ′ t to refine the predicted results of the current frame through two simple FFNs, which are utilized to estimate the confidence of the classification branch and the residual of the regression branch for further refinement. Besides, we follow [40, 35] to utilize the IoU branch to rectify the confidence score of prediction results in the current frame.

GLYPH(cmap:d835)

M

M

M

Classification

Regression

FFN

FFN

GLYPH(cmap:d835)-1

GLYPH(cmap:d835)

-GLYPH(cmap:d835)

(a) Couple

M

M

M

FFN

FFN

M

M

M

GLYPH(cmap:d835)

GLYPH(cmap:d835)-1

GLYPH(cmap:d835)

-GLYPH(cmap:d835)

GLYPH(cmap:d835)

GLYPH(cmap:d835)-1

GLYPH(cmap:d835)

-GLYPH(cmap:d835)

Classification

Regression

(b) Decouple

3.3 Decouple Strategy

We observe that there is an imbalance in the classification and regression learning for temporal fusion. This phenomenon is also mentioned in VISTA [10] for 3D detection task. However, this decoupled manner brings more computation costs and latency, which is unacceptable for most BEV-based and proposal-based methods. In contrast, benefiting from our efficient design of temporal fusion, the computation cost and latency of QTNet are negligible compared with the whole 3D detection network. Therefore, we decouple the classification and regression branches as shown in Figure 4(b) instead of the coupling manner in Figure 4(a) that is usually employed in the previous temporal fusion methods [49, 5, 16]. Specifically, we separately implement the temporal fusion for the classification branch and the regression branch so as to achieve better detection performance.

4 Experiments

4.1 Experimental setup

Dataset We evaluate our method on the nuScenes dataset [3], which is a large-scale autonomous driving dataset. It contains 700, 150, and 150 scenes for training, validation, and testing. Each scene is roughly 20 seconds long and annotated at 2Hz, and provides point clouds acquired by 32-beam LiDAR and surrounding images acquired by 6 cameras. Note that each sample consists of one key frame and nine sweep frames. Besides, there are roughly 1.6M objects for 10 categories (barrier, bicycle, bus, car, motorcycle, pedestrian, trailer, truck, construction vehicle, and traffic cone) on the 3D object detection task. For the evaluation metric, Mean Average Precision (mAP) [11] and nuScenes detection score (NDS) [3] are used to evaluate the performance of 3D detectors on the nuScenes dataset.

4.2 Implementation details

Following the settings of nuScenes dataset, we concatenate point clouds of one key frame and nine sweeps as the input of a sample, which is roughly 0.5 seconds long. Each point is represented as ( x, y, z, r, ∆ t ) , where the ( x, y, z ) , r , and ∆ t denote the 3D coordinate of a point, the reflectance, and the time offset from the key frame, respectively.

Our method is implemented on 3D object detectors with LiDAR or multi-modality (LiDAR and images). For LiDAR, we take the TransFusion-L [2] as our LiDAR-only baseline. For multimodality, we take the TransFusion [2] and DeepInteraction [47] as our multi-modality baseline. The voxelization range is set to [ -54 m, 54 m ] for both X and Y axes and [ -5 m, 3 m ] for Z axis. The voxel size is set to (0 . 075 m, 0 . 075 m, 0 . 1 m ) . We set the number of queries as 200 for training and testing. For temporal fusion, we utilize 2 or 3 historical frames. Note that we do not adopt any test time augmentation (TTA), multiple models ensemble, or future frames during inference.

Our training process is divided into two stages. In the first stage, we train the DETR-like 3D detectors (TransFusion-L, TransFusion, and DeepInteraction) with their default settings. In the second stage, we generate the memory bank of query features and prediction results. Then, we train our QTNet for 10 epochs without GT-Sampling [45] and CBGS [58] on four NVIDIA RTX 4090 GPUs. To optimize our network, we adopt the AdamW [29] optimizer with a one-cycle learning rate policy, and the batch size is set to 16.

4.3 Comparison to the state of the art

We compare the proposed QTNet with previous state-of-the-art 3D detectors on the nuScenes [3] validation and test set. As shown in Table 1, QTNet shows consistent performance improvement when our module is integrated into the LiDAR-only detector Transfusion-L [2] or multi-modality methods Transfusion [2] and DeepInteraction [47]. Compared with the single frame LiDAR-only

baseline TransFusion-L, QTNet brings 1.3% (or 1.5%) mAP and 0.8% (or 0.9%) NDS improvements with 3 (or 4) frames, which achieves a new SOTA performance for LiDAR-only detectors. Besides, QTNet even outperforms MGTANet†, a manner of using the future frame information, by 1.5% mAP and 0.2% NDS. Note that QTNet does not utilize any future information since future information can not be fetched in the realistic scene of autonomous driving. Besides, compared with the single frame multi-modality baseline TransFusion, QTNet also produces 1.4% mAP and 0.9% NDS improvement. Based on the previous SOTA multi-modality method DeepInteraction, QTNet brings 0.4% mAP and 0.5% NDS improvement, achieving a new SOTA result for multi-modality 3D object detection.

On the nuScenes test benchmark, we take TransFusion-L as our baseline. As shown in Table 2, QTNet outperforms the advanced temporal fusion method MGTANet† by 2.8% mAP and 0.8% NDS for 3 frames as input, which illustrates the superiority of QTNet. QTNet achieves 68.4% mAP and 72.2% NDS with 4 frames, setting a new SOTA performance. Besides, we present the computation cost and latency by integrating our QTNet into different baselines in Table 3. It can be seen that our method brings consistent performance improvement with negligible computation cost and latency.

4.4 Ablation studies

Query-based v.s. Other paradigms For a fair comparison, we integrate the representative longterm fusion of MGTANet[16] and MPPNet[5] into our baseline TransFusion-L[2] and keep the same number of frames as 3. As shown in Table 4, we present the results of QTNet and other methods in terms of the performance and computation cost. QTNet outperforms the BEV-based method MGTANet and the proposal-based method MPPNet 0.9% NDS and 0.8% NDS, respectively. For computation cost and latency, QTNet only brings a tiny increment of 0.1 GFLOPs and 4.5 ms based on the baseline of TransFusion-L. However, MGTANet even additionally produces 193.0 GFLOPs and 22.1 ms, and MPPNet brings 131.3 GFLOPs and 127.9 ms on the same baseline. These experiments effectively demonstrate that QTNet possesses superior performance and less computation cost and latency when compared with other paradigms. Besides, QTNet is lightweight, which only brings tiny extra parameters of 0.3 M.

Ablation studies on our QTNet We adopt the TransFusion-L [2] as our baseline (I) under the case of 3 frames as input and conduct ablation studies on our QTNet as shown in Table 5(a). Here, we provide an available manner to model the relationship of adjacent frames for temporal fusion by the cross attention operation [39] (II) for comparison. It can be observed that there is only a minor performance improvement with the mAP of 0.2% over baseline. In contrast, our proposed MTM

ID:1

ID:0

ID:2

(a)

(b)

GLYPH(cmap:d835)

1

0

ID:0

ID:1

ID:2

ID:0

ID:1

ID:2

GLYPH(cmap:d835)-1

0

0

6

8

8

L

2

4

6

6

(III) obtains 1.2% mAP and 0.5% NDS performance gains over baseline. Besides, we apply the IoU branch [35, 40] to rectify the confidence score of prediction results (IV), which can further boost the detection performance. Finally, we decouple the classification and regression of temporal fusion (V), which further brings 0.2% NDS performance gains. The above experiments demonstrate the effectiveness of our proposed components in our QTNet.

Different Number of Frames We explore the impact of detection performance on the different number of frames in the temporal fusion, whose results are shown in Table 5(b). We can find that as the number of frames increases from 1 to 4, the performance gain is gradually strengthening. When the number of frames reaches 5, the performance starts to saturate. Besides, the computation cost and latency slightly increase as the number of frames increases. However, thanks to our efficient design, the extra computation cost and latency are much slight compared with our whole network.

Why MTM is better? Intuitively, the cross attention operation [39] can directly construct the relevance among query features. However, objects with similar geometric structures between adjacent frames are difficult to distinguish in LiDAR point clouds, which makes the construction of the relationship between two frames less reliable. On the contrary, MTM builds an explicit motion modeling for temporal fusion by taking advantage of the fact that the same 3D objects between adjacent frames do not shift too much in a short time. To further illustrate the effectiveness of our MTM, we present the visualization for the manners of cross attention and our MTM as shown in Figure 5. Here, for convenience, we select three objects from predictions with high scores in t and t -1 frames. The attention map generated by cross attention is ambiguous, which leads to sub-optimal performance for temporal fusion. On the contrary, the attention map generated by our MTM is more discriminative. Therefore, our MTM can establish more reliable relevance among queries than cross attention, which proves the rationality and superiority of our proposed MTM.

4.5 Limitation

Our method can be integrated into some advanced LiDAR-only or multi-modality DETR-like 3D detectors and brings performance improvements with negligible computation cost and latency. However, the main limitation is that the cost and latency of our method increase as the number of frames

increases. In the future, we will explore recurrent architecture for 3D object detection with temporal fusion.

5 Conclusion

In this paper, we propose a query-based temporal fusion network named QTNet for 3D object detection. Based on our query-based feature representation, we introduce a lightweight motion-guided temporal modeling MTM module to aggregate temporal features more efficiently and effectively. Extensive experiments demonstrate the superiority of our proposed QTNet by comparing it with the mainstream BEV-based and proposal-based paradigms. Besides, QTNet even achieves new state-ofthe-art on the nuScenes dataset for LiDAR-only and multi-modality 3D detectors with negligible computation cost and latency compared with the baseline network.

Acknowledgement This work was supported in part by the National Science Fund for Distinguished Young Scholars of China (Grant No. 62225603), in part by the Hubei Key R&D Program (Grant No. 2022BAA078), and in part by the Taihu Lake Innovation Fund for Future Technology (HUST: 2023-A-1).

References

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. In arXiv preprint arXiv:1607.06450 , 2016.

Xuyang Bai, Zeyu Hu, Xinge Zhu, Qingqiu Huang, Yilun Chen, Hongbo Fu, and Chiew-Lan Tai. Transfusion: Robust lidar-camera fusion for 3d object detection with transformers. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision , 2020.

Xuesong Chen, Shaoshuai Shi, Benjin Zhu, Ka Chun Cheung, Hang Xu, and Hongsheng Li. Mppnet: Multiframe feature intertwining with proxy points for 3d temporal object detection. In European Conference on Computer Vision , 2022.

Yukang Chen, Jianhui Liu, Xiaojuan Qi, Xiangyu Zhang, Jian Sun, and Jiaya Jia. Scaling up kernels in 3d cnns. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Yukang Chen, Jianhui Liu, Xiangyu Zhang, Xiaojuan Qi, and Jiaya Jia. Voxelnext: Fully sparse voxelnet for 3d object detection and tracking. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Autoalignv2: Deformable feature aggregation for dynamic multi-modal 3d object detection. In European Conference on Computer Vision , 2022.

Jiajun Deng, Shaoshuai Shi, Peiwei Li, Wengang Zhou, Yanyong Zhang, and Houqiang Li. Voxel r-cnn: Towards high performance voxel-based 3d object detection. In AAAI Conference on Artificial Intelligence , 2021.

Shengheng Deng, Zhihao Liang, Lin Sun, and Kui Jia. Vista: Boosting 3d object detection via dual cross-view spatial attention. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. In International Journal of Computer Vision , 2010.

Junjie Huang, Guan Huang, Zheng Zhu, Yun Ye, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. In arXiv preprint arXiv:2112.11790 , 2021.

Rui Huang, Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A Ross, Thomas Funkhouser, and Alireza Fathi. An lstm approach to temporal 3d object detection in lidar point clouds. In European Conference on Computer Vision , 2020.

Tengteng Huang, Zhe Liu, Xiwu Chen, and Xiang Bai. Epnet: Enhancing point features with image semantics for 3d object detection. In European Conference on Computer Vision , 2020.

Yang Jiao, Zequn Jie, Shaoxiang Chen, Jingjing Chen, Xiaolin Wei, Lin Ma, and Yu-Gang Jiang. Msmdfusion: Fusing lidar and camera at multiple scales with multi-depth seeds for 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Junho Koh, Junhyung Lee, Youngwoo Lee, Jaekyum Kim, and Jun Won Choi. Mgtanet: Encoding sequential lidar points using long short-term motion-guided temporal attention for 3d object detection. In AAAI Conference on Artificial Intelligence , 2023.

Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Jingyu Li, Zhe Liu, Jinghua Hou, and Dingkang Liang. Dds3d: Dense pseudo-labels with dynamic threshold for semi-supervised 3d object detection. In IEEE International Conference on Robotics and Automation , 2023.

Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. In AAAI Conference on Artificial Intelligence , 2023.

Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In AAAI Conference on Artificial Intelligence , 2023.

Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In European Conference on Computer Vision , 2022.

Tingting Liang, Hongwei Xie, Kaicheng Yu, Zhongyu Xia, Zhiwei Lin, Yongtao Wang, Tao Tang, Bing Wang, and Zhi Tang. Bevfusion: A simple and robust lidar-camera fusion framework. In Advances in Neural Information Processing Systems , 2022.

Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. In European Conference on Computer Vision , 2022.

Zhe Liu, Tengteng Huang, Bingling Li, Xiwu Chen, Xi Wang, and Xiang Bai. Epnet++: Cascade bidirectional fusion for multi-modal 3d object detection. In IEEE Transactions on Pattern Analysis and Machine Intelligence , 2022.

Zhe Liu, Xiaoqing Ye, Xiao Tan, Errui Ding, and Xiang Bai. Stereodistill: Pick the cream from lidar for distilling stereo-based 3d object detection. In AAAI Conference on Artificial Intelligence , 2023.

Zhe Liu, Xiaoqing Ye, Zhikang Zou, Xinwei He, Xiao Tan, Errui Ding, Jingdong Wang, and Xiang Bai. Multi-modal 3d object detection by box matching. In arXiv preprint arXiv:2305.07713 , 2023.

Zhe Liu, Xin Zhao, Tengteng Huang, Ruolan Hu, Yu Zhou, and Xiang Bai. Tanet: Robust 3d object detection from point clouds with triple attention. In AAAI Conference on Artificial Intelligence , 2020.

Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. In IEEE International Conference on Robotics and Automation , 2023.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In arXiv preprint arXiv:1711.05101 , 2017.

Tao Lu, Xiang Ding, Haisong Liu, Gangshan Wu, and Limin Wang. Link: Linear kernel for lidar-based 3d perception. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Chenxu Luo, Xiaodong Yang, and Alan Yuille. Exploring simple 3d multi-object tracking for autonomous driving. In IEEE International Conference on Computer Vision , 2021.

Ziqi Pang, Jie Li, Pavel Tokmakov, Dian Chen, Sergey Zagoruyko, and Yu-Xiong Wang. Standing between past and future: Spatio-temporal modeling for multi-camera 3d multi-object tracking. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Ziqi Pang, Zhichao Li, and Naiyan Wang. Simpletrack: Understanding and rethinking 3d multi-object tracking. In arXiv preprint arXiv:2111.09621 , 2021.

Charles R Qi, Yin Zhou, Mahyar Najibi, Pei Sun, Khoa Vo, Boyang Deng, and Dragomir Anguelov. Offboard 3d object detection from point cloud sequences. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Guangsheng Shi, Ruifeng Li, and Chao Ma. Pillarnet: Real-time and high-performance pillar-based 3d object detection. In European Conference on Computer Vision , 2022.

Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrcnn: 3d object proposal generation and detection from point cloud. In IEEE Conference on Computer Vision and Pattern Recognition , 2019.

Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. In Journal of Machine Learning Research , 2014.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , 2017.

Haiyang Wang, Chen Shi, Shaoshuai Shi, Meng Lei, Sen Wang, Di He, Bernt Schiele, and Liwei Wang. Dsvt: Dynamic sparse voxel transformer with rotated sets. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Xinshuo Weng, Jianren Wang, David Held, and Kris Kitani. 3D Multi-Object Tracking: A Baseline and New Evaluation Metrics. In IEEE/RSJ International Conference on Intelligent Robots and Systems , 2020.

Kaixin Xiong, Shi Gong, Xiaoqing Ye, Xiao Tan, Ji Wan, Errui Ding, Jingdong Wang, and Xiang Bai. Cape: Camera view position embedding for multi-view 3d object detection. In IEEE Conference on Computer Vision and Pattern Recognition , 2023.

Jianyun Xu, Zhenwei Miao, Da Zhang, Hongyu Pan, Kaixuan Liu, Peihan Hao, Jun Zhu, Zhengyang Sun, Hongmin Li, and Xin Zhan. Int: Towards infinite-frames 3d detection with an efficient framework. In European Conference on Computer Vision , 2022.

Junjie Yan, Yingfei Liu, Jianjian Sun, Fan Jia, Shuailin Li, Tiancai Wang, and Xiangyu Zhang. Cross modal transformer via coordinates encoding for 3d object dectection. In IEEE International Conference on Computer Vision , 2023.

Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. In Sensors , 2018.

Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3d-man: 3d multi-frame attention network for object detection. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Zeyu Yang, Jiaqi Chen, Zhenwei Miao, Wei Li, Xiatian Zhu, and Li Zhang. Deepinteraction: 3d object detection via modality interaction. In Advances in Neural Information Processing Systems , 2022.

Dongqiangzi Ye, Zixiang Zhou, Weijia Chen, Yufei Xie, Yu Wang, Panqu Wang, and Hassan Foroosh. Lidarmultinet: Towards a unified multi-task network for lidar perception. In AAAI Conference on Artificial Intelligence , 2023.

Junbo Yin, Jianbing Shen, Xin Gao, David Crandall, and Ruigang Yang. Graph neural network and spatiotemporal transformer attention for 3d video object detection from point clouds. In IEEE Transactions on Pattern Analysis and Machine Intelligence , 2021.

Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Multimodal virtual point 3d detection. In Advances in Neural Information Processing Systems , 2021.

Dingyuan Zhang, Dingkang Liang, Zhikang Zou, Jingyu Li, Xiaoqing Ye, Zhe Liu, Xiao Tan, and Xiang Bai. A simple vision transformer for weakly semi-supervised 3d object detection. In IEEE International Conference on Computer Vision , 2023.

Tianyuan Zhang, Xuanyao Chen, Yue Wang, Yilun Wang, and Hang Zhao. Mutr3d: A multi-camera tracking framework via 3d-to-2d queries. In IEEE Conference on Computer Vision and Pattern Recognition Workshop , 2022.

Yifan Zhang, Qingyong Hu, Guoquan Xu, Yanxin Ma, Jianwei Wan, and Yulan Guo. Not all points are equal: Learning highly efficient point-based detectors for 3d lidar point clouds. In IEEE Conference on Computer Vision and Pattern Recognition , 2022.

Zhishuai Zhang, Jiyang Gao, Junhua Mao, Yukai Liu, Dragomir Anguelov, and Congcong Li. Stinet: Spatio-temporal-interactive network for pedestrian detection and trajectory prediction. In IEEE Conference on Computer Vision and Pattern Recognition , 2020.

Wu Zheng, Weiliang Tang, Li Jiang, and Chi-Wing Fu. Se-ssd: Self-ensembling single-stage object detector from point cloud. In IEEE Conference on Computer Vision and Pattern Recognition , 2021.

Zixiang Zhou, Xiangchen Zhao, Yu Wang, Panqu Wang, and Hassan Foroosh. Centerformer: Center-based transformer for 3d object detection. In European Conference on Computer Vision , 2022.

Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. In arXiv preprint arXiv:1908.09492 , 2019.

A Appendix

A.1 Performance on the Camera-based method

Although we design our motion-guided temporal modeling (MTM) module based on the LiDAR domain, we also explore the performance of MTM on camera-based methods. Thus, we integrate the MTMinto the advanced camera-based detector CAPE [42] with two frames as input for temporal fusion on the nuScenes [3] validation set. As shown in Table 6, our MTM can also boost the performance of the camera-based method, which effectively demonstrates the generality of our method.

A.2 Performance breakdown for each category

Wereport the detailed performance of QTNet for each category on the nuScenes [3] testing benchmark, as shown in Table 7. Compared with our LiDAR-only baseline TransFusion-L [2], QTNet brings consistent improvements on most categories, especially on the construction vehicle (+7.0% AP), motorcycle (+6.6% AP), and bicycle (+6.3% AP).

Besides, we report the detailed performance of QTNet for each category on the nuScenes [3] validation benchmark, as shown in Table 8.

A.3 Visualization

To illustrate the superiority of our QTNet, we visualize the results of TransFusion-L [2] on the nuScenes [3] validation set for comparison. As shown in Figure 6, QTNet can detect the harddetected objects for TransFusion-L and boost the detection performance thanks to our proposed temporal fusion module MTM. As shown in Figure 7, QTNet successfully correct the angle error of objects for TransFusion-L thanks to our proposed temporal fusion module MTM. Besides, as shown in Figure 8, we compare TransFusion-L and QTNet along the temporal dimension for better presentation. It can be seen that the object on the lower left, which is moving away from the ego vehicle, is not detected in t frame by TransFusion-L. However, QTNet can still capture the object in t frame, benefiting from our effective temporal fusion.

(a)

(b)

00 00)

四

000 00D)

四

(a)

(b)

口

四

四

四

高

四

四

四

四

三四

(a)

(b)

T

四

四

8

8

四

GLYPH(cmap:d835)-2

GLYPH(cmap:d835)-1

GLYPH(cmap:d835)

A.4 Discussions of potential societal impacts

Effectively utilizing temporal information is vital for autonomous driving. QTNet improves 3D detection performance with negligible computation cost and latency by a lightweight temporal fusion module MTM, which can utilize temporal information to improve the safety of autonomous driving in the real world. However, temporal fusion usually requires sensor synchronization in time, which puts forward higher requirements for the hardware of autonomous driving.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a small component that can be added on-top of single-frame 3D object detectors and perform (late) temporal fusion. This comes at a tiny computational cost and provides some performance improvements.\n\nSTRENGTHS:\nS1) The background, as described in this paper in the introduction and related work sections, is apt yet concise.\n\nS2) The authors identify and tackle an important problem.\n\nS3) The idea -- to aggregate queries over time -- should be appreciated for its simplicity and is different from the approach taken by many prior works, e.g., BEVFormer.\n\nS4) The proposed approach can be added on-top of prior methods to obtain performance improvements (probably, see W4) at an insignificant computational cost. This is demonstrated on NuScenes with TransFusion and DeepInteraction.\n\nWEAKNESSES:\nW1) There are a few missing references in related work, for instance\n- SpatialDETR from ECCV2022.\n- The pioneering work DETR3D from conference on robotics learning.\n- TrackFormer, a simple yet elegant approach to multi-object tracking that is completely query-based.\n\nW2) One of the core motivations of the proposed approach is that BEV-based methods propagate information about the /background/ through time, which is unecessary (Section 1 second paragraph and Section 2 last paragraph). This hypothesis is not directly tested. I would have expected an experiment with BEV-based fusion in which the background was removed, to show that this is indeed unecessary.\n\nW3) Section 3.2, which constitutes the core of the proposed approach, is very difficult to understand (or possibly incorrect!). See questions Q1 to Q5 below.\n\nW4) The performance improvements are rather small (0.5 to 0.9 NDS). There is no analysis of standard deviation between different trainings, so it is difficult to appreciate whether these improvements are statistically significant.\n\nW5) There is no analysis of in what scenarios that the proposed approach provide performance improvements. While perhaps not strictly necessary, the performance improvements of the proposed approach are rather small and it is impossible to know whether this is because scenarios where temporal fusion is necessary are scarce in NuScenes or whether the proposed approach does not help so much in such scenarios. A more thorough analysis would be helpful to shed some light on the actual performance of the proposed approach.\n\nSome nitpicks)\n- The equations have poor formatting, always starting with a colon and not ending with a comma/period. Moreover, operators are written as variables, e.g., $and$ instead of $\\text{and}$.\n- $R$ is perhaps not the best choice for the transformation from world-coordinates to ego-coordinates, because $R$ is often used for rotations. In contrast, we also have a translation.\n- Equation 2 seems incorrect to me in that it behaves as if $R_{t-1}^t$ was in homogenous coordinates but I suspect that $C_{t-1}$ is not. Moreover, I think it is more common to let $C_{t-1}'\\in\\mathbb{R}^{3}$ than to put $C_{t-1}'\\in\\mathbb{R}^{1\\times 3}$ in order to have it left of the transformation matrix.\n- I think \"mathcal\" is usually used for sets or operators. It is a bit confusing to see it for the mask and attention matrices.\n- It is both clearer and more concise to write $A\\in\\mathbb{R}^{N_t\\times N_{t-1}}$ than \"matrix with shape of $N_t\\times N_{t-1}$\".\n\nQUESTIONS:\nQ1) How is $C_t$ obtained? Is it a fix anchor (like in AnchorDETR), a learnable anchor, or computed from $Q_t$ (like in DETR)?\n\nQ2) What are the details of the transformer architecture? Is it DETR, AnchorDETR, ConditionalDET, DeformableDETR, or something else?\n\nQ3) Is it correct that at each time-step that one would run the model, the MTM is run N+1 times (as shown in Figure 2)? That would not be as efficient as a fully recurrent model. Is this correct, and was a recurrent model considered?\n\nQ4) I do not understand what the cost matrix is, what it is used for, or how it is computed. How is this achieved?\n\nQ5) What is $C_t$? In l130 it seems as if it is a matrix of shape $\\mathbb{R}^{K\\times 3}$ (or possibly the transpose of that). In equation 3, however, the l2 vector norm is computed, which makes it seem as if $C_t\\in\\mathbb{R}^3$.\n\nQ6) What is the motivation for going as low as 200 queries? Image-based detectors seem to often adopt 300 queries (e.g., ConditionalDETR) whereas 3D detectors adopt 600 to 1200 (SpatialDETR). BEVFormer adopts 40000 queries, though it uses a sparse attention mechanism.\n\nQ7) What is the standard deviation between experiments? Please remember to not seed dataloading or weight initialization, and to retrain also the TransFusion/DeepInteraction backbone.\n\nQ8) Regarding cross-attention versus MTM qualitative results, could the same be achieved by temperature scaling the cross-attention? Or perhaps by performing cross-attention with positional encodings based on $C_{t-1}'$ instead of $C_{t-1}$ (i.e., correct the position with the predicted velocity of that object)?\n\nQ9) Why is relying on a DETR-like design a limitation (\"the main limitation\")?\n\nLIMITATIONS:\nThe authors provide some discussion of limitations. Though, it is unclear to me why the main limitation is the reliance of a DETR-like design.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-19"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduced a Query-based Temporal Fusion Network. It faciliate the object queries in previous frame to enhance the\ncurrent object queries by aproposed Motion-guided Temporal Modeling module, which utilizes both spatial and motion information to construct a cost matrix for efficient temporal fusion. The proposed method significantly improve upon the query-based baseline while incurring a negligible runtime cost.\n\nSTRENGTHS:\n1. The paper introduces a novel temporal fusion paradigm that directly utilizes query-based features for achieving temporal fusion.\n2. The proposed framework significantly improves upon the query-based baseline method while incurring a negligible runtime cost.\n3. The paper is well-written, and the clarity of the figure enhances the understanding of the proposed concepts.\n\nWEAKNESSES:\n1. The novelty of this work is limited. This work is built on top of existing DETR-based detector and perform temporal fusion on query-based features. The idea is quite similar to CenterFormer. What makes different is that the query-based features are generated from DETR detector instead of sampling from heat map in CenterFormer. The memory bank is also not new.\n2. As the method use TransFusion as backbone model, it would be worth exploring the use of intermediate BEV features as the key and value for temporal fusion.  These features are expected to offer a richer context and can be also cached in the memory bank.\n3. Using the velocity \\times time to align the features across frames is not able to generalize to long-sequence as the object motion is not always constant and the turning is not considered.\n\nQUESTIONS:\nIt is better to publish the results on NuScenes leaderboard and validate its method on Waymo Open Dataset.\n\nLIMITATIONS:\nThe author addressed the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new strategy of fusing temporal information for camera-lidar based 3D object detectors. The main method is to design a plug-and-play module, which uses predicted velocity and vehicle ego information to compute the correspondence matrix among queries explicitly.  This module can be integrated into some advanced LiDAR-only or multi-modality 3D detectors.\n\nSTRENGTHS:\n1. The paper provides a clear and well-structured overview of the approach, making it easy for readers to understand and follow.\n2. The proposed method brings competitive performance with negligible computation cost and latency on the nuScenes dataset.\n\nWEAKNESSES:\n1. The experiments are not very sufficient. For detectors of the DETR architecture, I think using explicit geometric constraint to match queries may not be necessary. Maybe directly conducting Attention operation among current queries and historical queries are enough to bring promising performance, just like StreamPETR. However, the authors do not analyze this issue. \n2. The method proposed in this paper does not bring enough benefits, This method works on the nuScenes dataset. It may be that  nuScenes has a large proportion of static data. I hope to verify it on other data sets. That will affect whether I am willing to improve the score.\n\nQUESTIONS:\n1. Could you please provide your baseline(DeepInteraction or Transfusion-L) multi-frame results, just aligning the previous frame to the current frame? I hope to know how much gain is brought by MTM, rather than multi-frame alignment enhancing feature representation.\n2. Could you please provide some visual samples? MTM should alleviate the problem of direction misdetection.\n\nLIMITATIONS:\nAs mentioned before, using explicit geometric constraint violates the simplicity of DETR. Thus, I do not think the proposed strategy will be widely adopted by future work, although the Query-based alignment strategy shows benefits compared with the BEV-based and Proposal-based methods. Anyway, I hope the authors can provide me a discussion comparing the proposed Query-based strategy and directly conducting Attention among current and historical queries.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors propose a simple and effective Query-based Temporal Fusion Network (QTNet). The main idea is to exploit the object queries in previous frames to enhance the current object queries by the proposed Motion-guided Temporal Modeling (MTM)  module,  Experimental results show the proposed QTNet outperforms BEV-based or proposal-based manners on the nuScenes dataset.\n\nSTRENGTHS:\n1. The proposed QTNet can be plugged into LiDAR-only or multi-modality 3D detectors.\n2. QTNet can boost 3D detector's performance with negligible computation cost and latency.\n\nWEAKNESSES:\n1) The innovation of the method is limited. The fusion of temporal features using motion information has similar ideas in both the proposal-based method （MPPNet，MSF [1]）and the query-based method （MOTRV2 [2]）. For example, MOTRV2 greatly improves the positioning and tracking performance of the 2D transformer-based tracker by fusing the timing query features and passing the location of the timing query.\n2) The comparison with proposal-based methods is not comprehensive.  \n\n      a) The fusion strategy of MPPNet only uses the historical proposal information that can match the current moment, that is, forms the trajectory, and discards other frames. This means that MPPnet cannot use historical information to restore frames missed at the current moment. The author can use motion information to transfer the unmatched past proposals to the current frame, like query-based startegy, and then verify the performance of MPPNet in this way, which can more fairly verify the performance of query-based strategy and proposal-based strategy.  \n\n     b) The comparison with SOTA methods, such as MSF, is missing. MSF also uses the motion-guided feature fusion strategy and achieve higher performance and efficiency than MPPNet.  \n\n3. The ablation experiments of MTM are not sufficient enough to verify the superiority of MTM design. The author should provide a simple baseline that just transfer all motion-aligned past queries to the current frame with an NMS to remove redundant temporal queries, and then send them to the decoder  with the current query together. Compared with this baseline, we can know more clearly whether it is the gain brought by the MTM attention design or just temporal queries.\n\n[1] MSF: Motion-guided Sequential Fusion for Efficient 3D Object Detection from Point Cloud Sequences  \n[2] MOTRv2: Bootstrapping End-to-End Multi-Object Tracking by Pretrained Object Detectors\n\nQUESTIONS:\nSee Weakness.\n\nLIMITATIONS:\nSee Weakness.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces an approach that leverages object queries as a form of temporal memory. By establishing an attention map between queries from current and previous frames, the method effectively captures and encodes explicit motion information using velocity prediction and pose transformation. The proposed approach, referred to as QTNet, achieves superior performance compared to BEV-based or proposal-based techniques when evaluated on the nuScenes dataset. Furthermore, the Motion-guided Temporal Modeling Module (MTM) module can be seamlessly integrated as a plug-in component into other existing methods.\n\nSTRENGTHS:\n1. This article is well-written and can be easily reproduced based on its content.\n2. The visualization of the attention map reveals the reasons behind the superior performance of the proposed method.\n3. Despite its simplicity, this approach achieves significant performance improvements.\n4. Decouple Strategy is interesting.\n\nWEAKNESSES:\n1. In line 126, the memory bank stores $Q_{t - 1}$, however in line 154, you actually use previous fused queries $Q_{t - 1}'$.\n2. This article lacks innovation as it employs a manually defined attention map based on simple projections of velocity and pose, and the memory bank consists of straightforward query storage. Additionally, the performance of this approach is reliant on velocity prediction and tends to degrade in congested scenes.\n3. There should be more exploration and comparison of different paradigms. Currently, each paradigm is represented by only one method, lacking universality. The author could validate their approach using a simpler paradigm, such as employing the straightforward BEV feature query method used in BEVfusion. This kind of comparison would be more reliable and credible.\n\nQUESTIONS:\nPlease see the Weaknesses\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-08"}