Real-time Stereo-based 3D Object Detection for Streaming Perception
Authors: Changcai Li
Zonghua Gu
Gang Chen
Libo Huang
Wei Zhang
Huihui Zhou
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D object detection; Streaming perception; Real-time perception
Primary area: machine_vision

Abstract: The ability to promptly respond to environmental changes is crucial for the perception system of autonomous driving. Recently, a new task called streaming perception was proposed. It jointly evaluate the latency and accuracy into a single metric for video online perception. In this work, we introduce StreamDSGN, the first real-time stereo-based 3D object detection framework designed for streaming perception. StreamDSGN is an end-to-end framework that directly predicts the 3D properties of objects in the next moment by leveraging historical information, thereby alleviating the accuracy degradation of streaming perception. Further, StreamDSGN applies three strategies to enhance the perception accuracy: (1) A feature-flow-based fusion method, which generates a pseudo-next feature at the current moment to address the misalignment issue between feature and ground truth. (2) An extra regression loss for explicit supervision of object motion consistency in consecutive frames. (3) A large kernel backbone with a large receptive field for effectively capturing long-range spatial contextual features caused by changes in object positions. Experiments on the KITTI Tracking dataset show that, compared with the strong baseline, StreamDSGN significantly improves the streaming average precision by up to 4.33%. Our code is available at https://github.com/weiyangdaren/streamDSGN-pytorch.

Real-time Stereo-based 3D Object Detection for Streaming Perception
Changeai Li
Zonghua Gu
Gang Chen 122 Libo Huang
Wei Zhang
Huihui Zhou²
Sun Yat-sen University
Pengcheng Laboratory
Hofstra University
#National University of Defense Technology
1ichc56mail2 sysu. edu. en
zonghua wwaaiaaagghffsttr.mo edu
cheng830mail sysu. edu cn
libohuang@nudt. edu. cn
zhangwei 1at.:www11110000222.222 com zhouhh@pcl ac.
Abstract
The ability promptly respond to environmental changes is crucial for the per- ception system of autonomous driving. Recently, a new task called streaming perception proposed. It jointly evaluates the latency and accuracy into single metric for video online perception. first real-time stereo- basod 3D object detection framework designed for stream- ing perception. StreamDSGN is an end-to-end framework that directly predicts the 3D properties of objects in the next moment by leveraging historical infor- mation, thereby alleviating the accuracy degradation of streaming perception. Further, StreamDSGN applies three strategies enhance the perception racy: (1) A feature-flowe flowbbaed fusion method. which generates a pscudo-next feature at the current moment address the misalignment issue between feature and ground truth. (2) An extra regression loss for explicit supervision of object motion consistency in consecutive frames. (3) A large kernel backbone with a large receptive field for effectively capturing long- rangrengee spatial contextual fea- tures caused by changes object positions. Experiments on the KITTI Tracking dataset show that, compared with the strong baseline, StreamDSGN significantly improves the streaming average precision by up to 4.33%. Our code available https://pp /tttttb//gitubb... com hatta//...///aayyanngaarrrrreeaa pyttohh
this work, we introduce StreamDSGN, the
Introduction
Stereo-based 3D object detection [53 presents notable advantage of low-cost in the deployment compared to LiDAR- -based methods. However, existing stereo- based methods encounter challenges in delivering timely responses due to the intensive computations. Although there have been efforts towards lightweight approaches 35 38 the latency of the inference process is still non-negligible. This is because the latency will lead to the misalignment between the prediction of the latest frame and the real-time changing scene, as shown in Figure [
proposed for real-time online B 50 which only emphasize the detection accuracy, it simultaneously considers both accuracy and latency in the performance evaluation. With this metric, the model is forced to evaluate the processed world state against the prediction of the received frame. Hence, many high-accuracy but low-efficienne detectors [3 [7] will result in significant performance degradation for real-time perception applications.
Recently,
metric 30 52
named streaming accuracy
perception. Different from previous offline metrics I0
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Stereo frame
Compuration consumption åz
Scene at ( At afler processing
Misaligned Vehicle
Mienarching Pedestran
Figure In the context online streaming perception, the environment changes during inference.
Challenges. A widely recognized solution for addressing streaming perception to combine historical features and directly predict future out- comes at the current moment within a real-time framework [59 15]. This is because when the model's inference speed exceeds the input frame rate, the metric of streaming accuracy con- sistently matches and evaluates the results of the current frame with the ground truth of the next frame. However, several challenges arise in the implementation of this solution. As delineated in Figure 2 these challenges include the follow- ing three specific aspects:
Large
Smull
(al Low relative velocity
(b) lligh relative velocity
Prodicted trajectory
Ground tnuth trajecsory
Prediction
Gioond truth
Featare
Festure 1-1
Figure 2: Illustration of the challenges.
The first challenge the misalignment between the future supervisory signals and the current features. It is observed that for moving objects, their ground truth positions in the next frame at (depicted by the red bounding box) consistently differ from their current positions at time step f. In such cases, the model needs to learn geometric information about the foreground from distant historical features. Furthermore, this misalignment will be exacerbated the increasing disparity in relative velocities between the ego vehicle and the surrounding vehicles.
time step
The second challenge lies in the implicit supervision when only using the ground truth of a single future frame. It is observed that motion objects with diverse relative velocities exhibit distinct trajectory lengths (depicted by the red arrow). Previous works [59
28 15 lack explicit trajectory
supervision, requiring models to spontaneously learn various object motion offsets within latent space. Further, due the complexity of 3D object detection tasks, single regression supervision method may not be sufficient to ensure accurate detection of objects moving at different speeds
The third challenge arises from the effective utilization of context information embedded in the combined features. Due to the generally lower frame rates of 3D datasets compared to 2D ones, for example, the KITTI Tracking dataset [10] has frame rate of only 10Hz while Argoverse- [30] has 30Hz. Therefore, in 3D datasets, larger intervals will result in larger spatial distances between the objects in adjacent frames. At this point, feature extractor with small receptive field (RF) (depicted by the cyan areas) is insufficient to capture information from distant historical features.
Our solution. In this paper, we propose StreamDSGN, a streaming stereo-based 3D detector based on the advanced DSGN++ 7 architecture. It directly predicts the 3D properties of the next frame by leveraging the fusion of bird's-eye view (BEV) features from both the current and historical frames. Further, StreamDSGN combines three strategies to tackle the above-mentioned challenges as follows:
Feature-Flow Fusion (FFF). To address the first challenge of misalignment between features and ground truth, introduce novel fusion method based feature flow. It computes the feature flow between the current frame and the previous frame through similarity matching, and subsequently warps the current feature into the BEV coordinates of the next frame according to the flow map. This yields pseudo-next features aligned with the ground truth of the next frame.
In offline perception, this issue does not arise, since the ground truth is synchronized with each input frame.

---

Motion Consistency Loss function (MCL). To address the second challenge regarding implicit supervision, we establish additional explicit supervision based on motion consistency between adjacent frames. Specifically, we utilize the historical ground truth trajectories to supervise the predicted trajectory, guiding the model to recognize the offset magnitude and direction of object positions in the next frame.
Large Kernel BEV Backbone (LKBB). To address the third challenge involving large-span contextual information, we adopt large receptive field backhone to extract fused features. This structure follows existing research on large kernel convolutions [12 25,, aiming to enhance the model''s capacity to capture long-range dependencies.
We perform streaming simulations 330 52 and conduct comprehensive experiments on the KITTI Tracking dataset [10]. showing significant improvements in the streaming perception task. To the best of our knowledge, our work represents the first endeavor to explore the implementation of 3D object detection for streaming perception.
2 Related Work
Stereo-based 3D object detection. categories as follows:
The methods in this field can be broadly classified into three
1) AD-detection-basee methods. These methods 31 56
typically begin by feeding stereo image pairs into 2D detector or instance segmentation task generate prior information before performing 3D object detection. The works in (35] 43
implement a single-stage stereo
detection framework. Furthermore, Wu et al. (55) employ a semi-supervised method to enhance the foundational model by adopting plentiful unannotated images.
2) Littdd--iDARRbaseee methods. These methods [53 62 45 9 employ the stereo matching task
to obtain depth information from the scene and then transform it into a data structure resembling a LiDAR point cloud. Further, Li al. (27 leverage the confidence derived from semantic segmentation to enhance the pseudo-LiDAR The works in [26 38 adopt a binary neural network (BNN) to quantize the disparity estimator for significantly reducing the computational overhead.
adopt a representation that encodes stereo features derived from Siamese networks into differentiable geometric volumes. Further, Wang et al. [54) employ 3D occupancy to directly supervise the depth estimation model. The works in [33 36) improve the performance by leveraging knowledge distillation. Chen et al. [ perform depth- wise reconstruction of the volume in geometric modeling to alleviate the bottleneck of 2D to 3D information propagation. These methods represent tfateoff-the-at (SOTA) for stereo-basee 3D object detection, and our work explores the application of the work in 7 for streaming perception.
3) Geometric-vollme-busse
methods. The methods in 6
54
13
36 7
Streaming perception. The concept of streaming perception was initially introduced in [30], which evaluates streaming average precision (sAP) while accounting for latency considerations. With this metric, non-real-time detectors
33 17 will lead to great performance degradation since they unavoidably miss some intermediate frames. Therefore, Li et al. (30) further propose to address this
issue through decision-theoretic scheduling, asynchronous tracking M. and future prediction 22
The work in [6 corroborates the findings of [30] and extends their analysis to object tracking. Ghosh al. propose learned approximate execution framework to balance accuracy and latency implicitly. Instead of seeking better trade-offs or enhancing base detectors, StreamYOLO [59] simplifies streaming perception an end-to-end task of "predicting the next frame" with real-time detector. Based on this principle [59], LongShortNet [28] and DAMO-StreamNet [15) improve the
streaming accuracy by leveraging longer temporal fusion and knowledge distillation respectively. The
work in [52) expands the nuScenes dataset (3] and introduces a 3D benchmark tailored for streaming perception assignments. To the best of our knowledge, there are currently no existing applications of 3D perception algorithms for this task.
Temporal 3D object detection. These methods can be broadly classified into three categories: (1) The LiDAR requences-basel methods (60 frame by integrating historical features through temporal modeling. (2) The streaming data-based treat each LiDAR packet independent sample without requiring complete
61 63 39 complement the 3D shape within a single
methods [4. 8 5
3

---

sweeping by the LiDAR. (3) The Video-based methods [8 2 extend the image-based based methods by tracking and fusing the same objects across different frames. Note that the essential difference between streaming perception and these methods is the misalignment of ground truth and features.
3
Our Approaches
3.1 Base detector
We choose the recently proposed DSGN++ 7 as the base detector in our study. Due to its high latency, first goal is optimize its pipeline ensure the completion of inference for the current frame before the arrival of the next frame.
Future supervision
Mattion Comstonccy Lone
|Mol MIM
Bufferr quacie
Den
Flow Fuscon
Reg
Training only branch
gueue
(c) BEV Detection
Large BEY backbone
Detection head
Dir.
Glos
|
Training only
Current supervision
Stroan data impur
(0),15)
Sinmese network
hourglass
volurme
Pork
Build Con-Volume
(a) Feature Extractor
(b) Depth Regression
(d) Feature-Flow Fusion
Information
now
Supervision
2D/3 Come.
Blocks
Coms
1x1 2D
sample
3D dpwdown
Concostratiin
operation
Add
Shored weight
Figure 3: The architecture of StreamDSGN pipeline. (a) The feature extractor retrieves features from streaming stereo image pairs {(I =
- 1} and flattens them into BEV features. (b) The depth regression component utilizes Gdeyth as the supervision. (c) The BEV detector predicts the object state of the next moment by merging features from the current and previous frames. (d) The Feature-Flow Fusion generates pseudo-next feature peseudo by extrapolating from past features ++1
and then concatenates with the existing historical feature set Fr-1)
The optimization strategies are detailed in Appendix A. These strategies aim to ensure real-time processing of the detector without significantly sacrificing the offline accuracy. Experiments in Appendix A.2 show that the optimized model incurs only minor losses in offline accuracy while achieving a significant reduction in latency, decreasing from 263.33ms to 80.7 1m.7lms (with a frame rate of 10Hz for the KITTI Tracking dataset (10)
Next, we extend the optimized model into streaming data detection framework, illustrated in Figure During the training phase, we take the consecutive stereo images (T/",. . Ii) as input and directly predict the result Phor of the next frame. This data organization allows for random shuffling of training samples. Note that in streaming perception tasks, the input data cannot include time step + since we cannot access future frames the current moment in real-world scenarios. During inference, as the input frames are continuous, we only need to input the current frame and store its intermediate feature in a buffer queue for fusion with the next input.
our temporal fusion is executed before the BEV backbone instead of preceding the detection head. This is because believe that the shallow convolutional layers in the detection head lack an effective receptive field to extract large-span contextual information from the fused features. We validate this conjecture in the experiments presented in Appendix A.4.
Different from previous 2D approaches [59 28 [5],
3.2 Feature-Flow Fusion (FFF)
Different from fusion methods for temporal 3D object detection [19 complement the geometric shape of objects, our Feature-Flow Fusion (FFF) is designed to warp current feature to align with the next ground truth. The pipeline of FFF shown in Figure 4 utilizes historical changes to accomplish the warping based on motion consistency.
34 58
which aim to

---

Feoture Fe-1
shifling similanty computing
Flow
(1,4
(d,++
...
(4.4)
Feature Fes
presess
Festure P.
Figure A toy example of pseudo-next feature generation.
Calculation of Feature Flow. Recall that our input features are F., F..F,-1 E HXXxx The first
step of FFF involves computing the feature flow between the F, and F;-1- This process is similar to the computation of similarity volume in optical flow (00 48). We assume that the search space for the computing is discrete and rectangular. Specially, define shift indices set S
S {(-, -d), (-d, 1), (0.0),
(d, (0..,.. (d, d)}
(1)
where d is the maximal displacement. Then we shift the elements ( (i.j) -0, H 1; F,-1 according to each shift index in S, and match the shifted F.-1 with F; to
O,..., W -13 }} compute the cosine similarity si) of each matched elements. When considering single match with a shift index of (ue, V2), the similarity at pixel index (u, v) can be defined as:
s(u, = | u, v.)l,,,, v) ((c-1 u. v +v)) c,
(2)
where T represents the transpose operator. Stacking the whole similarity matrix = (i
O...., H j = O...., W }} obtained from different shift indices along new dimension, we can yield the similarity volume V E R HxWxD where D (2d + 1)²
Next, we asceraiin the feature flow Fi-He1 IR Hx x2 between F, and F.-1 1 by identifying index with the maximum similarity. This process can be formulated as:
F = gather (, arg maxv)
Note that the acquisition of Fi-r-1 does not require any learnable parameters. In our implementation, we accelerate the above process through parallelization, and reduce computation by downsampling Fi-1 and F, via max poooling, then restore the resolution of Fi-1-1 through bilinear interpolation.
Pseudo-nes next Feature Generation. According to the motion consistency, with a sufficiently high frame rate, the magnitude of displacement of object from to t 1 is consistent with that from t to + 1, but opposite directions, i.e., Fi-e+1 R -c-+t--- Thus, we can warp F, to the + grid by using inverted Fi---11 Let W (u, v) (x(u, v), y(u, v)) denotes the flow at pixel index (u, v) in Fi---11, where x and represent values in the vertical and horizontal directions, respectively. Let p (cr (i,j)li i = O,... // 1; = O,... W - represents the warped feature in / + grid. The warped element at (u, v) can be formulated as 21 20 48)
paeudo (u, v) = c, (u x (u, v),
(u, v))
(4)
We refer to the paseudo obtained through this pixel-level backward warping as pseudo-next feature. In theory, with a sufficiently fast frame rate and accurate matching, it can be aligned well with real Fr+1- As this process does not require the real F F.+1. it satisfies the constraints of streaming perception.
Note that ppseudo has limitations when dealing with occluded truncated objects, and the warping operation may introduce additional edge noise into the scene. Thus, we fuse it with historical features complement the geometric shape information of the objects. Similar [59], initially utilize weight-shared convolutions to reduce the channels of F F,, and Fpeeado and then concatenate them together. The concatenated features are connected with F, via add operator to enhance the representation of static information, as illustrated in part (d) of Figure 3 The latency our FFF process is only 7.67ms, please refer to Appendix A.5 for more latency and hyperparameter reports.
5

---

3.3 Motion Consistency Loss (MCL)
To explicitly guide the model in learning the offset magnitude, we propose a Motion Consistency Loss (MCL) as supplementary regression supervision, which consists of velocity loss and acceleration loss. This loss also grounded on the principle of motion consistency. leverages historical motion trajectories to guide the prediction of the next frame, as illustrated in Figure
The initial step in calculating the MCL involves establishing correspondences between bounding boxes across different time steps. We establish these correspondences between the ground truth of (Gber G Ghoz) j by utilizing target IDs. For the prediction Plog we calculate an loU (Intersection over Union) matrix with respect to Gb and employ max operation to identify the index corresponding to the highest matching value, thus establishing their correspondence.
Original regression loss
Pi
Vi
Mi-1-t-1
Trjectory
G
Predicted Trajectory
Ae-c-z
Let G denotes the ground truth center posi- tion and rotation angle of the objects at time
OPON
Velocity loss:
Me V.-c-l
Accelemation lass: [A A8-e-al
step i. where = - 2. - 1, such that G -27, u 2/. 2².0%). Similarly, P+1 (o -1 21 0" 11+ represents the predic-
Figure 5: Illustration of MCL.
tion for the next frame. Base on the correspondence, now we can calculate the displacement vector and the sine difference of the rotation angles between Prose and G for the predicted offset
VP ++1-+ (Ar", Ay", AzP, A
Ax x 27
Ay" = yi- - Vi,
A= _ ze: "111
AO sin 0.)
(5)
We can similarly calculate the ground truth offset V V between G! and G as supervision for the predicted offset V 1++1- Thus, the regression for velocity loss can be formulated as:
Lv = SmoothLL V(++1)
(6)
Further consideration is given to velocity change. We calculate the ground truth offset V
2-1--122
between G and Gr and then utilize the ground truth A A
V
V 1-1-1-2 of
We also employ the velocity change to supervise the prediction A-41-1111 V i+1 V 1---1-
Smooth Ll loss 57 to regress the acceleration consistency:
LA S1 ((
A 2
The MCL can then be defined as:
LMCL = Lv TLA
(8)
where 0.8. The grid search of can be seen in Appendix A.6 Let Lore denotes the original loss of DSGN++ 7 with the supervision of the next frame, our total loss is finally therefore:
L
Npos (Cors + ALMCL)
where Npos is the number of positive anchors and À 0.5.
3.4 Large Kernel BEV Backbone (LKBB)
Our LKBB is built upon Visual Attention Network (VAN) 12 which comprises a Large Kernel Attention (LKA) 22] module and Feed-Forward Network (FFN) [II, shown in the Figure 6a
The LKA module is composed of cascaded depth-wise convolution (DW-Conv), depth-wise dilation convolution (DW-D-Conv), and
convolution.
migrate VAN into our BEV backbone network increase its receptive field, as illustrated in Figure 6b For fair comparison, adjust the number of VAN blocks and the multiplier for channel

---

(eggt cleted, lend soo srskt (30)2))
(0:2.2
Trinspooe oocswoknn Rncoookaioon
Trenspoor oornabatio
Readual
multplication
guooss
BRRWN
(a) Architecture of a VAN block.
(b) Architecture of LKBB.
Figure Illustration of LKBB.
transformations to keep the parameter count of LKBB similar to that of the original structure. Further, to maintain the sensitivity of the model to fine-grained geometric details, we incorporate multi-scale features via residual connections.
Let f.tk.* denotes the transpose convolution operation, where c, and respectively represent the number filters, kernel size and stride. The multi-scale fusion process can formulated as:
Fout = fT. (
C 2220222 (F2) F1)
(10)
where Fr, F2 and out represent the downsampled features from two stages and the final output feature, respectively, with dimensions / x W 2C, H W 2C and H x W C.
4
We report the comparison of parameter count and computational complexity between our LKBB and the original hourglass backbone [41] in Appendix [A.7 Specifically, our parameter count is similar to the original structure while the computational complexity is reduced by approximately 6GFLOPs.
Experiments
4.1 Experimental setup
The prevalent autonomous driving datasets, such as nuScenes 3. nuScenes-H 552 and Waymo Open [50]. lack stereo image provision. Further, the stereo frame rate (only 5Hz) within Argoverse 4 is insufficient for streaming simulation. Consequently, we conduct our experiments on KITTI tracking dataset [00, which provides stereo images and higher frame rate (10Hz). We partition the training set with 20 scenes into numerous frame sequences, each comprising 40 frames. Among them, the even-numin nem-nnebered sequences are used for training with total of 4,291 frames, and the odd-nuebered sequences are used for testing with total of 3,672 frames. We analyze the domain gap of this partitioning method in Appendix A.3
Dataset.
Evaluation metrics. We follow 30
52 conduct the streaming simulation to obtain the streaming average precision (sAP) for both BEV and 3D perspectives. Consistent with KITTI [00, objects are categorized into three levels based on their recognition complexity: Easy, Moderate, and Hard. All of our precision measurements for experiments and ablation studies are calculated at IoU 0.7 and with 40 recall positions. If not specified, all results for the "Car" category.
Implementation details. Our experiments are conducted on the NVIDIA TITAN RTX platform with a total of 20 epochs. During the training phase, the Adam optimizer 23 employed in conjunction with the OneCycle learning rate decay strategy [47]. The initial learning rate is set to le 3. progressively increased le 2, and finally decayed 1 8. Further, for fair comparison with offline perception, we extend the copy papy-patt data augmentation [57] to streaming data. It cascades moving foreground objects into consecutive training frames enrich the training samples.
4.2 Comparison with Meta-detector
We re-implement the meta-detector named Streamer for streaming benchmark in the stereo-based 3D object detection domain. Following works [30 [52), Streamer respectively employs DSGN++ and DSGN++apt (equipped with real-time optimization strategies) as the base detector to predict the current frame, and then utilizes a Kalman filter to combine historical outputs for forecasting the
future state. Note that Streamer 330 52 is non-end-to-cn solution. In Table we compare

---

Table 1: Comparison with the meta-detector named Streamer. DSGN++opt represents the method equipped with our real-time optimization strategies. The symbol "i" denotes the basic framework of Stream YOLO [59]. which is built upon the +ot architecture.
Methods
Base detector
Latency (ms)
sAPsEv (lo11-0.5)
(lol1-0.5)
(lotl=d.7)
(loU1=0.7)
Easy
Mod.
Hard
Easy
Mod.
Hard
Easy
Mod.
Hard
Fasy
Mod.
Hard
Cur
Streamer
DSGIN++
263.33
14.76
11.56
10.62
12.51
8.86
7.74
5.26
3.42
3.11
2.82
1.92
1.51
50.36
30.89
22.50
19.96
25.50
YOLO
DSG
81.32
92.22
85,75
82,45
89.40
$2.89
78.10
80.72
68.51
63.14
73.02
58.37
51.86
StreaDDSGN (ours)
DSGN
91.45
93.10
87.46
84.34
92.53
84.55
81.34
$5.40
72.47
68.66
77.47
63.76
57.42
Pedestrion
Streumer
DSGN++
263.33
7.38
7.22
7.42
6.58
6.85
6.59
1.87
1.68
1,51
1,55
1.13
1.12
Streamer
DSGN+
80.71
58.23
$4.57
50.25
56.56
53.97
49.70
31.26
28.92
26.55
25.18
23.50
21.49
49.35
StreamDSGN (ours)
91.45
80.70
75.38
69.23
80.41
75.12
68.96
62.33
57.51
51.34
54.12
50.05
44.89
Cyclist
Streamer
DSGN++
3.74
Streamer
DSGNN+*ox
80.71
41.11
40.73
40.07
40.77 78
40.37
39.24
7.56
15.52
14.82
6.77
14.35
14.15
StreamYOLLO' D
81.32
39.34
41.31
40.59
38.77
40.20
39.59
31.37
34.03
33.43
30.63
33.14
32,36
StreamDSGNN (ours)
91.45
44.62
48.92
48.74
43.52
48.03
47.46
37.42
41.10
40.64
35.01
37.37
37.17
Table 2: Ablation studies of our methods. Setting a denotes the DSGN++opt- Setting represents directly predicting future results. Setting e incorporates historical feature based b and serves baseline method (highlighted gray). Settings d, e, f respectively incorporate our enhancement strategies. Setting g is our final proposed solution (highlighted in green).
Pipeline
SAPREV
SAP3D
Setting ID
FFF
MCL
LKBB
predict
fuse
Easy
Mod.
Hard
Easy
Mod.
Hard
30.89
22.50
19.96
25.50
17.68
14.75
53.73
47.46
43.36
38.62
32.55
29.39
83.20
71.15
65.74
75.38
59.55
53.11
d
85.08
71.98
66.49
75.88
61.57
54.77
84.63
71.01
66.18
78.15
62.28
56.80
f
84.97
71.96
66.53
76.74
60.31
55.01
g
v
85,40
72.47
68.66
77.47
63.76
57.42
method with these non-real-time and real-time settings. From these comparisons, we can make the following observations.
First, the Streamer 30 52 combined with DSGN+ 7 setting results in a noticeable inferior in streaming accuracy. In streaming simulation, find that this is due to the large-span temporal interval, making it challenging for the Kalman filter 22 to accurately track each target in multi- object scenarios. Second, the Streamer [30, 52 combined with the DSGN++opt setting shows higher performance compared to the DSGN++ combined one due to the latency reduction. However, with high matching threshold (LoU 0.7), the improvement in streaming accuracy is quite limited. Third, the Stream (59] framework built upon DSGN++apt achieves significant improvements across all metrics. These results demonstrate the competitiveness of the end-to-end framework that directly predicts future states compared to non-end-tee con-enn-to-end solutions in the benchmark. Finally, our StreamDSGN reaches the highest level performance, with approximately a 5% improvement in each metric compared to the extended version of Stream YOLO [59]. Despite the latency in this setting reaching 91. m5,55m, the inference speed remains faster than the input frame rate, thus meeting real-time requirements.

---

Vanilla DSGN+
Misttatch
Misaliganent
Object PEST
pipeline
tranching
Firer dinection
Enhanced pipeline
muiching
Correct direction
Hybrid velocity environment
High relutive velocity environment
Low relative velocity environment
Figure 7: Qualitative analysis of different relative velocity scenarios. We visualize the predictions in point clouds for a clearer comparison. The first row describes DSGN++ (7] without any modifications. The second row integrates real-time optimization and fusion historical frames to predict the next frame. The third row showcases our three enhancement strategies. Ground truth and prediction instances are respectively delineated by red and green bounding boxes, with lines inside the boxes indicating the orientation of objects.
4.3
Ablation Studies
To validate the effectiveness of each strategy, we conduct ablation experiments by toggling them on and off. Experimental results is illustrated in Table 2 In the following, we provide a more detailed description of the ablation experiments.
Effectiveness of the Pipeline. The comparison between setting b and setting a reveals that directly predicting future states by using real-time detector can alleviate the accuracy degradation caused by streaming perception constraints. However, the improvement in accuracy at this time is limited due to the lack of information on each object's displacement magnitude. Furthermore, by incorporating historical features, streaming accuracy can be significantly improved (see setting e). This improvement occurs because the fused features implicitly embed motion cues, enabling the model learn different offsets for each object. Thus, setting e serves as our baseline and is utilized for subsequent ablation studies to assess each enhancement strategy's 's effectiveness.
Effectiveness of Enhanced Strategies. The comparison of settings d, e, and f with setting demonstrates the effectiveness of each enhancement strategy, respectively. From these comparisons, we can make the following observations.
First, since the fusion scheme used in setting is derived from the Dual-Flow module of Stream Y- OLO [59] the comparison between setting and setting e can be viewed as a comparison between our FFF and the SOTA streaming perception fusion method. The results show that our FFF achieves improvements across all metrics. Second, setting reveals the effectiveness of MCL, as the additional supervision of trajectory consistency yields significant improvements. Compared to the baseline, improvements across the three levels reach 2.77% 2.73%. and 3.69% in SAP3D. respectively. Third, due the consistency between the downsampling dimensions of the feature maps LKBB and the baseline, as well as the similar parameter counts between the two structures, it is evident that the improvements in setting are primarily attributed to the advantage of the large receptive field feature extractor in capturing long-range contextual features. Finally, when we combine all strategies together (highlighted in green), the improvement in streaming accuracy reaches peak, with a 4.33% increase in sAP3D compared to the baseline at the hard level.

---

10
002
(a) Previous feature
(b) Current feature
(c) Next feature
(d) Pseudo-next feature
Figure 8: Qualitative analysis of the pseudo-nexl feature. The first row displays complete feature maps from different time steps, while the second row shows corresponding local regions of the feature maps. The solid red box and dashed box respectively represent the ground truth the next frame used for supervision and the locally zoomed-in area.
4.4 Qualitative Analysis
Visualization of Streaming Detection. Figure presents the qualitative results of our proposed pipeline compared the vanilla DSGN++ (7). In scenarios where surrounding vehicles remain relatively stationary, DSGN++ [7] demonstrates robust capacity for accurately aligning with ground truth. However, as these vehicles initiate motion relative the ego vehicle, DSGN+ 77 exhibits misalignments or mismatches in predictions due to the latency. In contrast, our baseline pipeline adeptly addresses this issue, With the enhancement strategy incorporating FFF, MCL and LKBB, the alignment between predicted boxes and ground truth boxes is further improved.
Visualization Pseudo-next Feature. We visualize the pseudo-next feature Figure for qualitative analysis. We observe that due to the constraints of streaming perception, there is varying misalignment between the previous feature and the current feature with respect to the next ground truth used for supervision. However, after similarity matching and reverse warping, pseudo-next feature aligns well with the ground truth.
5
Conclusions
For the first time, we propose StreamDSGN, real-time stereo- stereo-based 3D object detection framework for streaming perception. is further equipped with Feature Flow Fusion, Motion Consistency Loss. and Large Kernel BEV Backbone to enhance performance.
Limitations and Future Work. When objects are occluded or truncated, FFF may produce incorrect pseudo-next features due erroneous similarity matching. Currently, mitigate this issue by simply integrating historical features. the future, we plan to leverage neural networks to directly predict the flow of dynamic foreground objects, aiming be more robust in addressing this challenge by exploiting the adaptability of neural networks. The second limitation of our approach is that it has only been validated using stereo-based methods. In contrast, currently mainstream autonomous driving perception tasks typically employ surrounding multi-view camera systems, which differ from stereo-based methods in their construction of BEV representations. Consequently, our future research will focus validating the effectiveness of this approach within the context of multi-view methods.
6 Acknowledgments
This research was supported by the National Natural Science Foundation of China under Grant 62027804, the Fund of National Key Laboratory Multispectral Information Intelligent Processing Technology (No. 202410487201), and the Major Key Project of PCL (PCL2021A13).
10

---

References
[1] Philipp Bergmann, Tim Meinhardt, and Laura Leal-Taixe. Tracking without bells and whistles. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 941-951, 2019.
[2] Garrick Brazil, Gerard Pons-Moll, Xiaoming Liu, and Bernt Schiele. Kinematic 3d object detection monocular video. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 135-152. Springer, 2020.
[3] Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631, 2020.
[4] Ming-Fand Chang, John Lambert, Patsorn Sangkloy, Jagjeet Singh, Slawomir Bak, Andrew Hartnett, De Wang. Peter Carr, Simon Lucey, Deva Ramanan, et al. Argoverse: 3d tracking and forecasting with rich maps. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8748-877 8788-8777,20 2019. 2019.
[5] Qi Chen, Sourabh Vora, and Oscar Beijbom. Polarstream: Streaming object detection and segmentation with polar pillars. Advances in Neural Information Processing Systems, 34: 26871-26883 2021.
[6] Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia. Dsgn: Deep stereo geometry network for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12536-12545, 2020.
[7] Yilun Chen, Shijia Huang, Shu Liu, Bei Yu, and Jiaya Jia. Dsgn++: Exploiting visual-spatial relation for stereo-based 3d detectors. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4416-4429, 2022.
[8] Davi Frossard, Shun Da Suo, Sergio Casas, James Tu, and Raquel Urtasun. Strobe: Streaming object detection from lidar packets. In Conference on Robot Learning, pages 1174-1183, PMLR, 2021.
[9] Divyansh Garg. Yan Wang, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. Wasserstein distances for stereo disparity estimation. Advances in Neural Information Processing Systems, 33:22517-22529 2020.
[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pages 3361- IEEE, 2012.
[11] Anurag Ghosh, Vaibhav Balloli, Akshay Nambi, Aditya Singh, and Tanuja Ganu. Chanakya: Learning runtime decisions for adaptive real-time perception. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[12] Meng-Hao Guo, Cheng-Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. Computational Visual Media, 9(4):333-752 752, 2023.
[13] Xiaoyang Guo, Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Liga-stereo: Learning lidar geometry aware representations for stereo-based 3d detector. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3153-3163, 2021.
[14] Wei Han, Zhengdong Zhang, Benjamin Caine, Brandon Yang. Christoph Sprunk, Ouais Alsharif, Jiquan Ngiam, Vijay Vasudevan, Jonathon Shlens, and Zhifeng Chen. Streaming object detection for 3-d point clouds. In European Conference Computer Vision, pages 421-441, Springer, 2020.
[15] Jun-Yan He, Zhi-Qi Cheng, Chenyang Li, Wangmeng Xiang, Binghui Chen, Bin Luo, Yifeng Geng, and Xuansong Xie. Damo-streammet: Optimizing streaming perception autonomous driving. arXiv preprint arXiv:2303.17144, 2023.

---

[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[18] Hou-Ning Hu, Qi-Zhi Cai, Dequan Wang. Ji Lin, Min Sun, Philipp Krahenbuhl, Trevor Darrell, and Fisher Yu. Joint monocular vehicle detection and tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5390-5399, 2019.
[19] Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camer 3d object detection. arXiv preprint arXiv:2203. 17054, 2022.
[20] Eddy 1lg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0: Evolution of optical flow estimation with deep networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2462 2470, 2017.
[21] Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. Advances in neural information processing systems, 28, 2015.
[22] Rudolph Emil Kalman. A new approach linear filtering and prediction problems. 1960.
[23] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[24] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 12697-12705, 2019.
[25] Kin Wai Lau, Lai-Man Po, and Yasar Abbas Ur Rehman. Large separable kernel attention: Rethinking the large kernel attention design in cnn. Expert Systems with Applications, 236: 121352, 2024.
[26] Changcai Li, Haitao Meng, Gang Chen, and Long Chen. Real-time pseudo-lidar 3d object detection with geometric constraints. In 2022 IEEE 25th International Conference on Intelligent Transportation Systems (ITSC), pages 3298-3303 IEEE, 2022.
[27] Chengyao Li, Jason Ku, and Steven L Waslander. Confidence guided stereo 3d object detection with split depth estimation. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5776-5783, IEEE, 2020.
[28] Chenyang Li, Zhi- Qi Cheng, Jun- Yan He, Pengyu Li, Bin Luo, Hanyuan Chen, Yifeng Geng, Jin-Peng Lan, and Xuansong Xie. Longshortnet: Exploring temporal and semantic features fusion in streaming perception. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1-5. IEER, 2023.
[29] Jinyu Li, Chenxu Luo, and Xiaodong Yang. Pillarnext: Rethinking network designs for 3d object detection in lidar point clouds. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 17567-17576, 2023.
[30] Mengtian Li, Yu-Xiong Wang, and Deva Ramanan. Towards streaming perception. In Com- puter Fision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 11 16, pages 473-488. Springer, 2020.
[31] Peiliang Li, Xiaozhi Chen, and Shaojie Shen. Stereo r-cnn based 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7644-7652, 2019.
[32] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's-eye-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pages 1-18. Springer, 2022.
12

---

[33] Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[34] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: A unified framework for 3d perception from multi-camera images. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3262-3272, 2023.
[35] Yuxuan Liu, Lujia Wang, and Ming Liu. Yolostereo3d: step back to 2d for efficient stereo 3d detection. In 2021 IEEE International Conference Robotics and Automation (ICRA), pages 1 13024, IEEE, 2021.
[36] Zhe Liu, Xiaoqing Ye, Xiao Tan, Errui Ding, and Xiang Bai. Stereodistill: Pick the cream from lidar for distilling stereo-based 3d object detection. arXiv preprint arXiv:2301.016/1 2023.
[37] Xinzhu Ma, Yinmin Zhang, Dan Xu, Dongzhan Zhou, Shuai Yi, Haojie Li, and Wanli Ouyang. Delving into localization errors for monocular 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4721-4730, 2021.
[38] Haitao Meng, Changcai Li, Gang Chen, Long Chen, et al. Efficient 3d object detection based on pseudo-lidar representation. IEEE Transactions on Intelligent Vehicles, 2023.
[39] Gregory P Meyer, Jake Charland, Shreyash Pandey, Ankit Laddha, Shivam Gautam, Carlos Vallespi-Gonzalez, and Carl K Wellington. Laserflow: Efficient and probabilistic object detection and motion forecasting. IEEE Robotics and Automation Letters, 6(2):526-533, 2020.
[40] Paulius Micikevicius, Sharan Narang, Jonah Alhen, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. In International Conference on Learning Representations, 2018.
[41] Alejandro Newell, Kaiyu Yang. and Jia Deng. Stacked hourglass networks for human pose estimation. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, pages 483-499, Springer, 2016.
[42] Wanli Peng, Hao Pan, He Liu, and Yi Sun. Ida-3d: Instance-depth-awarr 3d object detection from stereo vision for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13015-13024, 2020.
[43] Xidong Peng, Xinge Zhu, Tai Wang, and Yuexin Ma. Side: center-based stereo 3d detector with structure-awarc instance depth estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 119-128, 2022.
[44] Alex D Pon, Jason Ku, Chengyao Li, and Steven L Waslander. Object-centric stereo matching for 3d object detection. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 8383-8389, IEEE, 2020.
[45] Rui Qian, Divyansh Garg. Yan Wang. Yurong You, Serge Belongie, Bharath Hariharan, Mark Campbell, Kilian Q Weinberger, and Wei-Lun Chao. End-to-end pseudo-lidat lidar for image-based 3d object detection. In Proceedings of the IEEF/CV Conference on Computer Vision and Pattern Recognition, pages 5881-5890, 2020.
[46] Cur-Eyal Sela, Ionel Gog, Justin Wong, Kumar Krishna Agrawal, Xiangxi Mo. Sukrit Kalra, Peter Schafhalter, Eric Leong, Xin Wang, Bharathan Balaji, et al. Context-aware streaming perception in dynamic environments. In European Conference on Computer Vision, pages 621-638. Springer, 2022.
[47] Leslie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 369-386. SPIE, 2019.
[48] Deqing Sun, Xiaodong Yang, Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8934-8943, 2018.
13

---

[49] Jiaming Sun, Linghao Chen, Yiming Xie. Siyu Zhang. Qinhong Jiang, Xiaowei Zhou, and Hujun Bao. Disp r-cnn: Stereo 3d object detection via shape prior guided instance disparity estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10548-10557, 2020.
[50] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 2446 2454, 2020.
[51] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pvt v2: Improved baselines with pyramid vision transformer. Computational Visual Media, 8(3):415-424, 2022.
[52] Xiaofeng Wang, Zheng Zhu, Yunpeng Zhang, Guan Huang, Yun Ye, Wenbo Xu, Ziwei Chen, and Xingang Wang. Are we ready for vision-centric driving streaming perception? the asap benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1600-9610, 2023.
[53] Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8445- 8453, 2019.
[54] Yan Wang, Bin Yang, Rui Hu, Ming I Liang, and Raquel Urtasun. Plumenet: Efficient 3d object detection from stereo images. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 3383-3390 IEEE, 2021.
[55] Wenhao Wu, Hau San Wong, and Si Wu, Semi-supervised stereo-based 3d object detection via cross-view consensus. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17471-17488 2023.
[56] Zhenbo Xu, Wei Zhang, Xiaoqing Ye, Xiao Tan, Wei Yang. Shilei Wen, Errui Ding, Ajin Meng, and Liusheng Huang. Zoomnet: Part-aware adaptive zooming neural network for 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 12557-12564, 2020.
[57] Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018
[58] Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's s-eye-view recognition via perspective supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17830-17839, 2023.
[59] Jinrong Yang, Songtao Liu, Zeming Li, Xiaoping Li, and Jian Sun. Real-time object detection for streaming perception. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5385-5399 2022.
[60] Zetong Yang. Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3d-man: 3d multi-frame attention network for object detection. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2021.
[61] Junbo Yin, Jianbing Shen, Chenye Guan, Dingfu Zhou, and Ruigang Yang. Lidar-based online 3d video object detection with graph-based message passing and spatiotemporal transformer attention. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 11495-11504, 2020.
[62] Yurong You, Yan Wang. Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. Exeudo-lidar+++ Accurate depth for 3d object detection in autonomous driving. International Conference on Learning Representations (ICLR), 2020.
[63] Zhenxun Yuan, Xiao Song, Lei Bai, Zhe Wang, and Wanli Ouyang. Temporal-channel trans- former for 3d baser- video object detection for autonomous driving. IEEE Transactions on Circuits and Systems for Video Technology, 2264)22068-2778, 2021.
14

---

A Appendix supplemental material
A.1 Real-time Optimization
Our real-time optimization strategies are outlined as follows:
(1) We replace the original ResNet34 with ResNet18 [T6] as the backbone for stereo images.
We abandon the Dual-view Stereo Volume representation in DSGN++ 7 and instead retained only one of PSV or 3DGV. See more detail in 77].
(3) Inspired by 24 29]. we extend the grid size of the feature volume from (0.2m, 0.2m, 0.2mm
to (0.2m, 0.2m, 0.4m) along the vertical axis.
(4) We perform edge cropping for detection range [37]. The new range is set to [2m,, 53.2ml for the Z-axis (depth), [-28.8m, 28.8m] for the X-axis, and [-1m, 3m] for the Y-axis respec- tively (in the camera coordinate system).
(5) Referring to [40]. we incorporate Automatic Mixed Precision (AMP) techniques during both the training and inference stages.
A.2 Effectiveness of Real-Time Optimization
Table 3: Comparison of real-time optimization strategies. The term "r18" denotes the use of ResNet18 as the image feature extractor. "Only 3DGV" and "Only PSV" represent the exclusive adoption of 3DGV and PSV stereo features, respectivcly. "Low-res. (h)" indicates low-resolution representation in the vertical dimension. "EC" signifies edge cropping, and "AMP" stands for the utilization of automatic mixed precision 40 during both training and inference stages.
Ablation methed
Lasency (mS)
ofTAP
sAP
Only 3DGV
only PSV
Easy
Hard
Easy
Med.
Non-real sime IKF Forecasting
228 (5:))
91.79
69.79
1,92
180.96
90.60
74.08
68:35
6.46
4.26
3.58
150.55
90.77
68:73
5.80
91.98
75.91
68.00
5.15
134.21
91.81
77.22
69.98
9.52
6.08
5.15
Real-time (Endd-to-end)
V
W6.08
91.67
76.52
69.34
19.02
14,00
12.79
80.71
91 22
25.05
69.
14
25.50
17.68
14.75
We compare the effectiveness of various real-time optimization strategies, experimental results are described in Table For real-time settings (<100ms), each ground truth frame has corresponding predicted frame. making it an end-to-end approach. For non-real -ime settings (>100ms), we employ Kalman filter 22 30 52) to forecast and interpolate unmatched intermediate frames. Hence, Table 3 can also be viewed as a comparison between end-to-end methods and traditional SOTA methods. The table reveals that all optimization strategies mentioned in Appendix [A. effeetivlly (fffexitt reduce model latency with minimal impact on offline accuracy, decreasing from the initial 263.33m to 80.7 1ms. Concurrently, the SAPaD gradually increases due to the reduced latency, rising from 2.53% to 23.73% in the easy level, achieving an improvement of over 20%. Therefore, we choose the configuration highlighted in green row in Table 3 as the final optimization strategy.
A.3 Comparison of the Domain Gap
Table 4: Comparison of the domain gap between our split Tracking dataset and the 3D Object De- tection dataset.
We compare the domain gap between our split tracking dataset, which consists of 4.291 train- ing samples and 3,672 validation samples, and the widely recognized KITTI 3D Object De- tection dataset [10,, comprising 3,712 training samples and 3, 7,769 validation samples. Specif- ically, we train and test PointPillar (24) and DSGN++ (7) on both datasets, and then compare the AP3D for the Car category LoU 0.7.
APao
Method
Sensor
Dataset
Easy
Mod.
Hard
PointPillars
X
LIDAR
Object Detection
87.75
78.38
75.18
Pointillaass
X4
LIDAR
our split Tracking
94.57
88.35
84.85
DSGN++
t
Stereo
Objeet Detection
83.63
66.41
61.38
DSGN-+
Stereo
our split Tracking
91.79
78.35
69.79
15

---

The experimental results are shown in the Table 4 From the table, we observe that both methods perform better on our split Tracking dataset, indicating smaller domain gap compared to the Object Detection dataset. However, given that we have 579 additional training samples and that accuracy may further decrease under streaming perception constraints, this difference is deemed reasonable.
A.4 Impact of Fusion Stage
Table 5: Ablation studies of fusion stage.
To validate the hypothesis about insufficient re-
sAPBaV
sAPsD
ceptive field of shallow detection heads, we con-
Fusion Stuge
Fasy
Mod.
Hard
Easy
Mod.
Hard
ducted comparative experiments by setting the fusion stage before the detection heads and be- fore the BEV backbone, respectively. The com- parison results are described in Table [5] The table demonstrates that fusing historical features before the BEV backbone, rather than preceding the detection head as proposed Stream YOLO [59], leads to competitive improvements. Specifically, the improvements at the three difficulty levels exceed 2%, 1%, and 1%, respectively.
before [9
73.02
58.37
$1.86
before BEV backhone
83,20
71,15
65.74
75.38
59.55
53.11
Table 6: Comparison with SOTA fusion method and ablation studies FFF The term "ra" and "d" represent the downsample ratio and maximum displacement, respectively.
Table 7: Grid search of r in Equation for MCL.
sAP
sAP3o
Pasy
Mod.
Hard
Easy
Mod.
Hard
d'
Latency
sAPaRv
sAPan
MCL
Easy
Mod.
Hard
Easy
Mod.
Hard
83.20
71.15
65.74
75.38
59.55
53.11
Daal-Flow
ty/ MCV.
83.20
65.74
75.38
53.11
0.0
$1.99
71.71
66.04
76.13
61.26
$4,40
Feature- Flas Fusion (Owrs)
0.2
83.43
71.82
66.10
76.72
61.88
55.08
7.67
85.08
71.98
66.49
75.88
61.57
54.77
71.17
64.35
61.58
2
12.19
85.50
71.68
66.13
76.82
60.73
54.02
0.6
$4.68
71.63
65.85
76.30
61.68
54.74
14.63
85.48
71.93
66.22
75.91
59.83
54.46
0.8
84.63
71.01
66.18
78.15
62.28
56.80
6.26
83.27
71.35
64.24
76.11
59.40
53.82
1.0
85.09
71.49
66.03
75.74
59.62
54.24
A.5
Latency and Hyperparameters of FFF
We report the latency and accuracy of FFF under different feature downsampling ratios and maximum search displacement in T The Table shows that with downsampling ratio of 2 and maximum displacement of 3, FFF achieves the best performance while only taking 7.67ms (highlighted in green). At this setting, 3. shows a 2.01% % improvement at the moderate level compared to the SOTA Dual-Flow [59
A.6
Value of
We get the value of hyperparameter in Equation through grid search. Experimental results shown in Table We observe that when only utilizing the velocity loss setting in MCL (r = 0), the improvements at the three difficulty levels compared to the scheme without MCL are about 1%, 1.5%, and 1%, respectively. Further, when introducing the acceleration loss, the performance reaches its maximum improvement when the balancing parameter 0.8 (highlighted in green). At this setting, the improvements at the three difficulty levels are 2.77%, 2.73%, and 3.69%, respectively.
A.7 Complexity of LKBB
Table 8: Comparison of the Complexity of BEV Backbones.
We present in Table g comparison between LKBB and the original hour- glass structure 41 in terms of param- eter count, computational complexity, and latency. Our LKBB has simi- lar parameter count to the original structure, yet reduces computational complexity by nearly GFLOPs. At this juncture, the overhead incurred is merely additional approximate 3ms, while achieving competitively enhanced accuracy.
Method
Params
(M)
PLOPs
Laiency (ms)
Easy
APPay
Mod.
Hard
Easy
AAP
Mod.
Hard
Houegias (ocigial)
3,66
71.15
53.11
LXBR (nund)
195
673
84.97
60.11
55.00
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We delineate our motivations, claims, and contributions in the abstract and introduction sections. Please see Section I
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section for more detailed discussion limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
17

---

Justification:
Our work does not encompass theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
We have provided detailed descriptions of the experimental setup for repro- ducibility, including dataset configurations and model hyperparameters. For further details, please refer to Section
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: Our code is temporarily hosted on an anonymous platform: https:// anonymous , 4open . science/r/eeeeeeeecce http://sstteeeeDSGG.
If the paper accepted, we will
release the source code on GitHub.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//ipps.cc/pulli//uiiiee
wwsssssssiniyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Please refer to Section 4 for more detailed discussion of the experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Our experiments do not include error bars confidence intervals, but they support the main claims of the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
19

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We introduce the computational resources in the experimental setup. Please refer to Section for details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics Ethics/https:
Answer: [Yes]
Justification:
Our research adheres the ethical guidelines of NeurIPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work has societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
20

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not entail such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Our code is based on several released works. All have been cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
21

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have released an anonymous version of the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
22

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "3pFRWv1STy", "reviewer": "Reviewer_uBXC", "summary": "The paper presents stereo-based 3D object detection method designed for streaming perception, where the current frame (and past frames) are taken to predict the object bounding boxes in the next frame. The authors adopt a simplified DSGN++ as the backbone and add several components to enhance the perception accuracy. First, the authors proposed to estimate the flow in feature space from t-1 to t and use it to warp the features in the future frame at t+1. Besides, a motion consistency loss is added to refine the future trajectory. Last, the authors propose a large kernel backbone to process the BEV feature. The proposed method could reach a latency of 90ms and outperform a baseline (using Kalman Filter) on the KITTI dataset.", "questions": "1. Though the proposed method focuses on stereo-based 3D object detection, the new components proposed in this paper are not related to the stereo setting and seem to work generally for all BEV-based perception systems. Why not test it on more general settings and use more datasets (e.g., use the six cameras in nuScene)? \n\n2. For the Feature-Flow Fusion, is there any reason why the authors apply warping in feature space instead of using optical flow in the pixel space? \n\n3. For motion consistency loss, it makes sense to constrain the velocity and acceleration for the new prediction. But given an estimated bounding box, how do you know which object it is to retrieve its past trajectory? What if the object is only contained in the current frame (not appear in the past frames)? And what if the network predicts a wrong object (false positive) during the early training stage? How to calculate the loss then? It would be better to provide more explanations.\n\n4. In section 3.3, does $G_t^{pose}$ mean the same thing as $G_t^{box}$? \n\n5. The proposed method is compared with a baseline using Kalman Filter. But It seems like Kalman Filter is very easy to beat. What if using more advanced trajectory prediction methods as the baseline? Will the proposed method still show a large performance gain?", "limitations": "The authors have adequately addressed the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-motivated and aims to solve an important problem in the real world applications\n\n2. The proposed method yields an appealing performance in streaming perception in the KITTI dataset.\n\n3. The way of generating feature flow is interesting and could be used in other automonous driving applications", "weaknesses": "Though the proposed method shows a good performance, some details about the proposed new components are missing. Besides, the proposed method is only tested in one dataset and compared with one baseline, which may not be sufficient. Please refer to Questions for more details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer uBXC\n\n### Question\n---\n**Q1**: Though the proposed method focuses on stereo-based 3D object detection, the new components proposed in this paper are not related to the stereo setting and seem to work generally for all BEV-based perception systems. Why not test it on more general settings and use more datasets (e.g., use the six cameras in nuScene)?\n\n**A1**: In theory, our method is indeed applicable to other BEV-based approaches. However, when deployed to other multi-view BEV-based methods, the following challenges may arise:\n- Streaming perception evaluation requires high-frame-rate annotations to obtain accurate results, but these datasets often have low frame rates. e.g., the nuScenes dataset has a 12Hz image frame rate but only a 2Hz annotation frequency.\n\n- In multi-view BEV-based 3D detection, LSS series methods (e.g., BEVDet, BEVDepth, BEVStereo) have the limitation of generating discrete and sparse BEV representations (Ref from FB-BEV[1]). These sparse BEVs may lead to numerous redundant warping operations in FFF and may result in potential distortions of warped objects. please compare the Figure 2 in [1] with our Figure 8 to observe the BEV feature difference.\n\n- For methods in the BEVFormer series, such approaches typically have higher latencies (over 400ms). A large latency interval requires a larger spatial search for the FFF, and the motion consistency assumed by the MCL may no longer hold.\n\nThese challenges will be included in our discussion of limitations. We will also explore applications related to multi-view methods in the future.\n\n---\n**Q2**: For the Feature-Flow Fusion, is there any reason why the authors apply warping in feature space instead of using optical flow in the pixel space?\n\n**A2**: Here are the reasons for this choice:\n- The dataset lacks synchronized optical flow ground truth.\n- Even if optical flow ground truth were available, using an optical flow estimation model would introduce additional time overhead. In contrast, computing flow at an intermediate level does not require an extra feature extraction process.\n- Intermediate-level features include image features (img-coord) and BEV features (world-coord). Performing warping on image features inevitably lead to misalignment between the image features and the depth map ground truth. In contrast, warping on BEV features does not affect depth regression and better aligns with object movement in the real world.\n\n---\n**Q3**: For motion consistency loss, it makes sense to constrain the velocity and acceleration for the new prediction. But given an estimated bounding box, how do you know which object it is to retrieve its past trajectory? What if the object is only contained in the current frame (not appear in the past frames)? And what if the network predicts a wrong object (false positive) during the early training stage? How to calculate the loss then? It would be better to provide more explanations.\n\n**A3**: Our manuscript already explains the question about how to retrieve an object's past trajectory in Section 3.3. The initial step in calculating the MCL involves establishing correspondences between bounding boxes across different time steps. We establish the correspondence between $P_{t+1}$ and $G_{t}$ using an IoU matrix (same as streamYOLO), and establish the correspondence between $G_{t-2}, G_{t-1}, G_{t}$ using object IDs.\n\nAs for objects that appear in only a single frame, since there is no continuous trajectory, we do not calculate their MCL either.\n\nRegarding false positives, they occur when the classification head gives high scores to negative anchors, unrelated to the regression head. Our MCL is a regression supervision method applied only to positive anchors (as in PointPillar, SECOND, etc.). Please note that, whether an anchor is positive or negative is determined by the IoU between the preset anchor and the ground truth bounding box, not by the classification score.\n\n---\n**Q4**: In section 3.3, does $G_{t}^{pose}$ mean the same thing as $G_{t}^{box}$ ?\n\n**A4**: $G_{t}^{box} = \\\\{ x,y,z,l,w,h,\\theta \\\\}$, and $G_{t}^{pose} = \\\\{ x,y,z,\\theta \\\\}$, where $\\\\{x,y,z\\\\}$ denotes the object's spatial position, $\\\\{l,w,h \\\\}$ denotes the dimension and $\\theta$ denotes the heading angle. Since the dimensions of the object remain constant, our MCL only needs to compute changes in position and orientation.\n\n---\n**Q5**: The proposed method is compared with a baseline using Kalman Filter. But It seems like Kalman Filter is very easy to beat. What if using more advanced trajectory prediction methods as the baseline? Will the proposed method still show a large performance gain?\n\n**A5**: Previous methods to Streaming Perception can be divided as (refer from Ref [52]): (a) Velocity-based updating (non-end-to-end), where the Kalman filter is utilized to associate and multi-frame detection results, and the future state is predicted by the constant velocity motion model, e.g., The work in Ref [30] proposed a meta-detector named Streamer that can be combined with any object detector. (b) Learning-based forecasting (end-to-end), where the future state is directly estimated by the 3D-future detector, e.g., StreamYOLO from Ref [59].\n\nStreamYOLO has shown the advantages of end-to-end methods. Therefore, even when combined with more advanced trajectory prediction methods, non-end-to-end methods may struggle to achieve our level of accuracy. We show comparisons between both types of methods in our manuscript: the comparison with non-end-to-end methods is detailed in Table 1, while the comparison with end-to-end methods is shown in Table 2, settings c and g. We plan to consolidate both comparisons into a single table to avoid confusion.\n\n---\n### Reference\n[1] Li Z, Yu Z, Wang W, et al. Fb-bev: Bev representation from forward-backward view transformations[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 6919-6928.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ecfdsbT13i", "reviewer": "Reviewer_NjQK", "summary": "The work proposes StreamDSGN, a framework for steaming perception, evaluated on the KITTI dataset with sAP (steaming average performance metric), which takes into account the model latency and is a better fit for evaluating streaming perception than purely qulitative metrics. It is the first work to do 3D object detection in streaming perception setting. The DSGN++ base detector is used and performance is improved. The paper introduce three methods to improve the results: (1) Using past frames' feature positions to predict the next frames features, (2) additional supervision, (3) a larger kernel for bigger receptive field.", "questions": "See weaknesses 3 and expecially 4.", "limitations": "Limitations are discussed.", "rating": 7, "confidence": 2, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "1. The ablation study is exhaustive.\n\n2. Figures are well done and generally support the understanding (Figure 2 is adressed separately).\n\n3. The method achieves a better result than the baseline.", "weaknesses": "1. Limitations are discussed only really shortly\n\n2. Figure 2 needs a little more explanation. There is more than one GT-trajectory, are they from differnt frames? Prediction and GT are for the next frame, t+1?\n\n3. Unclear sentence in the introduction: \"It is observed that for moving objects, the ground truth of the next frame (depicted by the red bounding box) consistently precedes their current position.\" What does it mean for the ground truth to \"precede the current position\"?\n\n4. The largest concern: The test setup, with interleaved training / testing frames does not really provide a separate testing set. All scenes are split into sequences of 4 seconds, which are then used alternatingly for test and training. In urban driving scenarios, cars won't move that much in 4 seconds. This leads to a significantly smaller domain gap between test and training than when, for example, training on completely different sequences. It is unclear if this split is a common protocol for KITTI or though off for this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer NjQK\n\n### Weaknesses\n---\n**W1**: Limitations are discussed only really shortly.\n\n**A1**: Our discussions on limitations and future work are brief due to the page limit. We may add more detailed discussion in the appendix.\n\n---\n\n**W2**: Figure 2 needs a little more explanation. There is more than one GT-trajectory, are they from different frames? Prediction and GT are for the next frame, t+1?\n\n**A2**: Yes. The ground-truth trajectory (denoted by red dashed arrows) refers to different timesteps/frames, e.g., the two arrows denote the trajectory from $t-1$ to $t$ and from $t$ to $t+1$, respectively, while the predicted trajectory (denoted by the green solid arrow) is from $t$ to $t+1$.\n\n---\n**W3**: Unclear sentence in the introduction: \"It is observed that for moving objects, the ground truth of the next frame (depicted by the red bounding box) consistently precedes their current position.\" What does it mean for the ground truth to \"precede the current position\"?\n\n**A3**: The sentence is indeed unclear, and we plan to rewrite it as follows: “For a moving object with (approximately) constant velocity, its ground truth position in the next frame at time $t + 1$ (depicted by the red bounding box) is likely to be different from its position in the current frame at time $t$, and its movement is predictable based on its recent history of positions in frames $t − 1$ and $t$.”\n\n---\n**W4**: The largest concern: The test setup, with interleaved training / testing frames does not really provide a separate testing set. All scenes are split into sequences of 4 seconds, which are then used alternatingly for test and training. In urban driving scenarios, cars won't move that much in 4 seconds. This leads to a significantly smaller domain gap between test and training than when, for example, training on completely different sequences. It is unclear if this split is a common protocol for KITTI or though off for this paper.\n\n**A4**: Thank you for this good question; it will make our experiments more reasonable. We added an experiment to compare the domain gap between our split tracking dataset (train:val = 4291:3672) and the widely recognized KITTI Object Detection dataset (train:val = 3712:3769). Specifically, we trained and tested PointPillar and DSGN++ on both datasets and compared the $\\rm AP_{3D}$ for the Car category at IoU = 0.7. The experimental results are shown in the table below (we will add this experiment to our manuscript):\n\n| Method | Sensor | Dataset | Easy | Moderate | Hard |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| PointPillar | LiDAR | Object Detection | 87.75 | 78.38 | 75.18 |\n| PointPillar | LiDAR | our split Tracking | 94.57 | 88.35 | 84.85 |\n| DSGN++ | Stereo | Object Detection | 83.63 | 66.41 | 61.38 |\n| DSGN++ | Stereo | our split Tracking | 91.79 | 78.35 | 69.79 |\n\nFrom the table, we can observe that both methods perform better on our split tracking dataset, which indeed proves that our domain gap is smaller compared to that of the Object Detection dataset. However, considering that we have 579 additional training samples and that accuracy may further decrease under streaming perception constraints, this difference is considered reasonable.\n\nFurthermore, please note that, as this is the first work in this area, our focus is not on showing how high the accuracy is, but on demonstrating the effectiveness of our method. All our experiments were conducted on the same data split. In the future, we plan to conduct further validation on larger-scale datasets with higher frame rates and greater domain gaps.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KUDbb8g8Dt", "reviewer": "Reviewer_Ytfb", "summary": "In this paper, the authors propose a real-time stereo-based 3D object detection framework for streaming perception. Specifically, the authors design feature flow fusion, motion consistency loss and a Large Kernel BEV backbone to improve the performance. The authors validate the effectiveness of the proposed method and each module by conducting experiments on KITTI dataset. However, the author only compare their method with one baseline which they build their method on.", "questions": "There are some grammar errors in the manuscript.", "limitations": "The limitations are addreess in the paper, including incoreect feature fusion caused by occlusion and truncation.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. As claimed in the paper, this is the firstwork designed for 3D object detection streaming perception, which is a good setting that better aligns with real applications.\n\n2. The authors analyze the challenges in streaming perception, and add new modules into existing framework to improve the performance\n\n3. Experiments on the Kitti dataset validate the effectiveness of the three modules designed in this paper. \n\nThe paper is well structured and the visualization is good.", "weaknesses": "The novelty of the proposed method is not significant. The challenges in streaming perception are evident, and the technical novetly  of the proposed solution is not very significant to me. According to Table2, the biggest performence improvement comes from the fusion of (t-1). The three modules do not result in sigificant performance improvement.\n\nThe intuition of MCL needs more elaboration. Why do we need velocity loss and acc loss? The supervision of position already encodes velocity and acc.\n\nThe latency of the network is strongly dependent on the hardware. Also, the latency caused by camera exposure and data transportation are not considered. A comprarison under different latency can be added to better illustrate the usefulness of the method under different scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer Ytfb\n\n### Weaknesses\n---\n**W1**: The novelty of the proposed method is not significant. The challenges in streaming perception are evident, and the technical novetly of the proposed solution is not very significant to me. According to Table2, the biggest performence improvement comes from the fusion of (t-1). The three modules do not result in sigificant performance improvement.\n\n**A1**: Previous works proposed the concept of streaming perception, to the best of our knowledge,  we first apply the streaming perception for 3D object detection. Note that trying to predict future states using information from a single moment is an ill-posed problem. Therefore, fusing historical information is the basic idea behind streaming perception, and the differences lie in the details on how the prediction is done. In this work, this practice (fusion of $t-1$) builds the basic framework of our 3D object detection algorithm, in other words, it is the baseline of our method. So, it is not strange that it contributes to the biggest performance improvement. The three modules in this manuscript are our further attempts to enhance perception accuracy based on the baseline. Their contributions are indeed not as significant as fusing the historical information, but the combination of the baseline and these modules makes our method achieve the SOTA performance of 3D streaming perception. \n\n---\n\n**W2**: The intuition of MCL needs more elaboration. Why do we need velocity loss and acc loss? The supervision of position already encodes velocity and acc.\n\n**A2**: The basic premise of streaming perception is that the motion trajectory is smooth without abrupt changes in velocity and acceleration, thus we can use the recent past to predict the near future. If this assumption does not hold, fusing historical information is beneficial for predicting the future. Therefore, we introduce MCL to provide additional supervision of constant velocity and acceleration motion.\n\nRegarding the comment that \"The supervision of position already encodes velocity and acc\", please note that without MCL, the model is supervised for bounding box regression solely by a single $G_{t+1}$, which does not include velocity and acceleration information. By incorporating MCL, the model includes both the basic bounding box regression loss and motion constraints of the nearest historical intervals. This technique can improve convergence during training and reduce localization errors.\n\n---\n**W3**: The latency of the network is strongly dependent on the hardware. Also, the latency caused by camera exposure and data transportation are not considered. A comprarison under different latency can be added to better illustrate the usefulness of the method under different scenarios.\n\n**A3**: Our algorithm StreamDSGN has inference latency of 91.45 ms on the NVIDIA TITAN RTX GPU (16.3 TFLOPS). (We also tested StreamDSGN on the RTX 3090 GPU (35.6 TFLOPS), and the latency was only around 60ms.)\n\nTo quantify the impact of latency, we conducted additional experiments by adding artificial extra delays which can be seen as the latency caused by the camera exposure and data transportation. Specifically, we add random delays (Gaussian distributions with different means and the same variance) to the inference latency of each frame (Average about 91.45ms). The experimental results are shown in the table below\n\n| Random noise| Easy  |  Moderate  |  Hard  |\n|:-----:|:-----:|:-----:|:-----:|\n| 0  | 77.47 | 63.76 | 57.42 |\n| $X\\sim \\mathcal{N}(0, 2)$  | 75.86 | 62.35 | 57.17 |\n| $X\\sim \\mathcal{N}(5, 2)$  | 67.97 | 57.25 | 52.23 |\n| $X\\sim \\mathcal{N}(8, 2)$  | 38.79 | 30.12 | 26.82 |\n| $X\\sim \\mathcal{N}(10, 2)$ | 28.05 | 21.05 | 17.46 |\n| $X\\sim \\mathcal{N}(20, 2)$ | 24.73 | 18.72 | 15.89 |\n\nWe report the $\\rm{sAP_{3D}}$ for the Car category at IoU=0.7. We can see that when extra delay noise is initially introduced, the model's performance begins to slightly decrease. This is because the inference delay of a small number of samples has already exceeded the inter-frame interval of 100 ms. when the average additional delay is 8 ms, the performance degradation is particularly severe compared to the average additional delay of 5ms. This happens because the average latency reaches 91.45 + 8 = 99.45 ms, causing nearly half of the samples to have inference times that exceed the inter-frame interval. Consequently, the model can no longer respond promptly to each frame. In other words, in practice, ensuring the data and inference latency less than the inter-frame interval is important. Also, these results underscore the crucial role of real-time computation performance in streaming perception tasks, through using better hardware or more optimized algorithms.\n\n---\n\n### Questions\n---\n**Q1**: There are some grammar errors in the manuscript.\n\n**A1**: Thank you for bringing them to our attention. We have rechecked the grammatical issues.\n\n---", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YMixFXuuOJ", "reviewer": "Reviewer_D5z5", "summary": "This paper proposed a real-time stereo 3D object detection algorithm under the streaming preception framework setting. The proposed StreamDSGN build on the existing streaming perception work and the DSGN 3D object detection work and has three technical contributions: 1) the Feature-Flow-Fusion module predicts future-frame features using flow map which reduce the misalignment between supervision and current observation in the streaming preception setting, 2) Motion Consistency Loss function for explicit supervision based on motion consistency between adjacent frames, 3) Large Kernel BEV Backbone for capturing long-range dependencies in low framerate 3D object detection dataset. Experiments on the KITTI Tracking dataset show that the proposed StreamDSGN method achieves impressive result on KITTI Tracking dataset.", "questions": "1. How about the algrithm's (GPU) memory consumption?", "limitations": "The limitation of the proposed algorithm has been addressed in Section 5.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed algorithm of 3D object detection for streaming perception is interesting, the performance evaluation setting for considering both accuracy and latency is reasonable for practical applications such autonomous driving\n2. Experimental result is impressive and convincing: the end-to-end framework shows significant better performance than a straightforward combination of Streamer and DSGN++_{opt}, and the proposed FFF, MCL and LKBB techniques together achieve 4.33% increase in 3D object detection streaming average precision compared with the end-to-end baseline. The source code is provided and the experiment result is reproducible.\n3. The presenation is clear and easy to follow\n4. The ablation study is clear and the baseline setting is reasonable", "weaknesses": "1. The setting of 3D detection from stereo is not very common in autonomous driving and the relevant dataset is limited. It would be good to extend the stereo camera setting to multi-camera setting or camera+Lidar setting and test the generalization capability of the proposed algorithm on extended settings.\n2. There lacks detailed computational latency analysis of each algrorithm module as shown in Figure.3,  the overall latency of 91.45ms may not be enough to check the trade-off between performance & latency for each algorithm module", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Response to reviewer D5z5\n\n### Weaknesses\n---\n**W1**: The setting of 3D detection from stereo is not very common in autonomous driving and the relevant dataset is limited. It would be good to extend the stereo camera setting to multi-camera setting or camera+Lidar setting and test the generalization capability of the proposed algorithm on extended settings.\n\n**A1**: In theory, our method is indeed applicable to other BEV-based approaches. However, when deployed to other multi-view BEV-based methods, the following challenges may arise:\n\n- Streaming perception evaluation requires high-frame-rate annotations to obtain accurate results, but these datasets often have low frame rates. e.g., the nuScenes dataset has a 12Hz image frame rate but only a 2Hz annotation frequency.\n\n- In multi-view Query-based methods, these approaches (e.g., DETR3D, PETR) typically do not generate explicit BEV features. \n\n- In multi-view BEV-based 3D detection, LSS series methods (e.g., BEVDet, BEVDepth, BEVStereo) have the limitation of generating discrete and sparse BEV representations (Ref from FB-BEV[1]). These sparse BEVs may lead to numerous redundant warping operations in FFF and may result in potential distortions of warped objects. please compare the Figure 2 in [1] with our Figure 8 to observe the BEV feature difference.\n\n- For methods in the BEVFormer series, such approaches typically have higher latencies (over 400ms). A large latency interval requires a larger spatial search for the FFF, and the motion consistency assumed by the MCL may no longer hold.\n\n- Multimodal (camera + LiDAR) methods also have relatively large time consumption due to their dual-branch structure, which is not conducive to achieving an end-to-end streaming perception solution.\n\nThese challenges will be included in our discussion of limitations. We will also explore applications related to multi-view methods in the future.\n\n---\n**W2**: There lacks detailed computational latency analysis of each algorithm module as shown in Figure.3, the overall latency of 91.45ms may not be enough to check the trade-off between performance & latency for each algorithm module\n\n**A2**: For streaming perception, when the inference speed of the detector exceeds the frame rate, there is no need to check the trade-off between performance and latency. This viewpoint is supported in StreamYOLO (refer from Ref [59]): \"With these 'fast enough' detectors, there is no space for accuracy and latency trade-off on streaming perception as the current frame results from the detector are always matched and evaluated by the next frame.\"  The latency of our StreamDSGN meets this requirement.\n\nHowever, when facing higher frame rates, further acceleration optimization of the model is required to ensure that the detector can still predict the state of the next frame end-to-end. Therefore, we report the latency of each component to facilitate the next optimization efforts. The results are shown in the table below.\n\n| Module | Image Feature Extractor  |  BEV Downsampling |  Feature Flow Fusion  | Large Kernel BEV Backbone | Detection Head | Post processing | Total |\n|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|:-----:|\n| Latency | 73.62  | 0.02   | 7.71   | 5.06  |  1.52  | 4.10  | 92.03\n\nWe can see that the Image Feature Extractor has the highest latency at 73.62 ms, making it the primary source of delay. For faster real-time optimization, we might consider replacing this component with alternatives like MobileNet.\n\n---\n\n### Questions\n---\n**Q1**: How about the algorithm's (GPU) memory consumption?\n\n**A1**: When the batch size is 1, the memory consumption during the training phase is 11178M, while during the inference phase, it is only 2860M.\n\n---\n\n### Reference\n[1] Li Z, Yu Z, Wang W, et al. Fb-bev: Bev representation from forward-backward view transformations[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 6919-6928.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# Summary

---
### Multi-View Setup Limitations
Some reviewers have questioned why this work did not conduct experiments on multi-view setups. One reason is the lack of high-frame-rate datasets. Additionally, existing multi-view methods typically have high latency or sparse BEV features, making it challenging to directly apply our components to these methods. This will be a direction for our future exploration.

---
### Domain Gap Concern
Some reviewers have raised concern about domain gap in the dataset. We have supplemented our experiments by comparing our results with those from well-recognized datasets. The results show that our domain gap are indeed smaller. However, considering that our dataset has more training samples and that the detection accuracy further decreases under streaming evaluation constraints, our domain gap is therefore acceptable.

---
### Comparison on Advanced Trajectory Tracking Algorithms
Some reviewers have wondered what would happen if more advanced trajectory tracking algorithms were used for comparison. We compared with the Kalman filter because it is a benchmark method (non-end-to-end). StreamYOLO has already demonstrated the advantages of directly predicting future states in an end-to-end manner. Therefore, even when combined with more advanced trajectory prediction methods, non-end-to-end methods may struggle to achieve our level of accuracy. Also, our manuscript includes comparison with SOTA end-to-end methods (Table 2, settings c and g).

---
### Novelty and Impact of t-1 Fusion
Some reviewers have raised concern about the novelty of our work and noted that the main accuracy improvement comes from the fusion of $t-1$. To the best of our knowledge, we are the first to apply streaming perception to 3D object detection. The fusion of $t-1$ is a basic aspect of our framework, it is not strange that it contributes to the biggest performance improvement. The three modules in this manuscript are our further attempts to enhance perception accuracy based on the baseline.

---
### Latency of Each Module and Hardware Impact
Some reviewers have raised questions about the latency of each module and the impact of hardware latency. We have added experiments to address these issues. For the latency of each module, our experiments show that the image feature extractor has the highest latency at 73.62 ms. Further optimization efforts can focus on this aspect. Regarding the impact of hardware latency, when the combined hardware and inference latency is less than the inter-frame interval, the accuracy only slightly decreases; however, when the total latency reaches or exceeds the inter-frame interval, the accuracy drops significantly. This further underscores the importance of real-time performance in streaming perception tasks.

Meta-review: All reviewers recommended acceptance, though 3 are on the borderlines. This AC sees no reason to override the collective recommendations of the reviewers. The main contribution of this paper, 3D object detection for streaming perception, and the good results, make it a worthy paper.
Decision: Accept (poster)