Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection
Authors: Jinhyung Park
Chenfeng Xu
Shijia Yang
Kurt Keutzer
Kris M. Kitani
Masayoshi Tomizuka
Wei Zhan
Venue: ICLR 2023 notable top 5% (2023)
Keywords: Computer Vision
3D Object Detection
Stereo Matching

Abstract: While recent camera-only 3D detection methods leverage multiple timesteps, the limited history they use significantly hampers the extent to which temporal fusion can improve object perception. Observing that existing works' fusion of multi-frame images are instances of temporal stereo matching, we find that performance is hindered by the interplay between 1) the low granularity of matching resolution and 2) the sub-optimal multi-view setup produced by limited history usage. Our theoretical and empirical analysis demonstrates that the optimal temporal difference between views varies significantly for different pixels and depths, making it necessary to fuse many timesteps over long-term history. Building on our investigation, we propose to generate a cost volume from a long history of image observations, compensating for the coarse but efficient matching resolution with a more optimal multi-view matching setup. Further, we augment the per-frame monocular depth predictions used for long-term, coarse matching with short-term, fine-grained matching and find that long and short term temporal fusion are highly complementary. While maintaining high efficiency, our framework sets new state-of-the-art on nuScenes, achieving first place on the test set and outperforming previous best art by 5.2% mAP and 3.7% NDS on the validation set. Code will be released here: https://github.com/Divadi/SOLOFusion.

Published as a conference paper at ICLR 2023
TIME WILL TELL: NEW OUTLOOKS AND A BASELINE FOR TEMPORAL MULTI-VIEW 3D OBJECT DETECTION
Jinhyung Park Chenfeng Xu?*
Shijia Yang²
Kurt Keutzer2
Kris Kitani!
Masayoshi Tomizuka² Wei Zhan?
'Cameeie Mellon University University of California, Berkeley
{jinhyunl, kkitani |es cmu. edu (xuchenfeng, shi shijiayang, keutzer, tomizuka, wzhan)@berkeley,.ed
ABSTRACT
While recent camera-only 3D detection methods leverage multiple timesteps, the limited history they use significantly hampers the extent to which temporal fusion can improve object perception. Observing that existing works' fusion of multi- frame images are instances of temporal stereo matching, we find that performance is hindered by the interplay between 1) the low granularity of matching resolution and 2) the sub-optimal multi-view setup produced by limited history usage. Our theoretical and empirical analysis demonstrates that the optimal temporal differ- ence between views varies significantly for different pixels and depths, making it necessary to fuse many timesteps over long-term history. Building on our in- vestigation, we propose generate cost volume from long history of image observations, compensating for the coarse but efficient matching resolution with a more optimal multi-view matching setup. Further, we augment the per-frame monocular depth predictions used for long-term, coarse matching with short-term, fine- gree-griied matching and find that long and short term temporal fusion are highly complementary. While maintaining high efficiency, our framework sets new state- of-the-art art on nuScenes, achieving first place on the test set and outperforming pre- vious best art by 5.2% mAP and 3.7% NDS on the validation set. Code will be released here:
INTRODUCTION
Recent advances in calera-only 3D detection have alleviated monocular ambiguities by leverag- ing short history. Despite their improvements, these outdoor works neglect the majority of past observations, limiting their temporal fusion to few frames in a short 2-3 second window. These long-term past observations are critical for better depth estimation, which has been demonstrated through oracle experiments (Wang et al. 202 lb Jing et al. (022)) the main bottleneck of camera- only pipelines due to their lack of explicit depth measurements.
Although existing methods aggregate temporal features differently, in essence, these works all con- sider regions in 3D space and consider image features corresponding to these hypothesis locations from multiple timesteps. Then, they use this temporal information to determine the occupancy of or the existence of an object at those regions. As such, these works are instances of temporal stereo matching. To quantify the quality of multi-view (temporal) depth estimation possible in these methods, we define localization potential at a 3D location as the magnitude of the change in the source-view projection induced by change in depth the reference view. As shown in Figure [ larger localization potential causes depth hypotheses (aao et al., 2018) for reference view pixel to be projected further apart, giving them more distinct source view features. Then, the correct depth hypothesis with a stronger match with the source view feature can more easily suppress incorrect depth hypotheses with clearly unrelated features, allowing for more accurate depth estimation.
We evaluate the localization potential in driving scenarios and find that only using few recent frames heavily limits the localization potential, and thus the depth estimation potential, of exist- ing methods. Distinct from the intuition in both indoor works, which select frames with above minimum translation and rotation (Hou et al. 2019; Sun et al. (022)) and outdoor works, which often empirically select single historical frame (Huang & Huang 2022; Wang et al., 2022c Liu et al. 2022b), we find that the optimal rotation and temporal difference between the reference and
"Equal contribution.

---

Published as a conference paper at ICLR 2023
Two depth hypotheses velicle seen the current time stop,
Larger Localisation Potential: With longer history, the depth hypotheses are projected spart making depth estimation easier.
Lower Locallzation Potential: depth hypotheses projected close together, making multi-view stareo matching difficult.
Source
Roference View
1=T-16
1=7-1 1=T
16 source view are further apart, making multi-view depth estimation casier when compared to the t = T source view.
Figure 1: The depth hypothesis projections onto the 1. = T
source frame varies significantly over different pixels, depths, cameras, and ego-motion. Hence, it is necessary to utilize many timesteps over long history for each pixel and depth have access to setup that maximizes its localization potential. Further, we find that localization potential is not only decreased by fewer timesteps but is also hurt by the lower image feature resolution used in existing methods. Both factors significantly hinder the benefits of temporal fusion in prior works.
We verify our theoretical analysis by designing model that naturally follows from our findings. Although existing methods' usage of low-resolution image feature maps for multi-view stereo limits matching quality, our proposed long-term temporal fusion's 's dramatic increase in localization poten- tial can offset this limitation. Our model adopts the coarse but efficient low-resolution feature maps and leverages a 16-frame BEV cost volume. We find that such a framework already outperforms prior-arts, highlighting the significant gap in utilizing temporal information in existing literature. We extend our model by further exploiting short-term temporal fusion with an efficient sampling module, replacing monocular depth priors in the 16-frame BEV cost volume with two-view depth prior. This time offsetting the temporal decrease in localization potential with an increase in fea- ture map resolution, we observe a further boost in performance, demonstrating that short-term and forg-term temporal fusion are highly complementary. Our main contributions are as follows:
We define localization potential to measure the ease of multi-view depth estimation and use it to theoretically and empirically demonstrate that the optimal rotation and temporal difference between reference and source cameras for multi-view stereo varies significantly over pixels and depths. This runs contrary to intuition in existing works that impose minimum view change threshold or empirically search for a single past frame to fuse.
We verify our theoretical analysis by designing model, SOLOFusion, that leverages both ShOrt-term, high-resolution and LOng- -erm,t low-resolution temporal stereo for depth estimation. Critically, we the first, to the best of our knowledge, to balance the impacts of spatial resolution and temporal difference on localization potential and use it to design efficient but strong temporal nulti-view 3D detector in the autonomous driving task.
Our framework significantly outperforms state-of-the-ard methods in utilizing temporal in- formation, demonstrating considerable improvement in mAP and mATE over strong non- temporal baseline as shown in Figure 2 SOLOFusion achieves first on the nuScenes test set and outperforms previous best art by 5.2% mAP and 3.7% NDS on the validation set.
2 RELATED WORK
2.1 SINGLE- VIEW CAMERA-ONLY 3D OBJECT DETECTION
Many single-view, methods use mature 2D CNNs and predict 3D boxes from the image (Mousavian
et al. 2017] Brazil & Liu 2019; Qin et al. 2019 Xu & Chen 2018 Zhou et al., 2019). Some works
leverage CAD models (Liu et al. 202 I| Manhardt et al.] 2019 Barabanau et al. 20200) while others
set prediction targets keypoints (Li et al. 2022e Zhang et al.
2021) or disentangled 3D parame-
ters (Simonelli et al. 2019 Wang et al. 2021a) Another line of work predicts in 3D, using monoc- ular depth prediction networks (Fu et al. [2018] Godard et al.] 2017) to generate LiDAR (Wang et al. 2019; Weng & Kitani 2019) and applying iiDAR-basee 3D detection frameworks. Our paper addresses monocular 3D ambiguity through temporal fusion and is perpendicular.
2

---

Published as a conference paper at ICLR 2023
Improvement From Using Past Frames
mAP
mATE
SoLorusion R58
SOLDPusion R50
MAP improven improvement ment,
MATE Improvemen
5TS
#TSFomeeA100)
Polarfomer R101
sEviuteD Swint PETRig R101
D Samnd UVT R50
BEviomer R101
PrterFomer R101
B BEVStefeo R50
SEVDeMRD Swis-
DEVDe:db SwnT
UVTR #1 PaarDETR RIOI
UVTR 850
Aoco R50
BEVSte
WTT
SEVoupth R50
2022-3
Date of Publication
2022-9
2522-10
20223
20224
Date of Publication
2022-1
2022-9
2012-10
Figure We visualize the improvement of temporal models over their non-temporal baselines, using results from publications and released code. With short-term and long-term temporal fusion, our method best leverages temporal information, demonstrating 9.4% and 12.9% improvements in precision-recall (mAP) and localization quality (mATE), respectively.
2.2 MUL.TI- VIEW CAMERA-ONLY 3D OBJECT DETECTION
Most multi-camera methods operate in 3D space and are divided into grid and query based works. Following LSS (Philion & Fidler 2020), some grid briddbased methods predict a distribution over depth bins and generate point cloud with probability-weighte
image features be used for BEV detec- Reading et al. 2021: Muang et al. 2021 Li et al. 2022a). Followup works (Liu et al. 2022c Li
al., 2002cc) speed up voxelization and introduce depth supervision (Li al. (2022c). While these
methods generate point clouds from images and voxelize, other grid-based works instead project
voxels onto images and sample features (Wang et al.) 2022b Roddick et al., 2018 Rukhovich
et al. 2022). Another branch of works follow DETR3D Wang et al. (2022d) and adopt queries. These works use object queries (Wang al. 2022d Liu
2022a] Chen et al. 2022] Jiang al.]
2022 or BEV grid queries (Li et al. 2022d and project them to get image features.
Recent works extend grid and query frameworks to process several frames. Grid methods align and concatenate volumes from multiple timesteps (Huang & Huang 2022 Li et al.
2022c or
sample voxel features from past images (Wang et al. 2022b a Rukhovich et al. (022). Query-baser methods sample from past images through projection (Chen et al. 2022 Jiang al. 2022 Qin ct al.
2022] or attention (Liu et al., 2022b Li et al., 2022d). However, these methods demonstrate limited improvement from temporal fusion. Most of these works fuse multi-frame features at low resolution and use a limited number of timesteps over short time window both factors that we find severely decrease localization potential. In our work, we quantify these limitations and propose strong framework that explicitly considers the relationship between spatial resolution and temporal history.
2.3
MULTI VIEW STEREO
Multi-view stereo works use depth maps (Kang et al. 2001) and 3D volumes Kutulakos & Scitz, to conduct depth, mesh and point-cloud reconstruction tasks. Several methods (Galliani al, Ji al. (017)) use depth map fusion algorithm for large-scale scenes. Other works (Kar et al.,
2000
2015
2017
Yao et al.] 2018 Yang et al. 2020 Zbontar et al. 20166] Yao et al. 2019) generate volume by scattering pixel features to the 3D grid and estimating occupancy cost for each voxel. A few methods apply multi-view stereo works to 3D detection. DIM (Wang et al. 2022b) generates a plane-sweep volume from consecutive frames. STS (Wang et al.) 2022c) uses -tntialllyinerrasinng discretization (SID) depth bins (Fu ct al. 2018) for stereo matching, and BEVStereo (Li et al.] (022b) adapts MaGNet (Bae al. 2022 and dynamically selects candidates for iterative multi-view matching.
Although these methods demonstrate further improvement, the short history they use for detection limits their gain from temporal fusion as shown in Figure 2 We formulate and analyze the con- nection between temporal camera-only 3D detection and multi-view stereo and verify our analysis by leveraging the synergy of short-term and long-term fusion. Further, our proposed short-term temporal fusion is more efficient and extensible, and it also demonstrates larger improvement.
UNIFIED TEMPORAL STEREO FORMULATION FOR CAMERA-ONLY 3D DETECTION
3.1 BACKGROUND OF MULTI- VIEW STEREO MATCHING
Multi-view stereo matching estimates the depth map of image (reference view) by leveraging additional images taken of the same scene (source views). For each pixel in the reference view, multi-view stereo frameworks consider many depth hypotheses (Yao et al. 2018) locations this pixel could be in the 3D world. Of these hypotheses, the true depth of the pixel appears most similar through the reference and source views. Thus, each depth hypothesis is projected onto the source
3

---

Published as a conference paper at ICLR 2023
Table I: "Sampled image features are features weighted by probability prediction for that depth. TSSS and BEVStereo also perform the BEV temporal stereo done in BEVDepth. *BEVFormer ag- gregates previous timestep features recurrently. They set the previous time window to be 3 timesteps during training and maintain a running BEV feature map during inference.
Type Method
Candidate Loc.
Sampling Op.
|Sampling Rea] Temporal Age
|Pee Time Cand. Lac. Proc.
Task/Sugervision
MVSSNe
plane sweep volume
projection bilineer projection bilinesr
vatlance dot peoduct
3D coay cuav
depch estimatonnLi
MaGNett
prodicted gaussian
contidence inserval
depch estimatio'122
BEEDesDD
BEV grid cells
image feats" BEV pool
1.5%
2D coay
pred. localization
Grdd Busd
RFVDoyth
cells
image fests" BFV pool
1.0.%
carv
peed localization
sTst
SID plane-wweep vol. prodicted gausila
projection bilinzs
1/4
groupwise con:.
1,0.5
MLP
Jeyth eximation/BC
REVStcve 1
confidennz interval
projection bilinzue
1/4
groppiis con.
1,0.5
MLP
deyth esimationBCC
DMM
planzewwap vol. grid cells
projectios* biliness
1,0.3
2D
deyth dbj, pred,
MV-PC063D++
projection bilinter
concal
Is
2D
peed
BENFomer
query BEV quesy object query loc.
projection deform atn projectioss& bilinzie
1/6
1644
decocm attel
3:0 2s
cans, decoder
peed localization
Unimem
188
weighted
6,
trams decnder
map vegnestation
PolaDETR
projectios & deform atm dkform sitn eatin mmage feature polar DEV map deform altn feuture" 3D volume cross attention
unspeciliid unspecified
concat
1.5.
wans, decoder
obj, pred. localization
PolerFommer
object query loc.
align & concat
2:5
tans, decuder
obj pred. & localization
Quayy
UVTR
object query loc.
1/4 1/32
concat
5:0
obj. trans. decoder
obj pred. localization
PETRY2
object query lnc.
align cross attn
1:2.5
cans. decoder
obl peed. localization
views, and the depth with the best image feature match between the source view projection and the reference pixel is selected as the depth estimate. In the standard stereo setting, the reference and source views are left and right cameras with synchronized, rectified images. For temporal stereo, the reference and source views are images captured at the current and past timesteps as illustrated in Figure D
For additional details on multi-view stereo, we refer readers to MVSNet (Yao et al. 2018).
3.2 COMPONENTS OF A UNIFIED FORMULATION
Camera-only 3D detection methods using multiple frames each propose their own method for tem- poral feature aggregation. However, we find that these works are mostly grounded in the same core formulation of multi-view stereo matching. We identify the main components of such a formulation:
Candidate Locations- - the locations in 3D space considered for matching.
Sampling Operation the method used to obtain image features for a candidate region.
Sampling Resolution- the spatial resolution of the image features used for sampling.
Temporal Aggregation method of fusing features from different frames.
Timesteps Used- number and temporal difference of timesteps used in aggregation. Candidate Location Processing the modules used for processing the multi-timestep fea- tures aggregated to candidate locations.
Task/Supervision the task the candidate locations are used for and the corresponding loss.
Table 1 distills works into these components, and full details can be found in Appendix A
3.3 CONNECTING MULTI- VIEW STEREO AND TEMPORAL 3D DETECTION
To connect multi-view stereo to temporal 3D detection, we note the following key point: In multi- view stereo, the model predicts the likelihood that a candidate location occupied by anything while temporal 3D detection models predict whether the location is occupied by a certain object. At their core, both methods have the same goal given candidate 3D location, they both consider how that location is captured in multiple 2D views to determine whether something of interest is there.
This connection holds for both grid- and query-based methods. Grid-based methods generate dense candidate locations that cover the entire 3D or BEV region. Supervised by a heatmap loss Yin et al. (20T)) a candidate location predicts high probability if an object exists at that location. This is similar to SV which maximizes probabilities candidate locations where anything exists. On the other hand, query-based methods use sparse candidate locations around where objects are likely to exist. These queries are supervised by LI predict offsets that move query locations towards object centers. This can be formulated as the query predicting a Laplacian distribution center in 3D space with unit scale, which is directly analogous the predicted and iteratively refined L2- supervised Gaussian distribution over depths in MaGNet. Overall, only the specific details of the various components change between models in essence, both temporal grid-based and temporal query-based methods aggregate or compare multi-view 2D features to make occupancy predictions, which is intrinsically analogous to multi-view stereo matching (Yang et al. 2020).
3.4 ANALYSIS OF COMPONENTS
Having represented 3D detection methods as instances of temporal stereo under our framework, examine their various components to identify key points hindering their temporal fusion. From
4

---

Published as a conference paper at ICLR 2023
Optimal Time Difference for Various Depth Hypotheses and X Image Coordinates
FRONT LEFT
FRONT
CAN FRONT RIGHT
CAM BACK NIGHT
CAM BACK
CAM BACK
-15
-15
22
10
23
10
SS 00
Aefiraprs Aeieaggh Sirear
Figure 4: Optimal time difference over candidate locations. White regions have NO valid projection.
Table we observe that most 3D detection significantly limited in the number of frames and the time window they fuse, with only BEVFormer and UniFormer considering longer fusion. How- ever, BEVFormer demonstrates no improvement from training on more than 3 timesteps, show- ing that their sequential fusion framework hinders the potential for long-term temporal fusion. Further, UniFormer does not evaluate on 3D detection and their performance on map segmenta- tion peaks with 3 seconds of history. Compared with MVS works that all use high 1/4 reso- lution, many detection methods use low 1/16 resolution feature maps for object detection tempo- ral stereo. Although some works do benefit from higher-resolutee inspired works such as STS, BEVStereo, and DíM, their short temporal window and few frames caused by computational limitations hinder their localization ability. We will show in Section A that limited temporal fusion and low resolution features in prior work are both factors that limit the localization potential of existing methods, motivating the design choices of our method that lever- ages both long-term temporal fusion and high-resolution features while maintaining high efficiency.
feature maps, notably MVS-
4 THEORETICAL ANALYSIS
In Section 3 we formulated temporal camera-only 3D detection works as instances of multi-view stereo. This allows us to analyze the multi-view object localization ability of these works in context of multi-view stereo setup. In this section, we fo- the general two-view setting and perform theoretical analysis on how realistic changes be- tween views affect the ease of multi-view depth estimation. Specifically, we derive a formulation for our defined localization potential and examine how temporal differences between the views and image resolution affect the potential.
+2
+x *
Comera Reference
Camera B: Source
Figure 3: Reference and source coordinate sys-
tems. Arrows indicate positive angle direction.
4.1
DERIVATION OF LOCALIZATION POTENTIAL
Let image A be the reference view we predict depth in and let image B be the source view we leverage for multi-view depth estimation. Further let (Ia, Vo, da) denote a pixel (za, (a) in image with depth da, our candidate point, and let (2a.26, y6, db) be its corresponding projection onto image B. Camera and view details are in Appendix and shown in Figure 3 We previously defined localization potential as the magnitude of the change in the source-view view projection induced by a change in the depth in the reference view. Focusing our analysis on 2 ya is a sub-case of xb with 0, ly O, we represent localization potential at (Ia, Mo, da) as:
Localization Potential = 1xx
fīcos(a) sin(a (0 + 3))
(1)
adg (d,, cos(a 0) + t.2 cos(a))
where o; 3. 0 are the angles of image A x-coordinate, translation, and rotation between views shown
in Figure and is the magnitude of translation + 12. The full proof is in the Appendix
4.2 EFFECT OF TEMPORAL DIFFERENCE ON LOCALIZATION POTENTIAL
We consider how the temporal difference between the reference image A and the source image B effects localization potential. To do so, we wr-te Equation by introducing time offset t
axs
cos(a) sin(a (0 (0+03 B))
(2)
adg
(d, cos(a 8) + tgt cos
FIn the main paper. we limit the effect of time to translation and exclude because while it common for a vehicle to maintain the same rate of translation over long periods of time, rotation often varies from timestep to timestep as the vehicle makes small corrections. Over multiple timesteps. the aggregate 0 is close to 0, and we adopt this setting in the main paper. We do, however, analyze effects of rotation on localization potential in Appendix B.4 also examine situations such turns where @ varies with time in Appendix B.6
5

---

Published as a conference paper at ICLR 2023
(Multi-Timestep Localization Potential) / (Single-Timestep Localization Potential)
CAM FRONT LEFT
FRONT
(CAM FRONT RIGHT
CCM BACK RIGHT
CAM
LECK
73
222
23
0.40
40
Acteogh
Figure 5: Visualization of relative increase in localization potential from using multiple timesteps.
Note that each camera heatmap has different scale.
3D Object Centers with >5ub-Pixel Change Projection with 0.5m Change Depth Aggregated Timesteps
CAM FRONT LEFT
CAM FRONT
4. gon Dany
CAM FRONT NIGHT
CAM LEFT
CAM BACK
CAM BACK RIGHT
UHAT 201 25m
1457% 07
3407100 -259
5405% 200
busc
24
21 2 Ranivolend Rirden
Figure 6: Visualization of % of objects that can benefit from multi-view stereo.
For each candidate location over various depths and X-axis image A coordinates, we find the timestep difference between the reference and source views that maximizes its localization potential. First only allowing projections within the same camera, we observe Tigureettha each camera has unique trends and that closer depths prefer smaller time differences. A more extensive breakdown also considering projections between cameras and variations in @ over time is in Appendix B.5 To summarize this more complex case, we find that the optimal time difference between views varies wildly over even neighboring candidate points. As such, it is sub-optimal to determine a single time difference over which do matching as prior works do. Instead, leveraging many timesteps over longer history significantly increases localization potential over candidate locations.
4.3 IMPACT OF RESOLUTION ON LOCALIZATION POTENTIAL
We now consider the impact of image feature resolution on localization potential. Since downsam- pling image features by 4x decreases the effective focal length by 4x, the localization potential also divided by 4 from Equation 2 We consider& 4 because the standard multi-view matching resolution 1/4 and the common detection temporal stereo resolution from Section 3 1/16. Although deep features can maintain some intra-pixel localization information even at lower resolution, down- sampling still makes multi-view matching more difficult. As analyzed in Section 3 image feature resolution is often constrained by computational limitations. However, we find that the increased localization potential from aggregating more timesteps can compensate for the decrease in poten- tial caused by downsampling. In Figure 5 we show the relative increase in localization potential from aggregating more timesteps and find that for most of the candidate locations, the localization potential is improved by far more than factor of 4. Furthermore, the locations with less than factor of 4 improvement are generally close depths for which monocular depth estimation has good performance and localization potential already high enough. Thus, our analysis shows that aggre- gating more timesteps can compensate for a lower image feature resolution, maintaining high efficiency with same localization potential as high-resolution feature matching.
4.4 EFFECT OF TEMPORAL. DIFFERENCE ON MULTI- VIEW DEPTH AMBIGUITY
So far, we have shown theoretically that aggregating multiple timesteps allows for more diverse rotations and greater localization potential for various candidate locations. We conclude our analysis by directly verifying the impact of aggregating multiple timesteps on multi-view depth ambiguity. More specifically, we consider the projection difference induced by varying the 3D centers of objects in the nuScenes training set by 0.5m² Using frame-to-frame ego-motion provided by nuScenes, Figure x shows the percent of 3D centers with projection difference of at least one pixel, which is the minimum necessary to accurately localize point in 3D using multiple views. A more thorough analysis is in Appendix [.7] and we conclude that our theoretical analysis holds true empirically with larger number of aggregated timesteps, the percent of objects that can benefit from multi-view stereo dramatically increases over all cameras and depths.
METHOD
We propose SOLOFusion, natural extension of our theoretical analysis. Core to our method and design choices is balancing the impacts of image resolution and temporal aggregation on localization potential. SOLOFusion has two main streams: 1) LSS-based object detection temporal stereo for
Most methods adopt this granularity for depth prediction Li et
2022c b
Liu al. 2022c

---

Published as a conference paper at ICLR 2023
High Resolution-Short Fusion (HR-ST Fusion)
Plane-sweap cost volume
sample
depth Low-resolution-Long-eern Fusion
warp
BEV cost
Source view
Reference view Reference view
HR-ST Fusion
HR-ST Fusion
T Figure 7: The framework of SOLOFusion.
t=T-1
t=T
coarse image features using long history; 2) MVS-based depth prediction temporal framework for high resolution image features using a short history. The framework is presented in Figure] 7
5.1 NOW-RESOLUTION, LONG-TERM OBJECT DETECTION TEMPORAL STEREO
Using image features and depth prediction at a coarse 1/16 resolution, we densely generate a point cloud and voxelize into BEV feature map similar to (Huang al. 2021 et al. 2022cp. Different from these methods, we offset the deterioration in localization potential caused by the low image resolution by generating a BEV cost volume leveraging long history. Specifically, we align the BEV feature maps from the previous T timesteps to the current timestep and concatenate them. Then, we use the resulting long-term BEV cost volume for detection. We find that this straightforward and efficient pipeline already outperforms existing methods by a large margin.
5.2 HIGH-RESOLUTION SHORT-TERM DEPTH ESTIMATION TEMPORAL STEREO
We then consider using high-resolution image features to offset short temporal history. To improve the per-frame depth estimation used for object detection temporal stereo in S.c SOLOFusion leverages an MVS temporal stereo pipeline using two consecutive frames. We consider plane- sweep volume in the reference view at 1/4 resolution and perform stereo matching with the previous frame image features. Opting for a straightforward pipeline to test our analysis, we simply add the cross-view correlations with the monocular depth predictions, supervising them together.
As this naive setup is computationally expensive, we propose to use monocular depth prediction to guide the stereo matching. we draw inspiration from the exploration and exploitation trade-off com- mon in reinforcement learning. Stereo matching a candidate depth is expensive and is analogous to collecting observation in RL. We hypothesize that balancing exploitation, using the monocular prior, and exploration, considering other candidate locations for stereo, when generating depth hy- potheses best leverage the expensive temporal matching. Hence, propose Gaussian-S SSaced Top-k sampling, an iterative framework to generate sampling points. We first choose the depth hy- pothesis with the highest monocular probability, exploiting our prior. Then, to force exploration, we down-weight the monocular probability of its neighbors, forcing exploration beyond simple prior maxima. We repeat this process, yielding set of k candidate locations for each pixel, and perform temporal stereo on these points. Additional details can be found in Appendix C
We find that this pipeline is able to cover modal depth distributions and far outperforms simply top-k sampling with minimal computational cost. Our high-resolution, short-term depth estimation module further improves on strong low-resolution, long-term temporal stereo model, demon- strating that short and long-term temporal stereo complementary.
Necessity of Balance. Localization potential is maximized if we use high-resolution, long-term fusion. However, we find that even high-resolution, short-term temporal stereo imposes significant memory and runtime costs without our proposed matching point sampling method, making extension to high-resolution, long-term intractable. Our pipeline is built good trade-off between resolution and time difference, using improvements in one to compensate for deterioration the other, maximize localization potential under computational constraints.
theoretical analysis to find

---

Published as a conference paper at ICLR 2023
Table 2: Comparison on the nuScenes val set. Note that in each category. we evaluate against
methods with comparable or stronger setting than us. Initialized from FCOS3D backbone.
Methods
Backbone
Image
CBGS
mAPt
NDS
mATEL
mASEE
mAOE]
mAVE.
mAAEL
BEVDet
ResNet50
0.725
0.279
0.589
0.245
PEIR
ResNet50
384 x 1056
0.313
0.381
0.768
0.278
0.561
0.923
0.225
BEVDesAD
ResNet50
256 704
0.322
0.457
0.703
0.278
0.495
0.354
0.206
BEVDepch
ResNet50
256 x 704
0.351
0.475
0.639
0.267
0.479
0.428
0.198
STS
ResNet50
256 704
0.377
0.489
0.601
0.275
0.450
0.446
0.212
BEV Stereo
ResNet50
0.598
0.270
0.190
SOLOFusion
ResNet50
256 x 704
0.427
0,534
0.567
0,274
0,511
0.252
0,188
FCOS3D
ResNet1OJ-DCN
900 x 1600
X
0.295
0.372
0,806
0.268
0,511
1,131
0.170
BEVFomert
ResNet101-DCN
900 x 1600
X
0.416
0.517
0.673
0.274
0.372
0.394
0.198
PolaDDETR-Tf
ResNet1O1-DCN
900 1600
0.707
0.269
UVTR
ResNet1O1-DCN
900 1600
0.379
0.483
0.731
0.267
0.350
0.510
0.200
PolarFormer!
ResNet101-DCN
900 x 1600
0,432
0.528
0.648
0.230
0.348
0.409
0.201
OOLOission
ResNet101
1408
0.518
0:275
0.210
DETR3D
RsNC
900 x 1600
0.349
0.434
0.716
0.268
0.379
0,842
0,200
PETR
ResNet101
512 x 1408
0.357
0.421
0.710
0.270
0.490
0.885
0.224
BEVDepch
ReNNe1011
512 x 1408
0.112
0.535
0.565
0.266
0.358
0.331
0.190
STS
ResNe1011
512 140%
0.431
0.542
0.525
10,262
0.380
0.366
0.204
SOLOFesion
ResNc1101
0.503
0.261
0.381
6 EXPERIMENTS
6.1 DATASET AND METRICS
We use the large scale nuScenes dataset Caesar et al. 2020). containing 750, 150, and 150 scenes for training, validation, and testing, respectively. Each sequence is 20s long and is captured using 6 cameras with resolution 900 1600. 3D box annotations are provided key frames every 0.5s. We refer readers to the official paper for metrics. The full experimental details are in Appendix D
6.2
MAIN RESULTS
We report our results on the nuScenes val and test sets in Table 2 and Appendix Despite often operating under weaker settings, SOLOFusion far outperforms prior tf-he-of under every setup. Notably, in the ResNet101, CBGS setting, our framework outperforms STS, which extends BEVDepth to incorporate multi-view stereo, by a substantial 5.2% mAP. This improvement, coupled with an increase in mATE which is also a object localization metric, verifies our theoretical analysis that longer temporal fusion can indeed allow for better multi-view localization. Our method is also much better predicting velocity, demonstrating 12.3% improvement in mAVE, which shows that long-term temporal fusion is not just better for localization also improves velocity estimation by observing dynamic objects for longer. SOLOFusion obtain 1st place the nuScenes test set camera-only track at time of submission. Our framework achieves this by training without CBGS, with shorter eycle, and without large-scale depth training or test-time augmentation, demonstrating the importance of using both short-term and long-term temporal fusion.
6.3 ABLATION STUDY& & ANALYSIS
Ablation of Time Window for Temporal Fusion.
We start with a BEVDepth baseline with temporal fusion and ablate the addition of low-
Table 3: Ablation for low-res, long-term fusion.
Timestpp
mAP1
mATE
mAVE
resolution, long-term fusion in Table 3 Al- though fusing single timestep improves veloc- ity prediction which improves NDS, the local- ization metrics mAP and mATE improve only slightly. As we fuse more timesteps into our BEV cost volume, the mAP and mATE increases, improving by 6.1% and and 7.9% from I to 16 timesteps. This supports our analysis that compared to using a single timestep, leveraging many timesteps over larger time window substantially increases localization potential. We find that per- formance saturates 16 timesteps as there is little overlap the visible region beyond 16 timesteps. Further analysis of the impact of long-term fusion on static and moving objects are in Appendix G
0.307
0.347
0,743
1.148
0.316
0.423
0.734
0.456
0.336
0.434
0.736
0.382
0.349
0.452
0.701
0.332
0.366
0.465
0.686
0.317
16
0.377
0.474
0.655
0.307
Ablation of Depth Hypothesis Sampling. In Table 4 we ablate various methods for choosing the depth hypotheses for high-resolution, short-term temporal stereo. Starting with the fingle-frame baseline, we find that attempting to match all depth hypotheses (112 for each pixel) increases runtime by 6x and significantly increases GPU memory usage. The training time and memory cost increase is even more significant, and we find it impossible to train this model properly. A model with 28 uniformly sampled hypotheses is able to improve performance, but it imposes 2.4x slowdown
8

---

Published as a conference paper at ICLR 2023
Table 4: Ablation of depth hypothesis sampling methods
Method Depth Sampling
Depth Hy potheses
PPS
Memory
mAP
NDST
mATE
Nonc; Single-Frame BEVDepth
17.6
GB
0.321
0.349
0.722
Uniform Sampling
112 (all)
2.9
8.5 GB
Unifoem Sampling
7.4
4.2
0.345
0.377
Uniform Sampling
7
12.0
3.3 GB
0.319
0.359
0.727
Top- Sampling
11.9
GB
0.336
0.390
0.674
Gausian-Spacdd Top-k Sampling
7
11.4
GB
0.343
0.389
0.670
Table 5: Ablation of short-term and long-term fusion components
High-Res, Short-Term
Low-Res, Long-Term
FPS
Memory
mAPt
NDS1
mATEJ
(a)
17.6
3.3GB
0.321
0.349
0,722
(h)
12.2
3.3GB
0.343
0.389
0.670
(6)
15.9
3.6BB
0.386
0.479
0.650
(4)
11.4
3.6GB
0.404
0.495
0.605
Table 6: Analysis of whether many-frame temporal fusion can compensate for low-resolution.
Method
Backbone
Image Size
FPS
Memory
MAPt
NDS1
mATE
BEVDepth
ResNet50
512 x 1408
2.3
7.3 GB
0.405
0.523
0,570
SOLOFusion
ResNet50
256 x 704
11.4
3.6 GB
0.427
0.534
0.567
and increase in GPU memory. Although uniformly sampling candidates improves runtime, demonstrates no improvement from stereo. Naively leveraging the monocular depth prior with top-k sampling, we do see improvement in all metrics. Interestingly, we find that top-k sampling performs even better than the 28 uniformly sampled hypotheses for localization (mATE), demonstrating that monocular priors are useful for guiding stereo matching. Replacing naive top-k with our proposed Gaussian-Spaced Top-k Sampling improves mAP and mATE with a minimal decrease in FPS.
Ablation of High-Res, Short-Term Fusion and Low-Res, Long-Term Fusion. Starting from a baseline BEVDepth model, we ablate the addition of our short-term and long-term fusion com- ponents in Table []] We find that although the addition of the high-res, short-term matching from (a) to (b) slows down runtime from 17.6 to 12.2 FPS, there no increase in GPU memory cost and the localization mATE is significantly improved by 5.2%. Instead adding the low-res, long-term matching from (a) to (c), the improvement in mAP is huge with minimal drop in FPS. We note that purely considering localization improvement, adding high-res, short-term improved by mATE by 5.2% while low-res, long-term improved by 7.2% The comparatively similar increase shows that both modules have a similar amount of localization potential, illustrating the trade-off we analyzed in Section 4.3 Finally, adding both modules further improves performance for all metries. We ob- serve that the the individual improvements from the two modules, ~5% and ~7% yield a combined improvement of ~ 12%, demonstrating that the two sides of the trade-off are highly complementary.
Balancing Temporal Fusion and Resolution. In Section 4.3 we found that extensive temporal fusion can compensate for lower resolution terms of localization potential. We verify this Table by comparing half- malf-resolution SOLOFusion, designed to offset low resolution with long-term tem- poral fusion, with full-resolution BEVDepth. SOLOFusion outperforms BEVDepth by 2.2% mAP and 1.1% NDS. Furthermore, SOLOFusion demonstrates comparable object localization (mATE) despite operating at half resolution, verifying our theoretical analysis. This also demonstrates an important property of SOLOFusion. By compensating for decrease in resolution, which dramat- ically decreases inference cost, with an increase in temporal fusion, which has minimal impact on latency, towrrssolution SOLOFusion achieves similar performance high- BEVDepth at roughly 5x the FPS and half the memory cost. Additional runtime analysis is in Appendix F
7 CONCLUSION
In this paper, we provide new outlooks and a baseline for temporal 3D object detection. We re- formulate the multi-view 3D object detection as an instance of temporal stereo matching and define localization potential to measure the ease of multi-vien view depth estimation. Through detailed theoreti- cal and empirical analyses, we demonstrate the importance of temporal difference and granularity of features for temporal stereo. We find that using long temporal history significantly increases local- ization potential even with low-resolution features. Our analyses lead to natural baseline, termed SOLOFusion, which takes advantage of the synergy of short and long-term -erm temporal information as well as the trade-off between time and resolution. Extensive experiments demonstrate the effec- tiveness of the proposed method and SOLOFusion achieves a new state-of-te satte-of-the-an art on the nuScenes dataset, outperforming prior-arts by 5.2% mAp and 3.7% NDS and placing first in the leaderboard.

---

Published as a conference paper at ICLR 2023
REFERENCES
Gwangbin Bac, Ignas Budvytis, and Roberto Cipolla. Multi-view depth estimation by fusing single- view depth probability with multi-view geometry. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Ivan Barabanau, Alexey Artemov, Evgeny V. Burnaev, and Vyacheslav Murashkin. Monocular 3d object detection via geometric reasoning on keypoints. ArXiv, htss/1905.05613 2020.
Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object
detection. In ICCV, 2019.
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Shaoyu Chen, Xinggang Wang. Tianheng Cheng, Qian Zhang, Chang Huang, and Wenyu Liu. Polar parametrization for vision-based surround-view 3d detection. arXiv preprint arXiv:2206.1099 2022.
Huan Fu, Mingming Gong, Chaohui Wang, Kayhan Batmanghelich, and Dacheng Tao. Deep Ordi- nal Regression Network for Monocular Depth Estimation. In CVPR, 2018.
Silvano Galliani, Katrin Lasinger, and Konrad Schindler. Massively parallel multiview stereopsis by surface normal diffusion. In Proceedings of the IEEE International Conference on Computer Vision, pp. 873-881. 2015.
Clément Godard, Oisin Mac Aodha, and Gabriel J. Brostow. Unsupervised monocular depth es- timation with left-right consistency. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 6602-6611, 2017.
Vitor Guizilini, Rares Ambrus, Sudeep Pillai, Allan Raventos, and Adrien Gaidon. 3d packing for self-supervised monocular depth estimation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Xiaoyang Guo, Kai Yang. Wukui Yang, Xiaogang Wang, and Hongsheng Li. Group-wiee wiuppwwsee&o correla- tion stereo network. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3268-3277 2019.
Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Yuxin Hou, Juho Kannala, and A. Solin. Multi-view stereo by temporal nonparametric fusion. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2651-2660, 2019.
Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues multi-camera 3d object detec- tion. arXiv preprint arXiv:2203. 02.2203..775.. 77054, 2022.
Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du. Bevdet: High-performance multi-camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11796 2021.
Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. Surfacenet: An end-to-end 3d neural network for multiview stereopsis. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2307-2315, 2017.
Yanqin Jiang, Li Zhang, Zhenwei Miao, Xiatian Zhu, Jin Gao, Weiming Hu, and Yu-Gang Jiang. Polarformer: Multi-camera 3d object detection with polar transformers. arXiv preprint arXiv:22066 15398, 2022.
Longlong Jing, Ruichi Yu, Henrik Kretzschmar, Kang Li, C. Qi, Hang Zhao, Alper Ayvaci, Xuanyao Chen, Dillon Cower, Yingwei Li, Yurong You, Han Deng, Congcong Li, and Drago Anguelov. Depth estimation matters most: Improving pbj-object depth estimation for monocular 3d detection and tracking. 2022 International Conference on Robotics and Automation (ICRA), pp. 366-373, 2022.
10

---

Published as a conference paper at ICLR 2023
Sing Bing Kang, Richard Szeliski, and Jinxiang Chai. Handling occlusions in dense multi-view stereo. In Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001, volume pp. I-I. IEEE, 2001.
Abhishek Kar, Christian Häne, and Jitendra Malik. Learning a multi-view stereo machine. Advances in neural information processing systems, 30, 2017.
Kiriakos N Kutulakos and Steven M Seitz. A theory of shape by space carving. International Joternationalluural journal of computer vision, 38(3):199-218, 2000.
Yanwei Li, Yilun Chen, Xiaojuan Qi, Zeming Li, Jian Sun, and Jiaya Jia. Unifying voxel-based representation with transformer for 3d object detection, arXiv preprint arXiv:2206.. 20X6::22066006300 2022a.
Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with dynamic temporal stereo. arXiv preprint MrXXiv2209.102444 2022b.
Yinhao Li, Zheng Ge. Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv preprint arXiv:2206. 10092, 2022c.
Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning bird's bird'--eye-viex representation from multi-camera images via spa- tiotemporal transformers. arXiv preprint 2rXiv:2203.17270. 2022d.
Zhuoling Li, Z. Qu, Yang Zhou, Jianzhuang Liu, Haoqian Wang, and Lihui Jiang. Diversity matters:
Fully exploiting depth clues for reliable monocular 3d object detection. ArXiv, abs/2205.09373, 2022e.
Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transformation for multi-view 3d object detection. arXiv preprint arXiv:2203.05623, 2022a
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang. Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint wrXiv:2206.01256, 2022b.
Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han. Bevfusion: Multi-task multi-sensor fusion with unified bird's-eye view representation. ArXiv, abs/2205.13542 2022c.
Zhuang Liu, Hanzi Mao, Chaozheng Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11966-11976, 2022d.
Zongdai Liu, Dingfu Zhou, Feixiang Lu, Jin Fang, and Liangjun Zhang, Autoshape: Real-time awape-aware monocular 3d object detection. In ICCV, 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 2019.
Fabian Manhardt, Wadim Kehl, and Adrien Gaidon. Roi-10d: Monocular lifting of 2d detection to 6d pose and metric shape. 2019 IEEE/CVF Conference on Computer Vision and Patlern Recog- nition (CVPR), pp. 2064-2073. 2019.
Arsalan Mousavian, Dragomir Anguelov, John Flynn, and Jana Kosecka. 3d bounding box estima- tion using deep learning and geometry. In CVPR, 2017.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. European Conference on Computer Vision, pp. 194-210. Springer, 2020.
Zengyi Qin, Jinglu Wang, and Yan Lu. Monogmeet geometric reasoning network for monocular 3d object localization. In AAAI, 2019.
11

---

Published as a conference paper at ICLR 2023
Zequn Qin, Jingyu Chen, Chao Chen, Xiaozhi Chen, and Xi Li. Uniformer: Unified multi- view fusion transformer for spatial-temporal representation in bird's s-eee-vew. arXiv preprint arXiv:2207.08536. 2022.
Cody Reading, Ali Harakeh, Julia Chae, and Steven Waslander. Categorical depth distribution- network for monocular 3d object detection. In CVPR, 2021.
Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv preprint arXiv: 1.0.:111//1100 1.08/88, 2018.
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Imvoxelnet: Image voxels projec- tion for monocular and multi-view general-purpose 3d object detection. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 2397-2406, 2022.
Andrea Simonelli, Samuel Rota Bulò, Lorenzo Porzi, Manuel López-Antequera, and Peter Kontschieder. Disentangling monocular 3d object detection. 2019 IEEE/CVF International Con- ference on Computer Vision (ICCV), pp. 1991-1999, 2019.
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. 2021 IEEE/CVF Conference on Computer Vision and Pultern Recognition (CVPR), pp. 15593-15602, 2021.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. FCOS3D: Fully convolutional one-stage monocular 3d object detection. In ICCV Workshops, 2021a.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Probabilistic and Geometric Depth: Detect- ing objects in perspective. In CoRL, 2021b.
Tai Wang, Qing Lian, Chenming Zhu, Xinge Zhu, and Wenwei Zhang. Mv-fcos3d++: Multi- view camera-only 4d object detection with pretrained monocular backbones. arXiv preprint arXiv:2207.12716, 2022a.
Tai Wang, Jiangmiao Pang, and Dahua Lin. Monocular 3d object detection with depth from motion. In European Conference on Computer Vision (ECCV), 2022b.
Tai Wang, Jiangmiao Pang, and Dahua Lin. Monocular 3d object detection with depth from motion. arXiv preprint arXiv:2207. 12988, 2022c
Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Camphell, and Kilian Wein- berger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019.
Yue Wang. Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang. Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Con- ference on Robot Learning, pp. 180-191. PMLR, 2022d.
Zengran Wang, Chen Min, Zheng Ge, Yinhao Li, Zeming Li, Hongyu Yang, and Di Huang. Sts: Surround-view temporal stereo for multi-view 3d detection. arXiv preprint arXiv:2208. 10145, 2022e.
Xinshuo Weng and Kris Kitani. Monocular 3d object detection with pscudo-lidar point cloud. In ICCV Workshops, 2019.
Bin Xu and Zhenzhong Chen. Multi-level fusion based 3d object detection from monocular images. 2018 IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2345-2353, 2018.
Jiayu Yang, Wei Mao, Jose M Alvarez, and Miaomiao Liu. Cost volume pyramid based depth inference for multi-view stereo. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4877-4886 2020.
Yao Yao, Zixin Luo, Shiwei Li. Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi-view stereo. In Proceedings of the European conference on computer vision (ECCV), pp. 767-783, 2018.
12

---

Published as a conference paper at ICLR 2023
Yao Yao, Zixin Luo, Shiwei Li, Tianwei Shen, Tian Fang, and Long Quan. Recurrent mysnet for high-resolution multi-view stereo depth inference. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 5525-5534, 2019.
Tianwei Yin, Xingyi Zhou, and Philipp Krähenbühl. Center-based 3d object detection and tracking. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11779- 11788, 2021.
Jure Zbontar, Yann LeCun, al. Stereo matching by training a convolutional neural network to compare image patches. J. Mach. Learn. Res., 17(1):2277-2118, 2016.
Yunpeng Zhang, Jiwen Lu, and Jie Zhou. Objects are different: Flexible monocular 3d object detection. 2021 IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp- 1988-3297, 2021.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. ArXiv, abs/1904.07850, 2019.
Benjin Zhu, Zhengkai Jiang, Xiangxin Zhou, Zeming Li, and Gang Yu. Class-balanced grouping and sampling for point cloud 3d object detection. ArXiv, abs/1908.09492, 2019.
13

---

Published as a conference paper at ICLR 2023
A ADDITIONAL DETAILS FOR THE UNIFIED TEMPORAL STEREO FORMULATION FOR CAMERA-ONLY 3D DETECTION
In this section, we explain how the various temporal methods can be organized into our framework in Section 3 We group methods by their type.
A.1 MULTI-VIEW STEREO METHODS
MVSNet. As the seminal multi-view stereo work, MVSNet fits directly into framework. Its candidate locations considered for matching is every (I, y, d) point in the reference camera view. Image features are obtained for these points via simple projection to each view a fine 1/4 resolution and subsequent bilinear sampling. Having obtained image features from multiple views two source views and one reference view for each candidate location, these features are aggregated through a variance metric. Finally, the candidate locations in the plane-sweep volume are processed using 3D convolutions, and MVSNet predicts a probability for every (z, y. d) candidate location using the aggregated temporal information. Intuitively, this probability value is prediction of whether the candidate location is occupied in space. Finally, this probability distribution over depths for every pixel is used to take a weighted sum over depth locations, yielding a single depth prediction for every pixel, which is then supervised by LI loss. Our formulation is able to encapsulate every component of representative multi-view/temporall stereo framework, and we find it generalizes naturally to temporal 3D detection.
MaGNet. As another MVS work, MaGNet can also be deconstructed into components. Its didate locations are (z, y, d) points within the confidence interval of the predicted depth Gaussian. The of the pipeline is similar that of MVSNet, only different in that it uses dot product instead of variance to aggregate temporal features and uses a 2D CNN to process the temporal features at the candidate locations. Using these processed features, MaGNet iteratively updates the depth Gaussian and is supervised via L2 to align the Gaussian with the true ground truth depth.
A.2 GRID-BASED 3D DETECTION METHODS
BEVDet4D. The candidate locations are BEV grid cells, and each cell samples features by pooling over (x, y, d) image points that fall within each cell. As pooling is done over densely generated depth hypotheses, the out out-projected image feature resolution is kept coarse 1/16. BEVDet4D aligns a previous BEV feature map with the current one using ego-motion and simply concatenates them. Considering this alignment in reverse, grid location receives features from (ır, y, d) past image points that were within that grid cell location before. In this manner, each grid cell receives image features of multiple timesteps from regions in 2D views around the grid cell's 2D projection. For inference, they use a single historical observation at 2.5s and process the fused BEV map with 2D convolutions for object prediction.
BEVDepth. BEVDepth, as an extension of BEVDet, maintains the exact same main components as BEVDe14D. It does, however, use more recent frame of 0.5s for aggregation.
STS. Already briefly discussed in the main paper, STS extends the BEVDepth pipeline, and hence the BEVDet4D pipeline, by adding an MVS depth estimation temporal stereo component. The com- ponents that make up the MVS depth estimation component of STS are mostly identical to that of MVSNet. Although STS appears to leverage both low and high resolution temporal stereo, their low- resolution temporal stereo inherited from BEVDepth is still severely hindered by the single temporal frame they use, limiting the improvements STS can obtain from utilizing temporal information.
BEVStereo. Similar to STS, BEVSterco extends the BEVDepth pipeline by adding an MVS depth estimation temporal stereo module. However, instead of drawing from MVSNet, BEVStereo instead adopts intuitions from MaGNet.
DEM. Similar to STS and BEVStereo, DIM is an MVS-inspired detection method. However, instead of group-wis wise correlation. DfM uses concatenation of temporal features. Further, they use 2D & 3D convolutions to process the aggregated features before using them for both depth estimation and object prediction.
14

---

Published as a conference paper at ICLR 2023
MV-FCOS3D++. For candidate locations, they use 3D voxel centers, in LiDAR coordinates, and they project these voxels to the image plane to sample features. This is unlike LSS- -SS-based works that generate a dense point cloud from image features and use voxelization steps. Similar to DfM, they concatenate multiple timesteps" image features in the 3D volume and process the 3D volume with 3DD and 2D convolutions. Unlike DfM, they do not do depth estimation, only doing object prediction.
A.3 QUERY-BASED 3D DETECTION METHODS
BEVFormer. The candidate locations are queries aligned to BEV cell locations, and each query has fixed 3D sampling locations along the z-axis which are projected onto images. BEVFormer uses several coarse image feature maps from 1/16 to 1/64 resolution. The previous timestep BEV queries are aligned to the current frame using ego-motion their features are aggregated to current queries using deformable attention. Similar the formulation of BEVDet4D, this temporal alignment al- lows current BEV query candidate locations to aggregate image features that correspond to it in previous timestep. We note that BEVFormer continuously saves the past BEV queries for the next timestep, allowing for continuous temporal fusion. However, they observe no benefit beyond train- ing using three historical timesteps, limiting their potential for long-term temporal fusion. Finally, a transformer decoder is used on the BEV queries for object prediction.
UniFormer. UniFormer adopts the same BEV queries as BEVror,r,, but uses sampling isntead of deformable attention aggregate image features. UniFormer uses 1/8 to 1/64 resolutions and aggregates previous timesteps' information by projecting sampling locations to previous timestep images and fusing multi-timestep features through learned weighted sum. This work experiments with longer time history, up to 10 past timesteps, but they observe no gains beyond 6 timesteps (3 seconds), and they adopt this setting for their final model. UniFormer is supervised with map segmentation only.
PoIarDETR. Although similar BEVFormer, PolarDETR instead uses moveable object queries as candidate sampling locations. Projecting the predicted object query locations on to images, us- ing temporal ego-motion transformation to previous timesteps, and aggregating features using de- formable attention, PolarDETR combines features from different timesteps though concatenation. PolarDETR only utilizes a single frame at 0.5s history and processes the object queries with trans- former decoder for object prediction. As explained in Section 3.3 in the main paper, the supervised offsets of the object query location that moves it towards where objects are likely to exist causes Po- larDETR to be sparse candidate location version of multi-view stereo, localizing object occupancy regions in 3D space using temporal information.
PolarFormer. Like other query- query-based based methods, the candidate locations are object queries. However, different from other works that directly aggregate image features to to object queries, PolarFormer generates an intermediate Polar BEV representation that image features are first projected to. The candidate locations then aggregate features from this Polar BEV feature map. Despite this interme- diate step, the key points are still similar to other query-basee works. As the object queries perform deformable attention on the Polar BEV representation, there is still a significant prior that forces queries to aggregate features from BEV locations close to it. Since this Polar BEV representation was generated using projection and alignment of image locations, there is still a strong spatial con- nection between regions where the object query's aggregated features come from and the object query's 's projection onto the image. Unfortunately, the paper does not specify the resolution of im- age features used. The object query also has access to multiple views of it over time as previous Polar BEV representations are aligned and concatenated. The remainder of the pipeline is standard for query-based methods, with a transformer decoder then predicting objects from the candidate locations.
UVTR. UVTR is similar to PolarFormer in its use of an intermediate representation. Instead of a Polar BEV feature map. UVTR generates a 3D volume whose voxels projected onto images to get their timesteps. Past 3D volumes are aligned and concatenated, and object queries, which are the candidate locations, aggregate image features through deformable attention onto this 3D volume, allowing our formulation for PolarFormer to be directly applied to UVTR.
PETRv2. PETRv2 is different from previous works in that candidate locations aggregate image features not through projection or deformable attention but through unconstrained cross- cross-attention over both previous and current image features. However, by out-projecting image features and
15

---

Published as a conference paper at ICLR 2023
decorating them with 3D positional emebddings, PETRv2 encourages object queries attend to image features that are spatially relevant to it. Furthermore, PETR shows that object queries attend most to image locations that they projected to, making this cross-attention operation form of "soft" projection and sampling. From this observation, the rest of the analysis is identical to that of PolarDETR.
B ADDITIONAL DETAILS FOR THEORETICAL ANALYSIS B.I DETAILS OF REFERENCE AND SOURCE CAMERA SETUP
We define the camera intrinsics, inverse intrinsics, and camera camera B transformation as:
[ 0
1/5
-c// -cy RLA-B 1
cos 0
0
sin 0
1
K-
f Cy 0
K
0
1/f
(3)
[
0
0
sin 0 0
Cos 0
tz
where 0 is the rotation from camera A to camera B in the XZ plane and ly. l are translations from We exclude transformations involving the vertical Y axis as
camera A to as shown in Figure 3
they are minimal in driving scenarios. Then, the projection of (Ia, Yo. du) onto image B is:
da cos daf sin + lxl
dayl
(rb, yb]
+ Cr. d, sin + da cos 0 + l2
+ +Cp
(4)
da + da cos 0 + l2
where 2/y Ia Cz. V/a = ye Cy. B.2 DERIVATION AND ANALYSIS IMAGE A TO IMAGE B PROJECTION In this section, we derive our formulation for the projection (Ib. yb. d:) in Equation and connect to the familiar standard stereo case.
Starting from Image coordinates (Ia, Ya. da) and applying homographic transforms:
[ No: da)
= (andaa yada, dal (xr Cx) da cy)
(Image A Image Hom Coords)
(Image Camera Coords)
f
dutta datla da
(Let x'2 = Da Cz. S/ Yo cry)
x's cos 0
dayls do sin
da sin + las
+ dn cos + 1 (Image B Camera Coords)
f
da cos 0 daf sin 0 + txf
+ Cx.
da sin d, cos
datha
+ Cyy
(Image B Image Coords)
da cos 0 11 daw,, sin
dat', dat's sin
+ dg cos
0
= [Tb. Ub. de]
(Image B Image Coords)
This formulation is applicable to any two-camera system with rotation and translation along the XZ. axis. For instance, consider standard stereo setup with cameras A and B the left and right cameras, respectively: = 0, t. = O, baseline -tp- > O. Then, the above reduces to:
[ab yo: db] = da + tzf
+ Cz. daYa da
t
+ cy. da = To
Yor da d,
d.
This yields the standard stereo disparity formula disparity to For depth estimation in stereo d. temporal stereo, project multiple depth hypotheses for pixel in image A onto image B and find the pixel along the epipolar line in image B that matches best with the original pixel in image A. Given such matching,
derive the depth using the transformation matrix between the two
16

---

Published as a conference paper at ICLR 2023
images/cameras. For such formulation to work well, it is beneficial for the image B projections of nearby depth hypotheses for a pixel in image A to be as far apart from one another as possible. For instance, if two candidate depths are projected to the same pixel in image B, it is impossible to determine which of the two candidates are a better match. Even beyond same-pixel projections, with downsampled feature maps and local homogeneity of features extracted from CNN backbones, more separated depth hypothesis projections allows for easier stereo depth estimation. To quantify changes in projection from changes depth, we defined in the main paper localization potential, which is closely tied to the ease of depth estimation.
We then examine localization potential in this standard stereo case by finding
add. For the standard
stereo setup, which means depth estimation is easier, if:
dara add di
(=t))f
This partial derivative tells us that localization potential is larger,
The baseline 11 larger. This in-line with our intuition; the further apart the cameras are, smaller depth changes can be captured.
The focal length f is larger. Intuitively, if we downsample the image resolution, the focal length decreases, causing more different depth hypotheses to project to the now "larger" pixels.
The depth at which we evaluate localization potential is smaller. Indeed, the projected difference between Im and 2m is larger than the difference between 59m and 60m.
We do comment, however, that these observations do not mean the depth estimation quality in stan- dard stereo can be simply improved by adopting larger baseline and focal length. A larger baseline significantly decreases the overlapping region in standard stereo while a larger focal length limits the scene captured. In addition, observe that unlike the general two-view case in Equation] [ the localization potential does not vary over different image A x-coordinates for standard stereo. This is because the stereo cameras such, choosing an optimal setup for standard stereo is much simpler compared to the more general multi-view, temporal stereo case.
aligned, causing the epipolar lines be parallel to the x axis. As
B.3 FULL PROOF OF FORMULATION OF LOCALIZATION POTENTIAL
In this section, we detail the steps we took to derive our formulation for localization potential in Equation| We first reparameterize Ia using as defined in Figure Let raa = (= and note that we have sin a cos a =
dr cos 0- - d sin 0 + 121
Tb d. siu0 + da cos 0 1
+ Cz.
f dar's cos 0- das sin 0 + Las
Cx
(multiply top & bottom with f)
da2%, sin daf cos + tzf
dar', E' cos 0 f
1 L 0)
+ 1/s + cz + tzf
ro
a/ dar -
sin 0 + cos 0
dar, (sin a cos 0 cos sin 0) + t..
+ Cx
(substitute sin a, cos a:)
darl (sin sin cos cos 0) + tzf
dar sin(a 8) + txf
+ Cr
(sin & cos identities)
dar', cos(a 0) + txf
d, sin(a 0) + tx f
Cv
(divide top & bottom with ri)
du cos(a 0) + 1
d sin(a 0) + 1x. cos
Cr
(substitute cos a)
de cos(ar 0) + 12 cos
17

---

Published as a conference paper at ICLR 2023
Optimal Theta for Varlous Depth Hypotheses and X Image Coordinates CAM FRON RiGHH
CAM FRONT LPFT
CAM FRONT
CAM RAE RIGH EAM BACK
CAN RAAC 1
23
222
42
241
3.56
2:35
MRPER
2
2
Ptage Lienlowee
Figure 8: We visualize the optimal theta for various candidate locations.
ae from this formulation, we get: Then, deriving Jd.
Omb
I [oss(a)))l, sin(a 0) lr cos(a- 0)
0d,
(da cos(a - 0) + lg cos(a))²
Note that cos(a) > 0 for all values of a. We can then rewrite the above further by using the angular
direction of translation 3 as defined in Figure 3 The total magnitude of translation is f = v / + t
giving us sin 5 = tz/t, cos 3 = /t/ Then, we get:
0m
f cos( (os((a)tz sin(a 8) tx cos(a- - 0)
0d.
(da cos(ax 0) + l2 cos(a))²
ftcos(a) cos(3) sin(a 0) sin(3) cos(a - 0)
(da cos((x - 0) + L1 cos(
fī cos((a) sin((x (0 + 3))
(da cos(a - 0) + t, cos(a)))
which is our formulation in Equation]
B.4 EFFECT OF VIEW ROTATION 0 ON LOCALIZATION POTENTIAL
In this section, we identify the impact of on localization potential formulation in Equation D
Again, recall that larger value of 32 dd
means easier depth estimation. Critically, we first observe
that & only affects the ease of depth estimation of a pixel through its relative angular difference with of that pixel and B. This important because means there no singular camera rotation that is best for all pixels or all camera translations. The optimal rotation maximizing localization potential changes based on the pixel and the current ego-motion.
Analyzing Equation D we find that if f is close to the da cos(a - O) term increases, making the depth estimation more difficult by a factor of d3- This in-line with our intuition; if the camera rotation is such that the resulting camera B's principal axis is in-line with the pixel ray, the depth hypotheses along that pixel ray be projected close together. Further, we also want 0 + 3 to be different from (otherwise the numerator decreases). This follows similar intuition as before for depth hypotheses along a pixel ray to be projected further apart, the ego-vehicle should not rotate towards move in the same direction the pixel ray.
We empirically verify our analysis by visualizing the optimal @ that maximizes localization potential over various depths and X-axis Image A Coordinates for the six cameras in nuScenes. For trans- lational movement, we take the average ego-motion for moving scenes in nuScenes, which yields approximately ix 0.05m, lz 3.19m between consecutive frames (0.5s difference) in the front camera coordinates. The results are shown in Figure g Indeed, we find that the optimal 0 vari- ous over different pixel locations, depths, and cameras, varying most significantly over image location. As the translation direction / is different for each camera, each with its own coordinate system, by observing varied rotation values over different cameras, we verify that translation direc- tion significantly affects the optimal 0 as well. Examining the front camera with translation direction close to O, we find that the seeks to maximize the difference between a - while keeping the candidate point in-view. Furthermore, the optimal 0 does change over depth as well. For instance, the optimal # along the center ray in the back right camera changes from 0 degrees at 10m to 30 degrees at 50m. That some candidate locations prefer smaller rotations runs contrary methods used to choose matching frames in indoor temporal stereo, which impose a minimum rotation and translation Hoou et al. 2019: Sun et al. 2021) between frames to be used to for matching. Hence,
18

---

Published as a conference paper at ICLR 2023
Optimal Time Difference for Various Depth Hypotheses and X Image Coordinates With All Ref-Source Camera Pairs
Jal CAN FRONT LETT
Sart CAN TROXTT
Bab CAM FRONT RIGH!
Mar: CAS BACK RIG
Hel: CAM BACK
Rait CAN BAC LEFT
Deghh 1ip-0
Optimal Source Camera
TelDeeii fywet am 850.50
To so-th Hopoth mooorfooo
855 35 25
5724
1507
5354
1501
2501
Figure 9: Allowing projections of depth hypotheses between different cameras, we visualize the
optimal time difference in the first row. The second row shows the source camera that the depth hypothesis was projected onto to maximize localization potential. The cameras are labeled in the order of front-left to back-left.
Best Single Timestep Log Localization Potential
Ref. CAN FRONT LEFT
25225 Srow
Ref CAM FRONT
5.0
jeft CAN FRONT RIGHT
855 505955 S.S.ooo
RRet CAN BACK RIGHT
Ret CAN BACK
10
Ref. CAM BACK LEFT
855 X
8505
Tatl Ceeth
85 25888
20.00
1126
2302
1004
0534
Nof mmgger Condhaas
megineg Imgge C00
Net Imggga 3003
Bet mages Kimge Gcrerls
Rpl Imagra
Coersintts
Nst Irage'
Condratt
Best Multi-Timestep Log Localization Potential
Sef. CAM FRONT LEFT
Ref CAM FRONT
jet: CAN FRONT RIGHT
Al Dunh Monntrro 1o0
Ref CAN BICK RIGHT
CAN BACK
CAM BACK LEFT
Fat Death Hopotheen
5558%
200
11776
11.09
1336
029
1552
61
1010
11244
Setrrage
Best Multi-Timestep, Multi-Camera Log Localization Potential
Mitta Hoxtroo 355
2.5
0.0
Easth
-25
$1502
Figure 10: Log of maximized localization potential values are visualized for single-timestep with
same-camera projections in row 1. Row 2 shows multi-timestep camera projections, and row 3 shows multi-timestep multi-camera projections. Note each vertical column has same scale
there is no globally optimal rotation between views. To allow different candidate locations to maximize their localization potential, it is important utilize many views with different ro- tations irrespective of their magnitude. In practice, we can obtain diverse rotations by utilizing many timesteps over long history.
B.5 OPTIMAL TIME DIFFERENCE CONSIDERING MULTI-CAMER, PROJECTION
We first further analyze trends in optimal time difference in 4 JIntuitively for the forward-facing cam- eras where a larger time difference increases the distance between 3D point and the vehicle. this is a trade-off between the tendency of further depth points to be projected closer together (the denomi- nator) and the larger difference in views generated through ego-motion (the numerator). The former wins out for closer points and the latter for further points. The different trends over various cameras representative of both their orientation w.r.t ego-vehicle movement (we see tilted trends for the left/right slanted cameras) as well as their general forward/backward facing orientation. Next, we visualize the optimal time difference for candidate locations when allowing for projections different cameras. We also visualize the optimal source camera that the depth hypothesis is projected onto. The results are in Figure Further, we also maximized log localization potential values these
19

---

Published as a conference paper at ICLR 2023
optimal locations in Figure 10 have valid projections. This is important for two reasons. First, this allows all pixel locations and depths to benefit from multi-view depth estimation. Second, it allows the multi-camera setting to exploit larger temporal differences without worrying about non-overlapping regions. To see this, consider the back camera in Figure TT] When considering multiple timesteps (row 1 to row 2), see that the localization potential dramatically increases for regions where same-camera projections are valid. However, the close-depth regions are unable to make use of the larger temporal differ- ences due to invalid same-camera projection. However, when considering all cameras, we are able to leverage larger temporal differences for both these close-depth and previously invalid regions. In standard stereo, larger baseline, despite the easier depth estimation, causes large portions of the left & right images to not overlap. However, via our formulation, in multi-timestep temporal stereo with multiple cameras, we can leverage larger temporal differences without worrying about lack of overlap. Finally, we also notice that the multi-camera setup, although better for effective disparity, has much more complex patterns for optimal time difference compared to the single-camera setup. Similar to our conclusions when analyzing rotation, we find that the optimal time difference various for different pixels, cameras, ego- mootonn,o, depth, and camera setup.
First, notice that in the nauuti-camera setup. all depth hypotheses
B.6 OPTIMAL TIME DIFFERENCE DURING EGO- VEHICLE ROTATION
In this section, we consider the case where theta varies with time. This happens during ego-vehicle turns, and we visualize the optimal time difference over the candidate locations in realistic scenarios of 30, 60, and 90 degree turns in Figure II We find that when 9 varies with time, the optimal time difference and optimal projected camera varies wildly over different candidate locations. This shows that it is suboptimal to choose just a few temporal differences for multi-view stereo a past frame that worked well when the vehicle simply moved forward might fail drastically in more complex ego-motion scenarios such as turns. As such, we conclude it is not only optimal but also necessary to leverage many past timesteps over long time window for multi-view stereo.
B.7 ADDITIONAL ANALYSIS EFFECTS OF TEMPORAL DIFFERENCE ON MULTI-VIEW DEPTH AMBIGUITY
We find that with the single timestep aggregation used in many methods, less than 20 % of change in object center projection larger than pixel for objects 40m 60m, making accurate multi-view localization impossible. By leveraging 16 past timesteps, we significantly ease multi-view depth estimation (note that for frames with less than 16 timesteps of history, we use as many is available). We do note, however, that the critical front camera is the most difficult view. This is because as seen in Figure 9 points in the front camera can only be projected to itself and are unable to leverage multi-camera depth estimation. However, we find that multi-timestep aggregation can bring % of change in object center projection > 1px from 17% and 0.4% to 53% and 22% for objects at 20m- 40m and 40m60m, respectively, significantly decreasing the safety risk of front depth estimation. The numerical values of changes in projected location can be seen in Figure [2] As ease of depth estimation isn't simply a binary indicator of "possible" or "not possible". the actual distance between projected locations of two depth hypotheses matters as well. We find that these values increase over various cameras and depths with more temporal aggregation, demonstrating that increased temporal history significantly eases multi-view depth estimation.
C ADDITIONAL DETAILS FOR HYPOTHESIS SAMPLING
SHORT-TERM DEPTH
In this section, provide more details for our Gaussian-Spaced Top-k depth candidate sampling method. More specifically, we outline the Gaussian-based down-weighting in greater detail. Let be a the standard deviation over depth used for Gaussian-base down-wer weiihhiigt and let (d)) as the monocular depth probability of some depth candidate d at sampling iteration I. Note that the monocular depth probability is updated over multiple iterations via down-weet weighting method.
Suppose de is the depth candidate chosen at sampling iteration Write PN (x) as the proba- bility mass at location & in a normal distribution with mean i and standard deviation a. Then, the
20

---

Published as a conference paper at ICLR 2023
30 Degree Clockwise Turn Over First Seconds
Optimal Time Difference for Vartous Depth Hypothesss and Image Coordinates
230
1092
315
185
639
1129
21.25
421
Optimal Source Camera
Cat CAN FICNT LETT
Au OF DRONT
Aifornoce RPST RO1
Cr (M (K3 09d
Dr Cor OM DC3
CarCam CAM CIC 1SIT
60 Degree Clockwise Turn Over First Seconds
Optimal Time Difference Various Hypotheses Coordinates
CIA FOST
CM FIBE 201T
brl Can Ai
CM RACE (997
30
2 02
8
5554
1507
Optimal Source Camara
05 35 35
8
939
3126
2334
550
251
$5552
145
65
1129
1324
90 Degree Clockwise Turn Over First Seconds
Optimal Time Difference for Verious Depth Hypotheses and X Image Coordinates
Ast
200
Optimal Source Camera
Fetfow CAM RICAT
Anf FEST W/T
Car CM
EdlCow CMM PaCK 19F7
1M02
Pel
Figure 11: Visualization of the optimal time difference for every candidate location when the ego-
vehicle turns 30, 60, or 90 degrees over the first 8 timesteps.
monocular depth probability at some depth location d is updated to be:
Pat (d) (d) (1 PPr a3 (d) +V/2T))
This down weights the monocular probability at d by normalized inverse Gaussian factor, with the decrease being larger the closer is di. We find that this simple formulation is enough force the model to choose depth hypotheses that maintain an effective trade-off between "cxploiting" the monocular prior and "exploring" other depth candidates.
D ADDITIONAL DETAILS FOR EXPERIMENTAL SETTING
D.1 MODEL DETAILS
We adopt state-of-ther ahr BEVDepth Li et al. 2022c) as baseline model and conduct experi- ments with ImageNet- 1k pretrained ResNet50 and ResNet101 (He et al. 2016) backbones (He et al. 2016) for validation set comparison. For the test set submission, we ConyNeXt-B (Liu al.
21

---

Published as a conference paper at ICLR 2023
Magnitude Change Projection
Change Depth Object Depths
Depth dm 20th
CAN FRONT LETT
CAN FiOhT
CAM RII
CAN BACK
BACK
CAM BACK-NGI SIGHT
Depth 20m
40m
CAM FRONT LEFT
CAN MONT
CAN FAOHT RIghT
CAM BACK LETT
BACK
CAN BACK nIGT
Depth 40m 60m
CAP LIOK
PONT
CAPITONNT NIGHT
CAM BACC LDTT
CAN BACK
CAN BAGK CIGH
Figure 12: Visualization of absolute value of change projected location for object centers induced by a 0.5m change in depth.
2022d) CNN backbone pretrained on ImageNet-Ik, using pre-trained weights* from the mmelas- sification repository?, which in-turn are from the official ConvNeXt repository We use T 16 timesteps for long-term fusion and = 7 depth hypotheses for short-term fusion. For short-term, high-resolution stereo matching, we use small FPN similar to MaGNet and GWCNet (Bac ct al. 2022 Guo et al. 2019) to reduce the matching channel dimension to 64. Group correlation (Guo al. 2019) is used for matching. For BEV pooling, adopt the fast implementation from BEV. Fusion (Liu et al. 2022c). We adopt the standard augmentation strategies used by BEVDet (Huang
al.
2021
D.2 TRAINING SPECIFICATIONS
All models were trained with the Adam (Loshchilov & Hutter 2019) optimizer with weight decay le-2 and betas 0.9 and 0.999. We used gradient clipping with max normalization and mixed precision training. Following prior work (Li et al. 2022c Wang et al. 2022e) Li et al., 2022b), we maintained a constant learning rate for all models and used EMA. Models with a ResNet backbone were trained with 2e-4 learning rate and 64 batch size and our test set model with ConvNeXt was trained with 7.5e-5 learning rate and 24 batch size. For specific details on ConvNeXt training, we simply followed ConvNeXt's setup for object detection. Similar NeuralRecon (Sun al.) (021), all long-term fusion models are initially trained for 6 epochs with only one frame, then the weights are loaded into the 16-frame model and trained for 18 more epochs (fairly training for 24 epochs total for the non-CBGS setting).
During both training and inference, save past BEV feature maps and use them for later timesteps through sequential dataloader (randomizing over sequences but iterating sequentially through frames within each sequence), keeping our pipeline efficient despite using long-term temporal fu- sion. As our method processes frames sequentially, the commonly used CBGS (Zhu et al., 2019) training scheme is not readily applicable to our framework. As such, to compare with methods that use CBGS, we simply increase the number of training iterations to match the CBGS cycle without other changes. We emphasize our setting is a significantly disadvantaged setting as CBGS is known to substantially boost performance in rarer categories.
E NUSCENES TEST SET RESULTS
The test set results of SOLOFusioo are presented in Table 7 At time of submission, we rank first on the public test set leaderboard without extra data or test-time augmentation.
http://wwww....ommmmmmmcccttttaa 32xb128- noema. inlk 20220222-bb:4955.phh http://gghhhccmmmooeemmmmmmmmmme https//ghhhaaccmmccccccctttcsccc
22

---

Published as a conference paper at ICLR 2023
Table 7: Comparison on the nuScenes test set. Extra data is depth pretraining (Guizilini al,
2020
Methods
Backbone
Image
Aug mAPP?
NDST
mATE!
mASE)
mADE.
mAVEL
mAAE,
ICOSID
R101 DCN
900 1660
0.358
0,690
0.452
1.434
DETRED
V2-09
16600
0.845
V2-99
Former
V2-00
900 X 1600
0.375
0.378
BEVDeND
Swin B
900 1660)
0.451
0.568
0.511
0.241
0.386
0.301
0.121
Polusames mar
V2-99
9001 x (660)
0.493
0.572
0.556
0.256
0.364
0.439
0.127
PETRVZ
GLOM-iike
640 1660)
0,512
0.592
0.547
0.242
0.350
0.367
0.126
REVDopth
Cam-NeXi-R
640 1660
0.530
0.69
0.445
0.341
0,352
0.347
0.127
BEVStereo
V2
640 1600
0.26
0.358
30LDFusias
CunwNeXi-B
640
Table 8: Runtime and improvement of methods that utilize short-term temporal stereo with ResNet50. * FPS and Memory are based on our reproduction. Although STS does not provide de- tails, reduce the matching feature dimension, SOLOFusion does, for fair comparison. tOnly the short-term fusion module is used here. Runtime is measured with a single A6000 GPU.
Method
FPS
Memory
AmAP
AMATE
BEVSterco STS
1.8
4,8 GB
1,9
4.8
4.9
5.5 GB
2.4
4.4
SOLOFusion i
12.2
3.3 GB
2.2
5.2
F RUNTIME ANALYSIS OF SOLOFUSION.
In this section, we compare the runtime and memory costs of SOLOFusion with STS and BEVStereo, two prior works that also leverage short-term stereo for 3D detection. The results, well as each method's improvement a slightly larger improvement in mAP, SOLOFusion has runs 2.5x faster and has a larger improve- ment in localization. Compared to the short-term temporal fusion methods used in existing works, our proposed module is straightforward and efficient with strong performance.
the baseline, shown in Table 8 Although STS has
G IMPACT OF LONG- TENG-TERM TEMPORAL FUSION ON STATIC AND MOVING OBJECTS
In this section, analyze the effects of long-term temporal fusion separately on static and moving objects. For a thorough analysis, we separate classes into "static" and "movable", referring to classes whose objects move in the nuScenes dataset and classes whose objects do not move. Static classes include construction vehicle", traffic cone, and barrier, and movable classes include car, truck, bus, trailer, pedestrian, motorcycle, and bicycle. We then examine the impact of long-term temporal fusion for static objects not in motion, movable objects not in motion, and movable objects in motion. The results are Figures [3 14 and [5| respectively. For fair presentation, as any amount of temporal fusion already dramatically improves results, the plots start with past frame fused (I = 1).
G.1 OBJECTS NOT IN MOTION
First, we focus on Figures 13 and 14 We observe that for objects not in motion, long-term temporal fusion significantly improves MAP, MATE, and @R@500, which evaluate precision- recall, object center translational error, and average recall, respectively. The significant improvements in mAP and mATE support our theoretical analysis in the main text that long-term fusion can improve calization potential. mAVE, measuring velocity error, does not significantly change as it is already very low for static objects (around 0.05 m/s).
We note that we consider construction vehieles as "static" because Figure 8 of the nuScenes (Caesar et al. 2020 paper shows that only around 5% of construction vehicles in the dataset are in motion. As this is already rare class, there are scarcely any moving construction vehicles in the nuScenes validation set, we exclude them from the moving object analysis.
23

---

Published as a conference paper at ICLR 2023
Change in Metrics of Aggregated Timesteps for Static Objects not in Motion
MATE
M
AR@5001
2.5
14
1.91
+8.3%
+12.3%
Con. Vat.
Traftic Cons
E27
6.91
Averogo
Barrer
6.83
4 Aggregsted Timesiep Figure 13: Effects of long-term on static objects not in motion. AR @ 500 is average recall, averaged over center distance thresholds 0.5, 1.0, 2.0, and 4.0, for 500 predictions per scene, which is standard for nuScenes. Note that traffic cone does not have orientation, so its mAOE is always O.
of Asgreggied Thmeetegs
Adgregreed Timestape
nf Aggregsied Timesteps
Change in Metrics vs of Aggregated Timesteps for Movable Objects not in Motion
MAP1
MATE
MADE
mAvE:
5001
6.5
Truck
bis
+3.3%
Vatorcyda
Bleydo
005
Avrrae
#0grapai Thrmmpps
saprpiis Trmastape
Aggapppa Timemtat
Aagamagas Thristes
of Apgrogatee Timeropo
Figure 14: Effects of long-term on movable objects not in motion.
Change Metrics Aggregated Timesteps Movable Objects
mAP1
MATE
mAGF
mAVE
Aflgp 5001
6.20
-16.1%
23.8%
Petestiann
Taker
daaaeeeeecce Timestes
of aiarepiee Threstess
srajeopiis trvese
Figure 15: Effects of long-term on movable objects in motion.
We do note, however, that mAOE, measuring orientation error, seems to increase over timesteps. We hypothesize that this increase in mAOE error comes from our training setup. When training multi-timestep models, we use a sequential dataloader as mentioned in Section D saving BEV features from past timesteps and training iterations to be used for the current timestep and training iteration. However, the image and BEV augmentations for past and current timesteps are different, and although these augmentations are reversed when warping BEV feature maps the current timestep, this introduces additional noise that might make learning orientation in BEV more difficult. However, we still decide keep such sequential dataloader, because saving past BEV features instead of re- re-predicting them each time during training speeds up the training cycle by around 10x, and simply training the model for longer with faster iterations can more than negate these small drops (see CBGS results in Table 2). Further, the dramatic improvements in all other metrics support the utility of our proposed method.
G.2 OBJECTS IN MOTION
Next, examine the impact of long-term fusion on movable objects in motion in Figure 15 We observe significant increases in mAP, mAOE, mAVE, and AR@500 with more timesteps. The dra- matic improvements in mAOE and mAVE tell us that our proposed method, by observing moving objects for longer, can better predict their orientation (direction of movement) and their velocity. Notably, AR @ 500 increases by 8.6%, which is more than the 7.9% increase we observed for mov- able objects not in motion. We also observe that the mATE for moving objects increases by 2.3% with more timesteps. The slight increase in mATE tells us that although our method significantly improve localization of stationary elements, it does not necessarily benefit precise localization of moving objects.
However, this does not mean that proposed method does not benefit detection of moving objects. First, we clarify that mATE is a "true- positivve""" metric (Caesar et al.) 2020). meaning it is the average error for ground-truth matched predictions. For such true- rrue-tositvee metrics, prediction is considered "matched/detected" if its center is within 2m of its matched ground truth. We observe that average recall increases with more timesteps, meaning that more predictions are matched with ground truth
24

---

Published as a conference paper at ICLR 2023
Change in Metrics vs of Aggregated Timesteps for Movable Objects (Excluding Pedestrian) in Motion
1.5
MAP1
MATE
mAGE
mAVE
ARg5000
6.9
2.90
Tirk
6.23 21
20,1N
23.3%
697
635
110
Assroge
dfpppeggei Timatess
ofAipagsiiss Timanass
Afgreggiie Timalas
Aigrengniss Timatazi
staigapiies Tirasesi
Figure 16: Effects of long-term movable objects excluding pedestrian in motion.
and thus are considered for the poss-tssiive metric. These objects only detected with more timesteps are more difficult and potentially have poorer localization. Thus, although recall is higher, the true-positive metrics, especially mATE which there isn't a clear improvement in, are penalized Evaluating up to the recall level achieved with fewer timesteps, the increase in mATE is instead +0.8%, which although still not an improvement, is a more accurate picture of the impact of long- term temporal fusion on mATE for moving objects.
Second, comment that not only does average recall (AR @ 500) increase, mAP increases as well. This tells us that our proposed method isn't merely scattering a lot of low-confidence boxes, also meaningfully improves precision-recall. Intuitively, although longer-term fusion isn't 't able to precisely localize moving objects, it can at least better perceive and generally capture their loca- tion. Third, the better orientation and velocity estimates for moving object is very important for downstream tasks such as tracking, forecasting, and planning that rely on 3D detection. To verify this, beyond detection metries, we apply SOLOFusion on the tracking task with a simple greedy- matching algorithm (Yin et al. 2021). Although tracking is not the target of this paper, we achieve 0.540/0.94) AMOTA/AMOTP on the test set, ranking third place (first two entries use sophisticated tracking algorithms). We also believe that this method has huge potential in the field of forecasting and planning as velocity and orientation are the key elements for state estimation. We hope that SOLOFusion can serve strong 3D object detection baseline to be used for these downstream tasks.
Finally, looking at change in metrics for individual classes, we observe that pedestrian stands out in Figure [3 as demonstrating huge improvement. This is notable, as pedestrian a slow-moving class with average velocity 1.3 m/s. This tells that our long-term fusion module benefits slow-moving objects more than fast-moving ones. We verify that our pipeline does still benefit classes other than pedestrian by plotting change in metrics for other movable objects in motion in Figure [6] The more accurate velocity estimates generated by our pipeline can potentially be used to further improve predictions for moving objects, and we hope explore such directions in future work.
'We verified this with a toy example by generating two modified sets of predictions. For the first, we re- moved predictions for some scenes. For the second, we kept the predictions but randomized velocity predictions for those same scenes, representing scenario where more objects captured, but with worse attribute pre- dictions. The latter had significantly worse mAVE than the former using official evaluation code, expected.
25


Reviews:
[{"review_id": "sIRxV4wFU_", "reviewer": "Reviewer_U6zw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigate how temporal context should be used for vision-based 3D object detection. They treat temporal 3D object detection as a multi-view stereo problem, where history frames can be seen as multiple views. A criterion termed as \"localization potential\" is defined to reflect the level of difficluty for multi-view stereo, upon which they did in-depth analysis and concluded that using long-term, low-resolution temporal context benefits detection.\n\nBased on the analysis, a new 3D object detection method named SOLOFusion is proposed. The core idea is using both long-term and short-term memory. The long-term memory stores low-res features, and is fused by BEV cost volume; the short-term memory stores high-res features from the very last frame, and is fused  by plane-sweep cost volume. Experimental results show the proposed method can serve as a new strong baseline. Despite its simplicity, the results are quite impressive, refreshing previous state-of-the-art by a significant margin.", "summary_of_the_review": "In general, this is a good paper that worths acceptance. \nMy biggest concern is the presentation so I give my recommendation as \"6: marginally above the acceptance threshold\". If the presentation could be improved, I will be willing to raise my rating to \"8: accept, good paper\".", "strength_and_weaknesses": "Strength:\n+ This work borrows a lot of knowledge and tools from multi-view stereo to help us understand and analysis how temporal context should be used for vision-based 3D object detection. Introducing multi-view stereo to 3D object detection is a valuable direction that probably worth much more research, and to my knowledge this work can be seen as one of the few works that open up this direction.\n+ The analysis based on the proposed \"localization potential\" well-explained the motivation of using both long-term and short-term memory, and the conclusions shown in Fig.4-6 are quite interesting and inspiring.\n+ The final method is quite simple, easy to be re-produced.\n+ Experimental results are very promising,  refreshing previous state-of-the-art by a significant margin, showing the importance of using long-temporal context.\n+ Thorough abation study.\n\nWeaknesses:\n- I think the biggest issue is the presentation. I appreciate the dense technical content, which indeed brings difficulty in clarification; But the presentation could definitly be improved. For instance, a reader who has weak background on multi-view stereo may be confusing when discussing the relationship between multi-vew stereo and 3D object detection, since there is little preliminary knowledge introduced in the main text.  Maybe it would be better to elaborate more on background, and defer other stuff to the appendices.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qU5BkdYaAE", "reviewer": "Reviewer_FyEa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper makes a systematic study in terms of the critical factors in temporal stereo matching for camera-only 3D object detection and proposes corresponding solutions with a framework, SOLOFusion, to address the problems. It concludes that the limited history usage and low granularity of matching resolution are the two bottlenecks for previous methods, among which the former is more important and can compensate for the latter to some extent. The resulting solution leverages both low-resolution, long-term and high-resolution, short-term information to construct temporal multi-view features effectively and efficiently. It achieves a new SoTA on the nuScenes benchmark. The ablation studies also support the analysis.", "summary_of_the_review": "This paper formulates the problem of camera-only 3D detection from videos from a temporal multi-view stereo perspective and provides an analysis based on the \"localization potential\" concept. It results in critical conclusions that the usage of long-term histories and high-resolution features for matching are the most important factors in this setting and proposes corresponding solutions. Experiments show the effectiveness of the proposed approaches and demonstrate their importance in real-time practical use. The main problem is focused on the organization of contents in the main paper and supplemental materials, the incremental technical contribution for framework design, and the missing discussion about object motion. I would recommend weak acceptance at the current stage because it is an important supplement for this line of work and believe it can bring new insight to this community.", "strength_and_weaknesses": "Strengths:\n\n- The basic idea is easy to follow and the main motivation is clear.\n- The theoretical and empirical analyses are insightful. They can be important supplements for this line of works in the community.\n- The conclusion is clear and the paper proposes an efficient way to leverage both conclusions.\n- The proposed method achieves strong experiment results, both compared to its baseline and other SoTA methods.\n- Some empirical studies are interesting and important for practical use, such as the analysis about balancing temporal fusion and resolution in Table 7.\n\nWeaknesses:\n\n- The analysis part is a little wordy (although I understand it is fruitful, there is no need to clarify such simple conclusions with so many paragraphs) while the methodology part is so brief, even without a figure showing the details of networks. The re-organization of these contents in the main paper and supplemental materials should be considered.\n- The technical contribution seems incremental (though it may be related to the presentation problems). The framework seems a simple combination of BEVDepth and BEVStereo/DfM while only adjusting some settings for temporal aggregation. The author should clarify this point more clearly, and add at least a specific figure showing the network architecture as well as the distinguished technical design details.\n- As the involved timestep increases, another problem is that the effect of object motion on the stereo matching is more notable. Analysis for this point is missing in this paper. Such problems can severely and intrinsically challenge the conclusion that \"long-term matching\" is better.\n- (Minor) It would be better to have more references at:\n1. In the first paragraph and the conclusion section, the author mentions that \"depth estimation is the main bottleneck of camera-only works\". Although it can be common sense for researchers from this community, it would be better to include some references to support such claims.\n2. In Table 1, it would be better to compare SOLOFusion with a missing reference about the discussion of long-range, \"UniFormer: Unified Multi-view Fusion Transformer for Spatial-Temporal Representation in Bird's-Eye-View, Arxiv 2022\". Besides, DfM is missing in the MVS part and there is another line of work that is perpendicular to LSS-based, like OFTNet, ImVoxelNet, and MV-FCOS3D++ (which do not predict depth probability for 2D-3D lifting, among them only the last one has temporal modeling).\n- (Minor) Minor typos/grammatical mistakes:\n1. Footnote1, \"because it while it is\"\n2. The caption of Figure 4 seems to be covered by Figure 5.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "Avk9yXWt6Vd", "reviewer": "Reviewer_Bu5e", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study vision only 3D object detection for autonomous driving. First, the authors formulate the recent multi-frame multi-view methods as temporal stereo matching. Then localisation potential is proposed and analysed theoretically and empirically to show the necessity of larger time window. To handle the efficiency of long term computation, coarser resolution feature maps are used in the long term and finer resolution in the short term. In the experiments, the proposed method outperforms others in nuScenes leaderboard.", "summary_of_the_review": "\nI would like to accept this paper. If the authors can provide more analysis for the method (for example, the question I have in the weakness), I tend to further increase my rating. ", "strength_and_weaknesses": "Strengths:\n\n1. New formulation of the exiting multi-view, multi-frame image based 3D detection methods.\n\n2. Use both theoretical and empirical analysis by introducing localisation potential, to unveil the importance of longer time horizon.\n\n3. Impressive performance in nuScenes leaderboard.\n\n4. Writing is clear and easy to follow.\n\nWeakness:\n\nI have one question regarding the experimental analysis. Did the authors analyse the performance improvement for objects with different speed? It is interesting to see the impact of long term fusion on static, slowly moving and fast moving objects. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-5%