Hindsight is 20/20: Leveraging Past Traversals to Aid 3D Perception
Authors: Yurong You
Katie Z Luo
Xiangyu Chen
Junan Chen
Wei-Lun Chao
Wen Sun
Bharath Hariharan
Mark Campbell
Kilian Q Weinberger
Venue: ICLR 2022 Poster (2022)
Keywords: 3D object detection
perception with historical context

Abstract: Self-driving cars must detect vehicles, pedestrians, and other trafﬁc participants accurately to operate safely. Small, far-away, or highly occluded objects are particularly challenging because there is limited information in the LiDAR point clouds for detecting them. To address this challenge, we leverage valuable information from the past: in particular, data collected in past traversals of the same scene. We posit that these past data, which are typically discarded, provide rich contextual information for disambiguating the above-mentioned challenging cases. To this end, we propose a novel end-to-end trainable Hindsight framework to extract this contextual information from past traversals and store it in an easy-to-query data structure, which can then be leveraged to aid future 3D object detection of the same scene. We show that this framework is compatible with most modern 3D detection architectures and can substantially improve their average precision on multiple autonomous driving datasets, most notably by more than 300% on the challenging cases. Our code is available at https://github.com/YurongYou/Hindsight.

Published as a conference paper at ICLR 2022
HINDSIGHT IS 20/20: LEVERAGING PAST TRAVER- SALS TO AID 3D PERCEPTION
Yurong You', Katie Z Luo', Xiangyu Chen', Junan Chen', Wei-Lun Chao², Wen Sun', Bharath Hariharan', Mark Campbell', and Kilian Q. Weinberger' 'Cornell University, Ithaca, NY
The Ohio State University, Columbus, OH
ABSTRACT
Self-driving cars must detect vehicles, pedestrians, and other traffic participants accurately operate safely. Small, far-away, or highly occluded objects are par- ticularly challenging because there is limited information in the LiDAR point clouds for detecting them. To address this challenge, we leverage valuable in- formation from the past: in particular, data collected in past traversals of the same scene. We posit that these past data, which are typically discarded, pro- vide rich contextual information for disambiguating the above-mentioned chal- lenging cases. To this end, we propose novel, end-to-end trainable HIND- SIGHT framework to extract this contextual information from past traversals and store in an casy-to-query data structure, which can then be leveraged to aid future 3D object detection of the same scene. We show that this framework is compatible with most modern 3D detection architectures and can substantially improve their average precision notably by more than 300% on the challenging cases. Our code is available at https:/ /gi wig.hu.........g com www.///ooooooooooooooooo Yow.//ooooooooooooooooooooooouuo Hindsight.
multiple autonomous driving datasets, most
INTRODUCTION
To drive safely, (semi-)autonomous vehicle needs to accurately detect and localize other partici- pants, such as cars, buses, cyclists, or pedestrians who might walk onto the road at any time. Such an object detection task is an extremely challenging 3D perception problem, especially when comes to small, highly occluded, or far-away objects. In spite of considerable progress on this task in the past years (Geiger et al., 2012; Grigorescu et al., 2020; Janai et al., 2020), we have arguably not yet reached the accuracy levels needed for safe operations in general (non-geo-fenced) settings.
One reason why 3D object detectors struggle that often there simply not enough information in the scene to make good call. For example, the best LiDAR sensors may only yield few tens of points on a small child s/he is some distance away. From this very limited data, the 3D object detector must decide if this is a pedestrian who may rush onto the road at any moment, or this is just a tree that can be ignored. Sophisticated machine learning models may bring all sorts of inductive biases bear (Lang et al., 2019; Zhou & Tuzel, 2018; Yan al., 2018; Shi et al., 2019; 2020a), but such few LiDAR points may fundamentally lack enough information to make an accurate decision (Kendall & Gal, 2017). What we need is additional information about the scene.
Current 3D object detectors treat every scene as completely novel and unknown - ignoring poten- tially valuable information from previous traversals of the same route. In fact, many of us drive through the same routes every day: to and from work, schools, shops, and friends. Even when we embark on completely new route, we are often following in the footsteps of other drivers who have gone through this very section of route before. In this paper we explore the following question:
If we collect and aggregate unlabeled LiDAR information over time, potentially across vehicles, can we utilize the past traversals of the same route to better detect cars, bicycles and pedestrians?
We provide an affirmative answer to the above question. Our intuition is as follows: while historical traversals through route will be different, i.e., will encounter different cars on the road and

---

Published as a conference paper at ICLR 2022
different pedestrians the crosswalks, taken together, these traversals yield wealth of informa- tion. For instance, they reveal where pedestrians, cars, and cyclists generally tend to be in the scene, and where stop sign some unknown background object is persistently present across traversals. With this contextual information at hand, a detector can better recognize, say, a pedestrian heavily occluded by cars on the roadside, since pedestrians have come and gone through these regions be- fore, and it is unlikely that they are persistent background objects. Thus, leveraging such contextual information could substantially improve the detection accuracy in these safety-critical scenarios.
Concretely, we formalize this insight by proposing a simple and efficient approach that uses past traversals of a scene to vastly improve perception performance. Off-line, we use neural networks to digest past traversals of the scene into a sparse, compact, geo-referenced representation that we call SQuaSH "Spatial-Quantized Sparse History features). While in operation, a self-driving car can then query the local SQuaSH context of every LiDAR point in the current scene, thus enriching the information available to perform recognition. This information can be added as features to any LiDAR- ReaaR- 3D object detector, and both the detector and the SQuaSH representation can be trained jointly without any additional supervision. The resulting detector is substantially more accurate, and in fact can become more accurate over time without any retraining more traversals of the scene are collected. This local development and improvement with unlabeled data is an important distinction compared other approaches, such using high definition maps which static and require extensive data collections and labeling.
We validate our approach on two large-scale, real-world self-driving datasets, Lyft Level 5 Percep- tion (Kesten et al., 2019) and the nuScenes Dataset (Caesar et al., 2020), with multiple representative modern object detection models (Lang et al., 2019; Yan et al., 2018; Zhou & Tuzel, 2018; Shi al., 2019; 2020a) under various settings and show consistent performance gains. Concretely, our contri- butions are three-fold:
1. We identify a rich trove of information in the form of hindsight from past traversals that can substantially boost 3D perception of challenging objects at no extra cost.
2. We propose a simple and efficient method to leverage such information without any extra labels, that can be incorporated into most modern 31) perception pipelines.
3. We evaluate our method 3D object detection exhaustively across two large real-world datasets, different object types, and multiple detection architectures and demonstrate re- markably consistent and significant improvements, especially on the challenging cases small, far-away objects- - by over 300%.
2 RELATED WORKS
3D object detection is one of the most important perception tasks for autonomous driving. Most existing algorithms take LiDAR sensory data as input, which provides accurate 3D point clouds of the surrounding environment. There are two popular branches of methods. The first is to voxelize LiDAR point clouds and use the resulting 3D-cube representation as input to 2D or 3D convolutional neural networks to infer 3D bounding boxes (Zhou& & Tuzel, 2018; Yan et al., 2018 8; Zhou et al., 2020; Li, 2017: Engelcke et al., 2017: Lang et al., 2019; Ku et al., 2018; Yang et al., 2018b; Liang et al., 2018: Chen et al., 2017: Shi et al., 2020a). The second is design neural network architectures explicitly for point cloud inputs (Qi et al., 201 8; 2017a;b; Shi et al., 2019; 2020b; Yang et al., 2020).
Our approach augments the input point cloud with features queried from the history and is agnostic to specific object detection pipelines as long as they can take point clouds with per-point features. In this work, we experiment with four representative, high perforrino 3D object detection models (subsection 4.1) and demonstrate significant and consistent improvements when using HINDSIGHT.
3D object detection in contexts. We propose to augment the raw point cloud captured at driving time by features queried from the previous traversals. This relates to recent body of literature that augment point clouds with extra information, e.g., high-definition (HD) maps (Yang et al., 201 8a; Ravi Kiran et al., 2018: Seif & Hu, 2016; Liang et al., 2020a) or semantic information from synchronized images (Chen et al., 2017: Qi et al., 2018; Ku et al., 2018: Liang et al., 2018; Wang et al., 2018: You et al., 2020; Vora et al., 2020). Our approach is orthogonal to and compatible with - these methods, but also differs from them in key ways. In contrast to HD Maps, our history features are learned without additional labels, can be updated easily with newly collected traversals,
2

---

Published as a conference paper at ICLR 2022
(d)
inferenze-lime Only
(a)
upntial--uunteed
Traversals
Pre-computation Ony
SQuaSH Training/ Pre-computation
Convolution Kernel f
Geo indexed
Query
SQuaSH Store
Voxel wwise Aggres Iggregation
Quary
@ Store
Kernel Size K
SQUaSH
Query from SQuaSH (b)
3D Detection w/ SQuash
Eantoyy (P))
Query from SQUaSH (b)
Model
Perception
Current Stan
Current Scan (Endowed)
(c)
Object Detection
Figure 1: Overall pipeline of HINDSIGHT. The pipeline divided into four parts: (a) SQuaSH computation; (b) endowing the LiDAR scan by querying from SQuaSH; (c) detection with the en- dowed scan; (d) pre-computing SQuaSH off-line for deployment. The coloring on points in the endowed scan (the bottom-center and -right figures) means we endow history features in Riivr to each of the points. Please refer to section 3 for definitions of the symbols. Best viewed in color.
and are not limited to certain pre-defined information types. In contrast to images, querying our pre-computed history features adds infinitesimal latency to the perception pipeline (Table 3); the history features can disambiguate hard cases even if they are faraway from the ego car, whereas the corresponding image patches can be very small and hard recognize.
Perception with historical context. There are works utilizing temporal scans short-term context for perception (Liang et al., 2020b; Huang et al., 2020; Yang et al., 2021). However, there very limited work on using long-term historical context for perception. A few works use videos of people interacting with a scene to discover the affordances
the scene, c.g., where people sit, stand, or walk (Delaitre et al., 2012; 2; Fouhey al., 2012: Sun al., 2020). Closer in spirit to our work is the work by Barnes et al. (2018), which uses multiple traversals of a scene to map out regions that are ephemeral, namely, that represent transient objects. In these works, past observations are used to make specific, hand-crafted measurements. In contrast, our key idea is to automatically learn what relevant from the history by training the SQuaSH representation end-to-end with the detector.
3 HINDSIGHT
Problem setup. We assume operating in typical self-driving system: the autonomous vehi- cle is equipped with various synchronized sensors including LiDAR, which senses the surrounding scene by capturing 3D point cloud, and GPS/INS, which provides accurate 6-Degree-of- -oFereo (6-DoF, 3D displacement + 3D rotation) localization. The self-driving system reads the sensor in- puts and detects other participants (cars, pedestrians and cyclists) in the scene. Most tor-performing detection algorithms heavily rely on LiDAR scans for accurate detection. We denote the LiDAR point cloud captured at the current location during driving as P. {( denote the corresponding 6-DoF localization Ge. We assume that transformed by G., Pa is in fixed global 3D coordinate system. Atypical this conventional setting, we assume we have access to unlabeled point cloud scans from past traversals through the same location.
4 and
Overview of our approach HINDSIGHT. In this work, we propose a novel approach, which we term HINDSIGHT, to endow the current scan P. with information from past traversals improve 3D detection (Figure 1). Given the current spatial location Ge, HINDSIGHT retrieves a set point clouds recorded by past traversals of the same location, encodes them by spatial featurizer, and aggregates them into a spatially-inder indexed feature grid (which we call SQuaSH, Figure (a)). Then, for each point in the current sean P.. HINDSIGHT queries the local neighborhood of that point in
3

---

Published as a conference paper at ICLR 2022
the feature grid, and linearly combines the features in this neighborhood to produce "history" feature vector (Figure 1 (b)). This feature vector is concatenated as additional channels into the representation of each 3D point, producing enriched with the past information, is then fed into a 3D object detector (Figure 1 (c)). The whole pipeline is trained end-to-end with detection labels annotated only on current seans.
endowed point cloud. The endowed point cloud,
While this process seems cone-mnnnning at first glance, many of the computations, especially those related to the past scans, can indeed be carried out off-line. Specifically, we deliberately separate the computation such that the feature grid SQuaSH from the past scans is pre-computed and stored in a geo-indexed manner (Figure (d)). While in operation, the self-driving system only needs to query the pre-computed SQuaSH to endow the current point cloud with little latency.
Below, we describe each of these components detail.
3.1 DENSE 3D DATA FROM PAST TRAVERSALS
We first describe the data we get from past traversals. We assume access to T 2 past traversals; to ensure the information is up-to-date, we maintain the most recent Tmax traversals. A single traversal I consists of sequence of point clouds (Pj) and the associated global localization (Gt) recorded as the car was driven; here f is the index of the frame, We transform each point cloud into a fixed global coordinate system. Then, at location every metres along the road, we combine the point clouds from range | H., He] to produce dense point cloud S = UGGeB-HH..HH. (Pj} (where with a slight abuse of notation, we have used Cj for the 3D location where f was captured).
Combining multiple scans in this way can densify the scene, providing us with detailed and high- resolution point clouds for regions faraway from the current position. A static bush in distance can yield very sparse LiDAR points similar to that of a car, causing a false positive detection. However, can be disambiguated with multiple past scans that have dense points, recorded when the car was much closer to the bush in past traversals. These dense scans thus provide context for our method to "see" further into the distance or around occlusions.
3.2 SPANTIL- SPARSE HISTORY FEATURES (SQUASH)
Next we take each dense point cloud S from each traversal t and encode it using spatial featurizer that yields
-uaaniize feature tensor Qi In dense form, Qi can be viewed as a 4D tensor in R D danare, where H. W. and D are the nominal 3D spatial dimensions and duistory the feature dimension. The quantization step size is 8, so the point located at (z, 2) will be encoded at ([0/ō), [y/5], (=/ Any number of architectures can used to map point clouds to such voxelized tensors. In this work, we adopt the Sparse Residual U-Net (SR-UNet) (Choy et al., 2019) as the spatial featurizer due its high efficiency and performance. Additionally, because most parts of the scene are unoccupied, we represent Qf as sparse tensor.
For every location we compute T such tensors, one for each traversal. We aggregate them into a single tensor Q by applying per-voxel aggregation function sagg:
Q Sage (Q) Qf)
There are a number of available choices for the aggregation function, e.g., max-pooling, mean- pooling, etc. We do not observe significant performance difference among them (possibly due to the small T ) and thus use max- max-pooling due to its simplicity. We term the aggregated tensor Q for location as the Spatial-Quantizr Sparse History tensor (SQuaSH tensor) for this location.
3.3 QUERY FROM SQUASH
Suppose now that the self-driving car captures new scan P. at location Ge, and suppose that the SQuaSH tensor at this location Q We endow each of the points in P. by querying their surrounding features in Qi To do so, we convolve Q with a x K x K filter 0, and look up each point p = (x, y, 2) from the current sean the resulting tensor after appropriate quantization:
Fi 0 Q%
(2)
fnistory (p) [12/11 Lu/6,, [=/8]],
(3)
4

---

Published as a conference paper at ICLR 2022
where fjistory (p) is the corresponding history feature for point p. In practice, this entire operation can be easily implemented by 3D sparse convolution layer in Minkowski Engine (Choy et al., 2019) and runs extremely fast on a GPÚ (Table 3).
3.4 3D DETECTION WITH SQUASH
Most modern LiDAR- detection algorithms are able to take this point cloud P. with per-point features, e.g., LiDAR intensity. To make use of these off-the-shelf, perffornfoninng detection algo- rithms without heavy modifications to their frameworks, we adopt the simplest way to incorporate the queried history features: attaching flustory (P.) as per-point features. Even with this simple strat- egy, we observe significant and consistent performance gain. Our entire pipeline is simply trained an end-to-end manner, including the spatial featurizer, the query filter 0 and the detection model, via minimizing the corresponding detection loss from the detection algorithms on a dataset.
3.5 DEPLOYMENT
For deployment purposes, the self-driving system generates the SQuaSH tensors Qi for all loca- tions in an off-line manner once the spatial featurizer is trained. The SQuaSH tensors can then be retrieved at run-time using a simple geo-location query. We intentionally disentangle the com- putation between the past traversals and the current scan so that such pre-computation is possible. This allows us to use large and complex spatial featurizers to process the past traversals without introducing latency into the on-board autonomous driving system. Moreover, these features can be stored efficiently, since the SQuaSH tensors are sparse and can be produced only once every s = 10 meters without losing performance (Appendix B). As mentioned before, obtaining history features from the retreived SQuaSH is extremely fast, adding little burden to on-board systems (Table 3).
3.6 IMPLEMENTATION DETAILS
Localization. With current localization technology, we can reliably achieve very high localiza- tion accuracy (e.g., 1-2 em-level accuracy with RTK! 10 em-level with Monte Carlo Localization scheme (Chong et al., 2013) as adopted in (Caesar et al., 2020)). We assume an accurate localiza- tion within least 30 em-level, and show our method is robust to such localization error (Table 6).
SR-UNet inplemented in Minkowski Engine (Choy et al., 2019) as the spatial featurizer. It is a 14-layer U-Net (Ronneberger et al., 2015) following the conventional ResNet (He et al., 2016) basic block design with sparse 3D conv-ldcconvolutions The SR-UNet was originally designed by Choy et al. (2019) and achieved significant improvement against previous methods on various challenging semantic segmentation benchmarks. In this work, we adopt this architecture to process each past traversal, and aggregate them by max-pooling at each quantized coordinate index into a SQuaSH tensor. In Table 4, we include an ablation study on the relation between the final detection performance and the complexity of the spatial featurizer.
Spatial featurizer. We
Slyper-parameters We set | H., He). the range of scans that are combined to produce dense point clouds, to be [0, 20) m since we experiment with frontal-view detection only, and we combine only one scan into the dense point cloud S every 5 within this range. When performing inference, we use the geographically closest possible SQuaSH tensor: we observe no performance drop if the available SQuaSH is within 5 m (Appendix B). We use default quantization size 5 = 0.3 m for all the experiments (except we use 0.4 m for PointPillars due to limited GPU memory in training), and conduct an ablation study training, we use T maxx = 5 available past traversals to compute the SQuaSH. The filter kernel size K is 5. The dimensionality of the history features dhistory is 64.
the quantization sizes in Table 3. Due to GPU memory limit in
4 EXPERIMENTS
4.1 SETUP
Dataset. We validate our approach, HINDSIGHT, on the Lyft Level 5 Perception Dataset (Kesten et al., 2019) (Lyft) and the nuScenes Dataset (Caesar et al., 2020). To the best of our knowledge,
hhttps://en woin.wike wikipedia..oggee /weal-aaa---- 1-alltt me_ nemaat ww.psiiiioniii
5

---

Published as a conference paper at ICLR 2022
Table 1: Detection performance of four detectors with and without HINDSIGHT on the Lyft Dataset. We break down AEV by depth ranges. "+ H" stands for methods with point clouds endowed with history features by HINDSIGHT; "A AP indicates the absolute gain. HINDSIGHT improves the baselines in all but one case; several challenging cases (i.e., far-away objects or pedestrians and cyclists), the exact gains are larger than 20% in AP. Corresponding AP3D results and results under other IoU metrics are included in Appendix A where we observe similar trend.
Car
Pedestrian
Cyclist
Method
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
PointPillars
77.5
67.6
42.8
64.4
40.2
22.1
2.1
17.2
53.7
21.3
1.6
29.5
PointPillars + H
78.6
72.0
51.7
68.9
65.5
49.1
17.0
43.6
62.3
46.3
5.0
46.6
A APBEV
+1.1
+4.4
48.9
+4.5
+25.3
+27.0
+14.9
+26.4
+8.6
+25.0
+3.4
+17.1
SECOND
76.2
66.2
40.6
62.9
45.4
27.2
6.7
22.1
74.0
33.6
1.8
38.9
SECOND + HI
76.9
72.3
52.3
68.7
55.6
45.9
26.2
42.1
68.2
44.1
12,4
49.1
A APBEY
+0.7
46.1
+17.7
45.8
+70.2
+18.7
+19.5
+20.0
558
+10.5
+0..6
+T0.2
PointRCNN
74.0
72.3
45.9
65.1
40.7
29.5
5.7
22.1
69.7
37.9
1.0
44.8
PointRCNN + H
75.2
77.5
56.2
70.7
61.8
53.1
19.3
43.7
69.7
47.3
7,2
53.0
A APREY
+1.2
45.2
+10.3
45.6
+21.1
423.6
+13.6
421.6
40.0
45.4
46.2
+8.2
PV-RCNN
77.3
71.9
43.2
65.8
38.5
16.6
4.8
14.9
63.2
38.2
3.9
37.0
PV-RCNN H
80.8
77.4
56.4
73.5
44.6
42.0
24.9
35.5
63.2
47.7
12.7
48.1
A APBEV
+3.5
45.5
+13.2
+7.7
+6.1
+25.4
+20.1
+20.6
+0.0
+9.5
+8.8
+11.1
these two datasets are the only two publicly available, large-scale autonomous driving datasets that have both annotated object bounding-bou box labels and multiple traversals with accurate localization. Since they are not originally designed for evaluating the efficacy of utilizing historical traversals, some samples (i.e.. point clouds at specific timestamps) in the datasets do not have past traversals. Thus, re-split the datasets that each training and test sample has at least 2 past traversals;
quences in training and test set are geographically disjoint. This results in 12,407/2,274 training/test samples in the Lyft dataset and 3,985/2,324 training/test samples in the nuScenes dataset.
To apply off-the-shelf 3D object detectors which are typically built to take KITTI data (Geiger et al., 2012), we convert the raw Lyft and nuScenes data into the KITTI format and only train and evaluate frontal-view, frame- frame-by-frame, by-frame, and LiDAR-onf, -nly detections. We use the roof LiDAR signal (40 or 64 beams in Lyft: 32 beams in nuScenes), and obtain the global 6-DoF localization and extrinsic transformations between LiDAR and GPS/IMU directly from the raw data.
Evaluation metric. We follow KITTI (Geiger et al., 2012) and evaluate object detection both in the sirdes-eye view (BEV) and in 3D for Car, Pedestrian, and Cyelist on the Lyft dataset (Cyelist includes Bicycle+ Motorcycle in the raw Lyft dataset), and Car and Pedestrian in the nuScenes dataset, since there are too few cyclist instances. We also follow the KITTI convention to report average precision (AP) with the intersection over union (IoU) thresholds at 0.7/0.5 for Car and 0.5/0.25 for Pedestrian and Cyclist. We denote AP for BEV and 3D by AP and AP3D- Since the Lyft and nuScenes datasets do not provide the official definition of object difficulty as that in KITTI, follow Wang al. (2020) evaluate the AP various depth ranges. In Tables 1 to 5 of the main paper, due to the limited space, we only present results with loU=0.7 for car objects and loU=0.5 for pedestrian and cyclist objects. Please refer to Appendix A for results on AP3D and other IoU metrics, where we observe a similar trend.
Base 3D object detectors. We adopt four representative, high-performing 3D object detectors: PointPillars (Lang et al., 2019), SECOND (VoxeINet) (Yan et al., 2018; Zhou & Tuzel, 2018), PoinRRCN (Shi al., 2019). and PV-RCNN (Shi et al., 2020a), as the base detector models to augment (and compare) with our HINDSIGHT framework. These models are all state-of-the- art LiDAR-based detectors and are distinet in architecture: pillar-wise (PointPillars) vs. voxel- wise (SECOND, PV-RCNN) vs. woint-wise (PointRCNN) features, and single-stage (PointPillars, SECOND) vs. multi-stage (PoinIRCNN, PV-RCNN). We use publicly available code from Open- PCDet (OpenPCDet, 2020) for all four models. We use most of the default haper-parameters tuned for KITTI, with the exception that we enlarge the perception range from 70m to 90m since Lyft provides labels beyond 70m and we reduce the number of training epochs by 1/4 Lyft- since its train set is about three times of the size of KITTI. We observe consistent results with such hyper- parameters. We use exactly the same parameess and detection model architectures for the

---

Published as a conference paper at ICLR 2022
Table Detection performance of two detectors with and without HINDSIGHT on the nuScenes Dataset. Please refer to Table for naming. HINDSIGHT improves the baselines in all but one case; the gain more pronounced on the challenging pedestrian objects.
Car
Pedestrian
Method
0-30
30-50
50-80
0-80
0-30
30-30
50-80
0-80
PointPillars
24.5
4.2
0.0
11.7
13.8
1.4
0.0
6.3
PointPillars H
26.6
6.1
0.1
13.5
21.8
3.5
0.0
11.1
A APauv
+2.1
+1.9
+0.1
+1.8
+8.0
+2.1
+0,0
+4.8
PointRCNN
25.8
5.1
0.8
13.0
23.5
1.2
0.0
10.7
PointRCNN + H
28.5
7.3
1.5
14.7
33,9
4.5
1.4
17.3
A APBEV
+2.7
+2.2
+0.7
+1.7
+10.4
+3.3
+1.4
+6.6
Detection performance with various quantization sizes with PointRCNN. We show APBEV / AD evaluated in 0-80m. "H-ō m" stands for HINDSIGHT with quantization size m. We also report the average SQuaSH storage size and the average latency of querying the SQuaSH per scene under different quantization sizes.
Table 3:
Method PointRCNN
Query Latency
SQuaSH Size
Car
Ped.
Cyc.
ms
0 MB
65.1 / 41.8
22.1 / 16.8
44.8 1../36.00
H-0.2 m
4.1 ms
55.0 MB
59.7 / 39.8
46.8/25.1
46.7 39.3
+ H-0.3 m
3.4 ms
32.3 MB
70.7 / 46.3
43.7 /3.44
53.0 46.8
+ H-0055
2.4 ms
15.3 MB
68.0 44.6
20.9
45.2 / 38.2
m-
1.8 ms
5,2 MB
61.3/36. /6.77
33.2/
42.1 /34.1
baseline and baseline + HINDSIGHT, except change in the input dimension of the first layer due to the endowed hindsight features. We experiment with all four models on Lyft, and PointPillars and PointRCNN on nuScenes, training diverged for the other two models with the above default hyperparameters. All models are trained with 4 NVIDIA 3090 GPUs.
4.2
EMPIRICAL RESULTS
4.2.
3D OBJECT DETECTION RESULTS
In Table and Table 2 we show the detection performance of the four representative 3D object detection models with and without the aid of HINDSIGHT two different datasets and various object types. (Please refer to Appendix A for corresponding AP3D results and results under other IoU metrics.) We observe that HINDSIGITT consistently improves the detection performance for all detectors in almost all cases. In particular, models with HINDSIGHT enjoy substantial improvement over baseline counterparts on mid- to far-range. small (pedestrian and cyclist) objects, frequently yielding accuracy gains of over 20 percentage points in AP. As also shown in Figure 2 (more in Appendix C), compared to base detectors, HINDSIGHT maintains significantly higher precision under the same recall for detection objects. We note that these are typically considered the hardest parts in LiDAR- R-baeed object detection, not only because small objects have much fewer training examples in these datasets, but also because small physical shapes and farther distance from the ego car inherently result in much sparser LiDAR signals. thus crucial to incorporate the historical prior of the scenes (captured by HINDSIGHT) for improved detection performance.
4.2.2 LATENCY AND STORAGE
Low latency is crucial for bn-board perception models since they need to make critical decisions real- time. Table 3, we examine the extra latency that HINDSIGHT introduces to the detection model. We observe exceptionally low latency from our query operation (average 3,4 ms for quantization size 0.3 m). It adds little burden to the real-time pipeline whose perception loop typically takes 100 ms (10 Hz).
In Table we also show the average uncompressed storage of one SQuaSH for a scene. Each SQuaSH roughly covers 90 80 range in BEV. The storage highly depends the quantization size; it takes 32.3 MB for the default quantization size 0.3 m. Though already small enough in light of current cheaper and cheaper storage technology, in practice, the autonomous driving system
7

---

Published as a conference paper at ICLR 2022
Table 4: Detection performance of different HINDSIGHT spatial featurizer with PointR- CNN. We show BEV 3D evaluated in 0- 80m. "+H (x)" stands for HINDSIGITT with spa- tial featurizer x.
Detection performance with vari- ous number of available past traversals with PointRCNN. We show A 3D evaluated in 0-80m. "+H (S x) stands for HINDSIGHTT with using S x past traversals.
Table 5:
Method
Car
Ped.
Cyc.
Method
Car
Ped.
Cyc.
PointRCNN
65.1/4188
16.8
44.8 36.0
PoimRRCNN
65.1 65.1/4188
22.1 / 16.8 44.8 36.0
+ H (Identity)
67.0/44.9 36,4/25.2
51.6743.7
+H (= 1)
66.87 44.9
31.4/22.8
46.3 38.5
H (FCN)
69.7
38.6/28.4
48.7 44.3
+H (5 2)
70.0 / 46.3
42.7
29.0
52.9 / 43.5
H (SR-UNet)
70.7 /4..3
43.7 33.4 33.0/46.8
+H (S
70.7 46.3
43.7 / 33.4
5 46.8
Table 6: Detection performance with simu-
1.0
lated localization error with PointRCNN. We
PRCNN PRCNN--Hindight
PRCNN
show APBEV APsd evaluated in 0-80m. "H Ao m" means adding a random localization er- ror of standard deviation o m in evaluation.
Presision
0.6
0.4
Method
Car
Ped.
Cyc.
PointRCNN
65.1741.8
22.17 16.8
44.8
1/88/36. 36.0
+ A0.0 m 70.7 46.3
/33.4
53.0
46.8
0.0
+ H A0.1 m 70.6/
43.0/
52.9
45,4
0.0
0.2
0.4
0.6
0.8
1.0
+ +H A0.2 m
70.3
42.7
52.5 /3..3
Recall
H A0.3
70.1
37.9
49.4/41.0 /410
Figure 2: Precision-recall curves for Pedestrian of PointRCNN with and without HINDSIGHT, under metric BEV IoU=0.5 in 0-80m.
A0.5 m
67.2/40.8
31.0
20.8
44.5 / 36.3
+ H m
60.9 33.8
16.7 / 12.3
30.1 24.2
can build the SQuaSH at least every 10 metres without losing performance (Appendix B) to have a storage-efficient engineering implementation.
4.2.3 ABLATION STUDY
Impact of the HINDSIGHT spatial featurizer. In Table 4 we study the relation between the com- plexity of HINDSIGHT spatial featurizer and the final detection performance. We compare three possible spatial featurizers with different complexities: Identity, which directly quantizes each his- torical traversal and uses nominal "I" as features in Q to indicate the non-emptiness; FCN, which is simple two-layer, fully convolutional network: SR-UNet, the model we as default. It is interesting to observe that by simply querying from the history occupancy tensor (Identity), HIND- SIGHT enjoys a reasonable performance boost; the detection model benefits even more with more complex feature backbones (FCN and SR-UNet). This suggests that even the simple presence or absence of objects in past historical traversals valuable signal. More sophisticated processing of the past traversals may also characterize exactly what objects were observed in the past, which is richer contextual information.
Impact of the amount of available past traversals. In Table 5 we study the effect of using dif- ferent numbers of past traversals during testing. We observe that with just one past traversal, the detection model can already benefit from the history features; with only two, the model can achieve performance close to using five traversals (our manually-selectee maximum). This result indicates that HINDSIGHT does not need a large number of past traversals to extract relevant information.
Impact of quantization sizes. The spatial quantization sizes of SQuaSH can affect the granularity and the receptive field of HINDSIGHT. With fixed spatial featurizers architecture, the smaller the quantization size is, the higher the granularity but the smaller the receptive field is. In Table 3, study how different quantization sizes affect the final detection performance. It can be seen that the performance gain is more pronounced with a proper quantization size (0.3m). When the quantization is too small, the model cannot capture sufficient surrounding priors and is susceptible to random errors in localization; when the quantization size is too large, HINDSIGHT loses its ability to differentiate points within one voxel and it becomes extra burden to the learning process.
Impact of localization error. HINDSIGHT relies on (accurate) localization to query the correct history features. Thus, it is important to investigate the robustness of HINDSIGHT against potential
8

---

Published as a conference paper at ICLR 2022
Cve
PoinERCNN
PointRCNN-Hindsight
PointRCNN
PointRCNN--iindiihht
Figure 3: Qualitative visualization of detection results. We visualize two randomly picked scenes the Lyft test set, and plot the detection results from the base detector (PointRCNN) with and without HINDSIGHT bounding boxes are shown in green, base and HINDSIGHT detections are shown in yellow and cyan. Detection classes and confidence scores attached. Zoom in for details. Best viewed in color.
LiDARs (upper) and images (bottom, not used in detection). Ground-truth
localization error. In Table 6, we take the model trained with ground-truth localization (directly obtained from the dataset, without correcting for already present, minor errors), and evaluate its performance with the presence simulated random localization error. We simulate the error by sampling unit vector in 3D space, scale by c (0,, d²), where the error magnitude, and apply to the test scan. We observe that HINDSIGHT robust to localization error, where it can still bring substantial improvement on detection performance for all noise less than a 0.3 (m). Such localization error is well within the reliable range of current localization technology. showing the superior real-world application of HINDSIGHT. We also test the model under the presence of random bearing eгror (please refer to Appendix D), where we observe similar robustness.
4.3 QUALITATIVE RESULTS
Detection performance. In Figure we visualize the detection results of PointRCNN and PointR- CNN+HINDSIGHT at two random scenes of the Lyft test set (please refer to Appendix F for more). We observe that the base detector tends to generate many false positives to ensure a good coverage of the true objects; our HINDSIGIIT has significantly fewer false positives while being able to detect true objects exhaustively (also illustrated by Figure 2). Specifically, for faraway objects (two distant pedestrians in the left scene) and small objects (the close cyclist in the left scene), the base detector cither completely misses them or misclassifies them, while HINDSIGHT works like a charm.
DISCUSSION
Limitations. Our approach relies on past traversals being informative current state scene. Therefore, sudden changes to the current scene (e.g.. snow-storm problems to the detector, since the scene observes will be inconsistent with the history. Note that such adverse conditions may impact traditional detectors well. Unfortunately, existing datasets are collected in relatively mild conditions (e.g., clear weather for Lyft and sunny, rainy, or cloudy conditions for nuScenes). As such, they do offer the opportunity test our approach under adverse weather or other abnormal changes to road conditions. We hope this work can facilitate future research on this topic and inspire the creation of more related diverse datasets.
accident) may pose
Conclusion. In this work, we explore a novel yet highly practical scenario for self-driving, where unlabeled LiDAR scans from previous traversals of the same scene are available. We propose HIND- SIGHT, which extracts the contextual information from multiple past traversals and aggregates them into a compact and geo-indexed SQuaSH. While the self-driving system in operation, SQuaSH can be used to enrich the current 3D scan and substantially improve 3D perception. The whole pipeline can be trained end-to-end with detection labels only. Once trained, the SQuaSH can be pre- computed off-line and queried on-line with little latency. We show on two large-scale, real-world self-driving datasets that HINDSIGHT consistently and substantially improves the detection perfor- mance of four performeroriin 3D detectors, taking With this work, we hope to inspire more research into exploring the rich trove of information from past traversals of the same scene.
step closer towards safe self-driving.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENT
This research is supported by grants from the National Science Foundation NSF (III-1611134, III-1526012, IIS- IIS-114900 149882, IIS-1724282, TRIPODS-1740822 IIS-2107077, DAC-2118240, OAC- 2112606 and IIS-2107161), the Office of Naval Research DOD (N00014-17-1-2179)) the DARPA Learning with Less Labels program (HR001 11800044), the Bill and Melinda Gates Foundation, the Cornell Center for Materials Research with funding from the NSF MRSEC program (DMR- 1719875), and SAP America.
ETHICS STATEMENT
This work data from past traversals of scene to substantially improve 3D perception, espe- cially for far-off or occluded objects. We believe that our approach will lead to safer self-driving cars. Our approach will also allow for self-driving cars be driven beyond tight geo-fenced areas where HD maps are available, thus making this technology more broadly accessible.
Because our approach stores data that is typically discarded, potential concern is the privacy of pedestrians, cars and cyclists on the road. However, we note that the data we store is heavily quan- tized (6 0.3m). featurized and only based on the sparse LiDAR point cloud. Therefore the stored representation is likely to contain only aggregate information about the presence or absence of cer- tain kinds of objects, and unlikely to have any identifying information.
REPRODUCIBILITY STATEMENT
In subsection 3.6 and subsection 4.1, we exhaustively list the hyper- spper-paramtter for our approach, the dataset details, evaluation metrics and the hyper-parametens for training 3D object detectors. Our code is available at https: : / thub. com/ www./o Hinds ght. As mentioned in subsection 4.1, since the Lyft and the nuScenes dataset are not originally designed for evaluating the efficacy utilizing historical traversals, we use custom split to have valid evaluation. We also re- lease the split in the link for future comparison.
REFERENCES
Dan Barnes, Will Maddern, Geoffrey Pascoe, and Ingmar Posner. Driven to distraction: Self- supervised distractor learning for robust monocular visual odometry in urban environments. In ICRA, pp. 1894-1900. IEEE, 2018. 3
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, pp. 11621-11631, 2020. 2,
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3D object detection network for autonomous driving. In CVPR, 2017.
Zhuang Jie Chong, Baoxing Qin, Tirthankar Bandyopadhyay, Marcelo H Ang, Emilio Frazzoli, and Daniela Rus. Synthetic 2d lidar for precise vehicle localization in 3d urban environment. In ICRA, pp. 1554-1559. IEEE, 2013.
Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. In CVPR, pp. 3075-3084, 2019. 4,
V. Delaitre, D. Fouhey, I. Laptev, J. Sivie, A. Gupta, and A. Efros. Scene semantics from long-term observation of people. In ECCV, 2012.
Martin Engelcke, Dushyant Rao, Dominic Zeng Wang. Chi Hay Tong, and Ingmar Posner. Vote3deep: Fast object detection in 3D point clouds using efficient convolutional neural networks. In ICRA, 2017. 2
David F. Fouhey, Vincent Delaitre, Abhinav Gupta, Alexei A. Efros, Ivan Laptev, and Josef Sivic. People watching: Human actions cue for single-view geometry. In ECCV, 2012. 3
10

---

Published as a conference paper at ICLR 2022
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012. 1,6
Sorin Grigorescu, Bogdan Trasnea, Tiberiu Cocias, and Gigel Macesanu. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics, 37(3):362-386 2020. 1
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, pp. 770-778, 2016. 5
Rui Huang, Wanyue Zhang, Abhijit Kundu, Caroline Pantofaru, David A Ross, Thomas Funkhouser, and Alireza Fathi. An Istm approach to temporal 3d object detection in lidar point clouds. In ECCV, pp. 266-282. Springer, 2020. 3
Joel Janai, Fatma Güneyy, Aseem Behl, Andreas Geiger, et al. Computer vision for autonomous vehicles: Problems, datasets and state of the art. Foundations and Trends@ in Computer Graphics and Vision, 12(1-3))1-308 2020. 1
Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? NeurlPS, 2017.
R. Kesten, M. Usman, Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and
V. Shet. Lyft level av dataset 2019. https:///eveee .lyft. com dataset/. 2019. 2,
Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven Waslander. Joint 3D pro- posal generation and object detection from view aggregation. In IROS, 2018. 2
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. ICCV, pp. 12697-12705, 2019. 2,
Bo Li. 3D fully convolutional network for vehicle detection in point cloud. In IROS, 2017. 2
Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun. Deep continuous fusion for multi- sensor 3D object detection. In ECCV, 2018.
Ming Liang, Bin Yang, Rui Hu, Yun Chen, Renjie Liao, Song Feng, and Raquel Urtasun. Learning lane graph representations for motion forecasting, 2020a. 2
Ming Liang, Bin Yang, Wenyuan Zeng, Yun Chen, Rui Hu, Sergio Casas, and Raquel Urtasun. Pnpnet: End-to-end perception and prediction with tracking in the loop. In CVPR, pp. 11553- 11562, 2020b. 3
Development Team OpenPCDet. Openpedet: An open-source toolbox for 3d object detection from point clouds. https: //ghthub.bb/////// com/ opennoope---eeee mm1 /wwpa///pppp..e..e.cc 2020. 6
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets
for 3d classification and segmentation. In CVPR, 2017a. 2
Charles Qi, Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas, Frustum pointnets for 3d
object detection from rgb-d data. In CVPR, 2018. 2
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS, 2017b.
Ravi Kiran, Luis Roldao, Benat Trastorza, Renzo Verastegui, Sebastian Suss, Senthil Yogamani, Victor Talpaert, Alexandre Lepoutre, and Guillaume Trehard. Real-time dynamic object detection for autonomous driving using prior 3d-maps. In ECCV Workshops, September 2018.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015. 5
Heiko G. Seif and Xiaolong Hu. Autonomous driving in the icity hd maps as a key challenge of the automotive industry. Engineering, 2:159-162, 2016. 2

---

Published as a conference paper at ICLR 2022
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. In CVPR, 2019. 2,
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. Pv-rcnn: Point-voxel feature set abstraction for 3d object detection. In CVPR, pp. 10529-10538, 2020a. 1, 6
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. TPAMI, 2020b.
Jin Sun, Hadar Averbuch-Elor, Qianqian Wang, and Noah Snavely. Hidden footprints: Learning contextual walkability from 3d human trails. ECCV, August 2020.
Sourabh Vora, Alex H Lang. Bassam Helou, and Oscar Beijbom. Pointpainting: Sequential fusion for 3d object detection. In CVPR, pp. 4604-4612, 2020. 2
Yan Wang, Xiangyu Chen, Yurong You, Li Erran Li, Bharath Hariharan, Mark Campbell, Kilian Q- Weinberger, and Wei-Lun Chao. Train in germany, test in the usa: Making 3d object detectors generalize. In CVPR, pp. 11713-11723, June 2020. 6
Zining Wang, Wei Zhan, and Masayoshi Tomizuka. Fusing bird's eye view lidar point cloud and front view camera image for 3d object detection. In IVS, pp. 1-6. IEEE, 2018. 2
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10):3337, 2018. 1,2, 6
Bin Yang, Ming Liang, and Raquel Urtasun. Hdnet: Exploiting hd maps for 3D object detection. In CoRL, 2018a.
Bin Yang. Wenjie Luo, and Raquel Urtasun. Pixor: Real-time 3d object detection from point clouds. In CVPR, 2018b.
Zetong Yang, Yanan Sun, Shu Liu, and Jiaya Jia. 3dssd: Point-based 3d single stage object detector. In CVPR, pp. 11040-11048, 2020. 2
Zetong Yang, Yin Zhou, Zhifeng Chen, and Jiquan Ngiam. 3d-man: 3d multi-frame attention net- work for object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1863-1872, 2021.
Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q. Weinberger. Pseudo-lidart+: Accurate depth for 3d object detection in autonomous driving. In ICLR, 2020. 2
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection.
In CVPR, 2018. 2,
Yin Zhou, Pei Sun, Yu Zhang, Dragomir Anguelov, Jiyang Gao, Tom Ouyang, James Guo, Jiquan Ngiam, and Vijay Vasudevan. End-to-end multi-view fusion for 3D object detection in lidar point clouds. In CoRL, 2020. 2
12

---

Published as a conference paper at ICLR 2022
Appendix
We provide details omitted in the main text.
Appendix A: detailed evaluation results on the Lyft and the nuScenes dataset. (subsubsec-
tion 4.2.1 of the main paper, same naming as in Table and Table 2)
Appendix B: ablation on using pre-computed SQuaSH from a distance.
Appendix C: more precision-recall curves of detection results (as in Figure 2).
Appendix D: evaluation under random bearing error.
Appendix E: evaluation results of Table 3 under different metrics.
Appendix F: more qualitative visualization (as in subsection 4.3).
A MORE EVALUATION RESULTS ON THE LYFT DATASET AND THE NUSCENES DATASET
In Table and Table of the main paper, we present evaluation results across various depth ranges on APBEV with 1oU=0.7 for car objects and loU=0)5 for pedestrian and cyclist objects, on both the Lyft and the nuScenes dataset. In Tables to 12, we present the evaluation results on APBEV and APaD with two sets of IoU metrics (IoU=0.7 for car objects and IoU=0.5 for pedestrian and cyclist objects; loU=0.5 for car objects and 1oU=0.25 for pedestrian and cyclist objects).
Car
Pedestrian
Cyclist
Method
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
0.30
30-50
50-80
0-80
PointPillars
85.9
77.4
51.0
74.0
47.1
27.0
3.4
21.0
64.1
29.5
3.9
38.0
PointPillars +
87.9
83.1
64.4
80.2
73.9
60.0
21.5
52.4
66.5
49.8
11.9
52.9
A AP
+2.0
45.7
413.4
46.2
+26.8
433.0
F18.T
431.4
72.4
420.3
48.0
+14.9"
SECOND
87.3
75.8
47.9
72.4
58.2
36.5
9.9
29.8
81.4
44.7
7.1
47.7
SECOND + H
88.0
83.2
66.2
80.8
72.6
61.1
37.3
56.3
77.6
53.7
24.4
59.6
A APREV
+0.7
F77
+18.3
78.4
+I4.4
724.6
727.4
726.
338
79.0
7173
+I119"
PointRCNN
86.8
78.0
58.1
76.9
50.9
36.4
7.5
27.7
74.5
48.4
51.8
PointRCNN + H
86.9
83.1
72.2
82.8
79.5
63.3
26.2
54.7
74.1
54.2
13.5
59.5
A APPEE
+0.T
+5.7
+14.1
75.9
+28.6
+26.9
+18.7
727.0
0.4
75.8
+10.7
47.7
PV-RCNN
86.3
78.0
49.7
74.3
47.6
24.3
6.8
20.4
65.9
42,8
6.6
41.0
PV-RCNN + H
86.5
85.1
67.0
81.7
56.4
52.3
32.9
45.9
69.9
55.3
24.1
55.7
A APDEV
+0.2
+7.1
+17.3
+7.4
+8.8
+28.0
+26.1
+25.5
+4.0
+12.5
+17.5
+14.7
Table 7: HINDSIGHT detection results in the Lyft dataset, in APBEV with IoU=0.5 for car objects and loU=0.25 for pedestrian and cyclist objects.
Car
Pedestrian
Cyclist
Method
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
PointPillars
58.3
34.9
13.1
36.7
28.9
14.0
1.4
11.7
39.5
15.1
20.3
PointPillars + H
59.1
33.1
16.0
37.4
43.2
29.9
13.8
28.2
54.1
34.7
1.9
38.8
A APsEV
+0.3
-7.8
72.9
+0.7
+T4.3
+15.9
712.4
716.3
+14.6
+19.6
7T.77
+18.5"
SECOND
60.3
36.5
15.5
39.1
30.9
16.7
4.3
13.9
59.2
27.0
0.6
30.7
SECOND + H
60.3
39.3
16.4
41.0
36.1
31.0
19.5
27.5
61.8
29.7
3.3
37.1
A APPEV
+0.0
+2.8
+0.9
+1.9
+5.2
+14.3
F13.2
+13.6
+2.6
+2.7
+2.7
+6.4
PointRCNN
60.0
42.2
17.9
41.8
33,0
22.3
3,9
16.8
59.7
26.5
0.3
36.0
PointRCNN 1I
59.5
48.0
24.0
46.3
53.3
40.4
13.0
33.4
66.1
39.5
2.9
46.8
A APREV
-0.5
+5.8
+6.1
+4.5
+20.3
+18.1
+9.1
+16.6
+6.4
+13.0
+2.6
+10.8"
PV-RCNN
64.3
46.2
18.0
45.2
25.0
11.3
4.2
10.4
58.4
30.4
1.4
30.9
PV-RCNN +
67.1
46.1
21.4
46.8
33.2
29.8
17.7
25.7
53.8
32.3
34.9
A APWEV
+2.3
-0.1
43.4
+16
+8.2
+18.5
+:3.5
+15.
-4.6
41.9
44.T
"4.00"
Table 8: HINDSIGHT detection results the Lyft dataset, in AP with IoU=0.7 7 for car objects and loU=0.5 for pedestrian and cyclist objects.
13

---

Published as a conference paper at ICLR 2022
Car
Pedestrian
Cyclist
Method
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
PointPillars
85.0
73.5
46.0
70.6
47.1
26.9
3.4
21.0
64.1
29.2
3.8
37.8
PointPillars +
85.7
77.3
58.6
75.6
73.0
60.0
21.5
5188
66.5
49.8
11.2
52.0
A APREV
+0.7
+3.8
+12.6
+5.0
+25.9
+33.1
+18.1
+30.8
+2.4
+20.6
+7.4
+14.2
SECOND
85.6
72.7
43.8
70.1
$7.3
36.3
9.9
29.7
81.4
43.6
47.2
SECOND + H
85.9
80.2
59.0
76.5
72.3
61.1
37.3
56.2
76.3
53.1
22.8
58.6
A APDEV
+0.3
+7.5
+15.2
+6.4
+15.0
+24.8
427.4
+26.5
-5.1
49.5
+15.9
+1174
PointRCNN
86.7
75.9
53,9
74.7
50.2
35.5
7.5
27.4
74.5
48.4
1.8
51.4
PointRCNN + HI
86.8
82.7
64.9
80.1
78.0
62.9
26.2
54.6
74.1
54.3
13.3
59.5
A APREV
+0.I
+6.8
+1..0
75.4
+27.8
+27.4
+18.7
+27.2
-0.4
+5.9
+11.3
+8..
PV-RCNN
86.2
76.1
45.5
71.3
46.1
24.3
6.8
20.3
66.3
42.8
6.3
40.8
PV-RCNN +H
86.4
81.1
60.2
78.8
54.9
52.3
32,9
45.8
69.9
54.3
22.5
55.9
A APsEv
+0.2
+5.0
+14.7
+7.5
+8.8
+28.0
+26.1
+25.5
+3.6
+11.5
+16.2 +15.1
Table HINDSIGHT detection results the Lyft dataset, in AP3D with IoU=0.5 for car objects and IoU=0.25 for pedestrian and cyclist objects.
Car
Pedestrian
Method
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
PointPillars
31.9
6.9
0.1
16.0
19.4
3.1
0.0
9.7
PointPillars +
34.4
10.2
0.1
18.3
34.9
8.0
0.1
18.8
APBEY
+2.3
43.3
40.0
42.5
+75.5
44.9
+0.T
49.7
PointRCNN
29.0
7.1
1.5
15.6
31.8
3,4
0.3
15.4
PointRCNN +
33.9
10.9
18.8
47.5
10.2
3.0
A APREV
44.9
43.8
40.3
43.2
+15.7
46.8
42.7
410.3"
Table 10: HINDSIGHT detection results in the nuScenes dataset, in APREV with IoU=0.5 for car objects and IoU=0.25 for pedestrian objects.
Car
Pedestrian
Method
0-30
30-50
50-80
0-80
0-30
30-50)
50-80
(-80
PointPillars
11.5
0.7
4.8
10.8
0.0
PointPillars H
11.5
1.3
0.0
5.0
14.9
1.5
0.0
6,9
A AP
+0.0
40.6
40.0
40.2
74.1
40.9
+0.0
+2.2
PointRCNN
15.5
0.9
0.1
6.7
17.6
0.7
0.0
7.8
PointRCNN + H
16.3
2.0
0.1
7.5
26.1
2.9
0.4
13.2
A BEE
+0.8
+1.1
+0.0
+0.8
+8.5
+2.2
+0.4
+5.4
Table 11: HINDSIGHT detection results in the nuScenes dataset, in AP with IoU=0.7 for objects and loU=0.5 for pedestrian objects.
Car
Pedestrian
Method
0-30
30-50
50-80
0-80
0-30
30-50
50-80
0-80
PointPillars
28.7
4.9
0.0
13.6
18.8
2.8
0.0
9.3
PointPillars +
31.7
7.3
0.1
16.0
34.3
6.8
0.0
18.0
A APpLv
43.0
72,4
=6.1
-5.4
+15.5
74.0
+0.00
43.7
PointRCNN
28.2
5.5
1.1
14.0
31.3
3.1
0.3
15.3
PointRCNN
31.7
8.9
1.3
16.8
47.1
10.2
3.0
25.5
A AP
43.3
73.4
40.2
45.8
415.8
47.1
-5.7
4 T0.2"
Table 12:
HINDSIGHT detection results in the nuScenes dataset, AP with loU=0.5 for car objects and 1055 for pedestrian objects.
14

---

Published as a conference paper at ICLR 2022
Table 13: Detection performance using SQuaSH pre-computed various distance from the current ego car pose. We use base detector PointRCNN and report APBEV with loU=0.7 7 for car objects and loU=0.5 for pedestrian and cyclist objects. We do not observe significant performance difference among using SQuaSH with 0 m offset.
Method
Car
Ped.
Cyc.
PointRCNN
65.1/41.8
22.1/ 16.8
/
H-SQuaSH 0 offset
70.7 46.3
43.7//3.4
53.0/ 46.8
H-SSQuSSH 2.5 m offset (backward)
70.1/44.2
45.0/31.6
56.2
H-SSQuSHH 2.5 m offset (forward)
33.0/44.8
43.3/31.8
53.6/ 46.0
+H-SQuaSH 5.0 m offset (backward)
70.0/46.7
44.5/32.8
53.1/45.5
+H-SQuaSH 5.0 m offset (forward)
70.3
43.6/32.6
53.8
+ +H-SQuaSH 10.0 m offset (backward)
70.1
33.5
55.4/ 48.0
+H-SQuaSH 10.0 offset (forward)
70.2 46.3
36.5 29.1
29.1
52.9 /5.3
1.0
0.8
0.6
0.6
FRiNN--indsggt
PRCNN
0.2
0.2
PRONN
0.0
U.2
0.4
0.5
0.8
1.3
0.0
0.2
0.4
11.5
0.8
1.0
Recall
Recall
(a) Car, BEV 1oU=0.7, 0-80m
(b) Cyclist, BEV 1oU=0.5, 0-80m
PRCMN Hindsihht PRCMN
PRCNN Hinssghh: PRCMN
0.5
0.8
0.8
Peciion
0.6
FRCMMi-Hiniigh PRCNN
Procaton
Preckin
0.6
0.2
0.2
0.2
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.5
1.0
Recal
Recall
Rocal
(c) Car, 3D loU=0.7, 0-80m
(d) Pedestrian, 3D IoU=0.5, 0-80m
(c) Cyclist, 3D IoU=0.5, 0-80m
Figure 4: P-R curves for various objects of PointRCNN model with and without HINDSIGHT.
B SQUASH AT DISTANCE
For experiments in the main paper, we use the SQuaSH pre-computed at the closest possible location from the ego car position G. (i.c., the le is picked the closest possible one from the ego car). For practical use, the self-driving system should pre-compute the SQuaSH once every m to have storage-efficient implementation.
this case, the available SQuaSH can have s/2 m offset from the ego car localization. We evaluate this situation in Table 13, where we use the SQuaSHs that are 8/2 m away (either backward or forward) from the ego car localization. We observe no significant performance difference in s/2 0 5 m offset. It validates that in practice the self-driving system can pre-compute the SQuaSH at least every 10 m without losing performance gains.
C ADDITIONAL PRECISION-RECAL CURVES
In Figure 4, we show the detection mecision-rrecal curves of the base detector (PointRCNN) and the base detector with HINDSIGHT, on car, pedestrian and cyclist objects under various evaluation metrics. We observe similar trend as shown in Figure 2.
15

---

Published as a conference paper at ICLR 2022
Table 14: Detection performance with simu- lated bearing error with PointRCNN. We show BEV 3D evaluated in 0-80m. "H Ar-" means adding a random bearing emror of a stan- dard deviation o" in evaluation.
Table 15: Detection performance with various quantization sizes with PointRCNN, We show APBEV aD evaluated in 0-80m. Please refer to Table 3 for naming. We show the evaluation under Car: IoU 0.5, Ped Cyc: IoU 0.25.
Method
Car
Ped. Cyc.
Method PointRCNN
Car
Ped.
Cyc.
PointRCNN
65.1 41..8
22.1 16.8
44.8/ 36.0
76.9/747
27.7
51.4
+ H A0.0"
70.7 46.3
43.7 /3.4
53.07
46,8
+ m +1-0.3 m
50.07 78,0/72.8
55.9 55.7
Н 1
70.8 46,8
47.8/34.9
53.7 45.5
82.8 80.1
54.7 54.6
59.5 59,5
H. A0.5
70.1
48.2 4
+ m
51.2/5122 80.0/77.6
51.3/51.2
A1.0"
69.9/44.3
25.0/ 16.8
44.7
+H-1.0m
41.5/41.1 69.2
45.8/ 45.8
D ADDITIONAL BEARING ERROR EVALUATION
Typically, with current technology, the eπror on bearing angle is much smaller than that on localiza- tion (as small 0.08 degrees²). Nevertheless, we evaluate the performance under much harsher estimate of bearing rotation errors, and present detection evaluation results in Table 14. It can be seen that even under situations with large (potentially overestimated) bearing angle errors, HINDSIGHT models still significantly outperform the baseline model.
E ADDITIONAL EVALUATION RESULTS WITH DIFFERENT QUANTIZATION
We show the evaluation results of Table 3 under different metric (car object: IoU 0.5, pedestrian & cyclist object: IoU 0.25) in Table 15.
ADDITIONAL QUALITATIVE RESULTS
We show additional qualitative detection results in Figure
http:///wwaaaaaooooooooo.o.oooee waabblic//ovateee poovatel/assetee /assets/ /ospretttt///eeeeem/aaa/aat/tt/e
16

---

Published as a conference paper at ICLR 2022
OC
Ped
tar
PointRCNN
PointRNNN+Hindsight
Figure 5: Qualitative visualization of detection results. Please refer to Figure for coloring.
17


Reviews:
[{"review_id": "rTU2ZxN8NS", "reviewer": "Reviewer_TGjG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n  - [S1] The proposed technique seems to substantially and\n    consistently improve perception performance without the\n    need for any additional labels, or much additional\n    online compute. This has been shown on two datasets\n    with different LiDARs - nuScenes (32-beam) and Lyft L5\n    (64-beam).\n  - [S2] In-depth analysis and detailed discussion of\n    Hindsight's performance. The module is shown to help on\n    a wide range of base models - PointPillars, PointRCNN,\n    SECOND.\n  - [S3] The authors mention they plan to open source the\n    code upon paper acceptance, which would be very\n    beneficial to the research community.\n  - [S4] The writing is good and the main ideas of the\n    paper are presented clearly. For the few areas where\n    clarity could be improved I made corresponding notes in\n    the \"Suggestions\" section.\n  - [S5] Detailed ablation studies, including an assessment\n    of the impact of localization error on the approach\n    (i.e., what happens if the current LiDAR is misaligned\n    with the sweeps from the past?).\n\n\n## Limitations\n\n  - [L1] While HD maps can be more expensive than the\n    proposed approach, it is still necessary in my opinion\n    to compare against models which use them (I believe at\n    least nuScenes does come with HD maps). This can help\n    put the benefits of the current paper into perspective.\n    Moreover, HD map creation itself can be automated to a\n    substantial extent - things like BEV semantic maps,\n    lane boundaries, drivable regions, etc. can all be\n    computed without human intervention. Furthermore,\n    whether or not using HD maps is better than the\n    proposed approach would not weaken the paper. If HD\n    maps are better, then the proposed method is still\n    simpler and less expensive. If HD maps are not as good,\n    then that's one more pro for the current method.\n  - [L2] The proposed method requires prior passes through\n    the same area, even at validation time. Even though\n    these prior validation passes do not require labels, it\n    still means that the benefits of the model only come\n    after one, two, or more journeys have already been\n    completed through an area. (Note that this limitation\n    holds for HD maps as well.)\n  - [L3] (Minor) The proposed approach shares some of the\n    limitations of HD maps. For example, past observations\n    may become outdated as the world changes - previous\n    dangerous hidden driveways could disappear, but they\n    would still appear in the \"memories\" causing confusion,\n    potentially. However, the authors do touch on this in\n    the \"Limitations\" section.\n\n\n## Suggestions (Per-Section)\n\n### Introduction\n\n  - The setting is motivated overall very well, but I would\n    adjust the statement on \"Current 3D object detectors\"\n    to hint about HD maps. HD maps are discussed later in\n    the section, but I believe already mentioning them in\n    this paragraph can make the introduction a bit more\n    clear.\n\n\n### Method\n\n  - In Figure 1, adding colors to the spatial-quantized\n    feats and SQuaSH images could make it even clearer that\n    you are talking about feature point clouds, not just\n    the geometry. (It could be a different color scheme\n    from (c) to differentiate the offline features from the\n    endowed online point cloud.)\n  - Q1: What is the 'e' from $H_e$?\n\n\n### Experimental Results\n\n  - Thank you for providing brief details on the new split\n    as well as a recap of the metrics you are using. This\n    makes the paper more self-contained and easier to\n    follow.\n  - Q2: What is the size of the physical area stored? What\n    area does, e.g., the 55MB of SQuaSH in Table 3 cover?\n    How does this scale for the entire operational domain\n    of an SDV? It would be more informative in my opinion\n    to show something like SQuaSH size / km of road, so\n    that one may estimate the storage requirements for a\n    given city/neighborhood.\n  - Q3: Does the performance gap between H-0.2m and 0.3m\n    depend on the IoU threshold for the metric? I find it\n    odd that the difference for the car class is so large.\n    Showing this analysis for different IoU thresholds\n    could potentially help shed more light on the topic.\n  - Q4: What do you mean by the need to \"build SQuaSH\"\n    every K meters? Aren't the SQuaSH features essentially\n    a continuous (sparse) tensor covering the entire\n    operational domain? Can't this tensor simply be queried\n    using the online detector RoI at arbitrary positions?\n  - Q5: Do any of the evaluated detectors take multiple\n    LiDAR sweeps as input? If yes, in Section 4.2 is the\n    same localization offset applied to all sweeps?\n    (Localization errors tend to be highly correlated\n    temporally.)\n  - (Very Minor) The citation key for OpenPCDet is a bit\n    odd - is it possible to override it so that Bibtex does\n    not try to split \"OpenPCDet Team\" into a first and a\n    last name?\n\n## Video/Supplementary\n\n  The supplementary contains additional qualitative and\n  quantitative results such as $\\text{AP}_\\text{3D}$ metrics and\n  additional ablation studies.\n", "summary_of_the_paper": "  The paper tackles the task of object detection in LiDAR\n  point clouds, with a strong emphasis on automotive\n  applications.\n\n  The proposed approach is orthogonal to model, labeling,\n  or training improvements, and consists in retrieving\n  unlabeled geo-referenced past data from the area in which\n  the car is located at inference time in order to augment\n  the present observations. This is shown to have a very\n  significant impact in perception quality, especially for\n  smaller objects, such as pedestrians, and at longer\n  range.\n\n  Chunks of unlabeled data from the same geographic area as\n  the input frames are pre-processed by a sparse 3D fully\n  convolutional neural net, and the resulting features are\n  used as additional inputs to the primary object detector.\n\n  This pre-processor, which is called the \"hindsight\n  encoder\" net, is trained jointly with the detector, as it\n  is straightforward to backpropagate through the\n  concatenation of \"hindsight features\" with the original\n  point cloud. Since the hindsight features are essentially\n  just retrieved by indexing and concatenated with the\n  original input point cloud before it is fed into an\n  arbitrary object detector, they can be used and trained\n  in conjunction with a wide range of detector\n  architectures. Once training finishes, they can be\n  pre-computed offline to avoid inference-time overhead.\n\n  Given the method's reliance on past (unlabeled)\n  experiences through the area being evaluated, the authors\n  re-partition the datasets on which they evaluate in order\n  to match this setting (e.g., validation contains\n  unlabeled data which is just used to compute the\n  hindsight features for the labelled validation logs).\n  While this makes it a bit difficult to compare against\n  existing results, the authors run multiple baselines in\n  their setting and promise to release the split\n  information to the public. The proposed setting (lots of\n  unlabeled data from existing passes through the\n  \"inference\" area) is very common in the self-driving\n  industry.\n\n  The paper presents very promising results on Lyft and\n  nuScenes, especially for pedestrians, cyclists, and\n  distant objects (>50m) in general. The writing is clear\n  and most key design choices are evaluated using ablation\n  studies.\n", "summary_of_the_review": "  Apart from a few minor concerns, such as comparisons to\n  the use of \"traditional\" HD maps, I do not see any major\n  flaws with the proposed paper. The proposed technique is\n  orthogonal to improvements in architecture, data\n  augmentation, etc., and it seems to work in a wide range\n  of settings. The paper itself is well written and the\n  experimental evaluation is thorough.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "U0Gn1_bEW8l", "reviewer": "Reviewer_i3iC", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n# Pros:\n\n+ The overall idea is a good one. Leveraging these past point clouds, which would typically be thrown away, is sensible, and the algorithm is a good and simple approach to using these past traversals, which effectively give a kind of 'peak ahead and behind' into areas of the scene not currently seen well. As SLAM and relocalisation algorithms improve, the algorithm will be more and more suited to different scenarios.\n\n+ I like that fact that the authors used four different object detection algorithms to show the gains are independent of detection method used.\nExperiments are generally well thought out and well presented. A good set of experiments in different scenarios are shown, using two sensible datasets.\n\n* Including qualitative figures helps a little to understand the algorithms improvements.\n\n* The paper is very well presented, with clear figures and tables and nice typesettings\n\n* English is good and explanations are clear.\n\n* While related work feels a little short it gives a very clear overview of related works, categorising them nicely.\n\n\n# Cons:\n\n- My main concern with the paper is a lack of introspection. In the introduction, the authors say that past traversals \"...reveal where pedestrians, cars, and cyclists generally tend to be in the scene, and where a stop sign or some unknown background object is persistently present across traversals. With this contextual information at hand, a detector can better recognize, say, a pedestrian heavily occluded by cars on the roadside, since pedestrians have come and gone through these regions before, and it is unlikely that they are persistent background objects.\" But this is speculation; I was really hoping for (and expecting!) an experiment which confirmed this, shedding some light on what the network is actually learning, and how it is really helping improve scores. For example the 'pedestrian' class undergoes a surprisingly large increase (Table 5, Fig 2). \n\n- Some ideas for things that might help give introspection include:\n\n1) Looking at the split of improvements on moving vs static objects.\n\n2) Does detection rate only improve in regions where previous traversals actually contained instances of the object being detected? Or does the detection rate also improve where previous traversals contained no cars/pedestrians/cyclists?\n\n3) Some of the following experiments might also help:\n\n- I'd be keen to see an extra row in e.g. Table 5, which uses one 'past traversal', but where that past traversal is in fact just the entire current traversal. So the SQuaSH features would have access to information about the scene captured e.g. 5 or 10 seconds before and after the current scan P_c.\n\n- Similarly, I'd like to see an extra row in Table 5 where the 'past traversal' is a copy of the current scan *but only up to the time at which P_c was captured*.\n\n- It might not be easy, but: I'd have really liked to see a baseline which uses some kind of SOTA recurrent network to make use of the point clouds which immediately came before P_c temporally. And/or merging a short time history of pointclouds together into one single point cloud for input to the detection algorithms. (It feels a little unfair that the baseline only has access to point data at a single timepoint, when models already exist to make use of time series data). See e.g. the LSTM works I list below.\n\n- Related work feels a little short.  \n\n  - There are more 'change detection' works which could be worth mentioning. [City-scale Scene Change Detection using Point Clouds, ICRA 2021] is a recent example I found.\n\n  - In terms of \"Perception with historical context\" – it might be worth mentioning that *short-term* historical context is very often used where temporal data is present, e.g. in an LSTM-based model. It is just longer-term historical context which is more rarely used. For example: \n\n    - Liang et al. PnPNet: End-to-End Perception and Prediction with Tracking in the Loop, CVPR 2020\n    - Huang et al. An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds. ECCV 2020\n    - Yang et al. 3D-MAN: 3D Multi-frame Attention Network for Object Detection. CVPR 2021.\n\n", "summary_of_the_paper": " The paper proposes a method for improving object detection in lidar scans, which works in cases where a lidar scanner visits the same environment multiple times. The paper proposes to take point clouds from previous traversals and put them through a trainable network, before temporally aggregating features from each of the traverals. (It is assumed that camera pose is known for each scan). The aggregated features are then concatenated with the features from the current traversal, before being input into an off-the-shelf object detector. Results show that this method improves object detection rates in almost all cases, regardless of the object classification method used (the authors run experiments with four different object detectors).\n\n", "summary_of_the_review": "Overall I would like to see this paper in ICLR. \n\nIt is a nice idea, a well-written paper, and I would learn something from it. There are lots of results in the paper, but most of them are simply showing that the algorithm improves scores. I would really like to see a couple more experiments helping to show *why* the scores improve. I would also like to see some stronger baselines using point cloud data immediately preceding P_c as additional input.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "E_rKvpSCUw8", "reviewer": "Reviewer_iXT8", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a well-written paper and a good contribution to the field. AVs often repeat the same route over and over again and this work is a step in the direction of utilizing past information to improve object detection metrics along the route without any additional labeling and training requirement. This will save money, time and effort that would otherwise go into obtaining more data and manual labeling of objects along the route. There are good ablation studies done regarding the effect of localization error in terms of displacement, but what is the effect of bearing error in localization - and it would be good to have that as well. The choice of the Mikowski-net or SR-Unet with sparse 3D convolutions is obviously a significant contributor to the results. An obvious question that arises is how much would such a network contribute to the metrics if used as a backbone for the 3D object detection as well.\nThere is a typo on page 5: “deconvoluitons”.\n", "summary_of_the_paper": "This paper aggregates LIDAR scans from past traversals of an AV route to improve 3D object detection. Multiple scans from past traversals are aggregated into a voxellized representation using an accurate GPS-RTK localization system. Each scan is then passed through a state-of-the-art sparse 3D convolution network, SR-UNet, which generates a per-voxel feature that is stored as a representation for that part of the route. Such representations, called SQUASH, are stored periodically along the route, and added to the current LIDAR scan during inference as an additional input to a standard LIDAR based object detection network. The addition of these historical features improves object recognition results by ~20 average-precision points on NuScenes and Lyft Level 5 datasets, without any additional training. \n", "summary_of_the_review": "This is a well-written paper with a significant contribution towards using historical data from the route to aid in object detection along repeated routes. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "3Pv81b4dc2o", "reviewer": "Reviewer_9mnc", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n[1] The paper echoes with a few previous literature in the insight that previous observation may help scene understanding, which is intuitive and yet under explored. In this sense, the formulation of the motivation as well as the pipeline is novel, and successful experiment has proven its value to the community. In particular, the pipeline is especially useful for pedestrian detection, which is a difficult among all detection tasks, and the performance have somewhat saturated in recent years. With the additive nature of the proposed method, it may provide boost when combined with any state-of-the-art detection pipelines.\n\n[2] The model is simple yet effective. The pipeline takes an existing feature extraction model to acquire the offline feature of past traversals, but was able to prove the effectiveness of this simple strategy to summarize information from past observations. The addition of the history features to current scan is also simple, by simply fusion it with current feature, and keep the same detection framework.\n\n[3] The experiments are well carried out; in particular extensive ablative study is included on both the model design and results, which shows the pipeline is easy to integrate with existing detectors, and brings little overhead with noticeable improvement. \n\nWeakness:\n[1] As mentioned earlier, the pipeline is simple in that it reuses existing feature extractor and detector. However the paper did not includes more insights into the design choices in this part: (a) what if the feature extractor and the detector can be finetuned together? (2) how will be the offline history feature be updated when more data are observed? What will be the procedure and the cost?\n\n[2] Clarity. For example, in Sec. 3.1, how are the dense point cloud acquired from combining samples sparse ones? Is there an alignment step involved, or simply shift all points to center at the current location? Extra visualization/example on this would be helpful.\n\n[3] Results. To demonstrate how pedestrian detection benefits most, the paper should include more results and analysis on the pedestrian case. For example, how does improvement in pedestrian detection quantitatively compare to improvement in cars? Will there be any difference between moving people and standing ones, in that fusion will be worse for moving people.", "summary_of_the_paper": "The paper proposes a HINDSIGHT framework for object detection on lidar scans, with query of scene feature extracted offline from previous traversals. The pipeline is additive and complementary to any real-time detection pipeline, by introducing additional point-wise feature from offline feature structure to point feature of the current scan. The paper is able to report improved performance on standard detection tasks, and in particular boost on pedestrian detection where past observation can help differentiate transient objects. Complete ablation study is included; the experiments are well-designed and described.", "summary_of_the_review": "The major strength of the paper is the formulation of the problem and the insights it brings to this classic task of object detection in autonomous driving domain. The models are well designed, experiments are complete and results are convincing. Although more results/ analysis can be added, the paper is self-contained in the current stage.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a framework for object detection on lidar scans, with query of scene feature extracted offline from previous traversals. Overall there is good agreement among reviewers, with three recommending accepting the paper and one marginally accepting it -- to me the authors satisfactorily addressed most aspect raised in reviewing.
Decision: Accept (Poster)