Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a HINDSIGHT framework for object detection on lidar scans, with query of scene feature extracted offline from previous traversals. The pipeline is additive and complementary to any real-time detection pipeline, by introducing additional point-wise feature from offline feature structure to point feature of the current scan. The paper is able to report improved performance on standard detection tasks, and in particular boost on pedestrian detection where past observation can help differentiate transient objects. Complete ablation study is included; the experiments are well-designed and described.
main_review: Strength:
[1] The paper echoes with a few previous literature in the insight that previous observation may help scene understanding, which is intuitive and yet under explored. In this sense, the formulation of the motivation as well as the pipeline is novel, and successful experiment has proven its value to the community. In particular, the pipeline is especially useful for pedestrian detection, which is a difficult among all detection tasks, and the performance have somewhat saturated in recent years. With the additive nature of the proposed method, it may provide boost when combined with any state-of-the-art detection pipelines.

[2] The model is simple yet effective. The pipeline takes an existing feature extraction model to acquire the offline feature of past traversals, but was able to prove the effectiveness of this simple strategy to summarize information from past observations. The addition of the history features to current scan is also simple, by simply fusion it with current feature, and keep the same detection framework.

[3] The experiments are well carried out; in particular extensive ablative study is included on both the model design and results, which shows the pipeline is easy to integrate with existing detectors, and brings little overhead with noticeable improvement. 

Weakness:
[1] As mentioned earlier, the pipeline is simple in that it reuses existing feature extractor and detector. However the paper did not includes more insights into the design choices in this part: (a) what if the feature extractor and the detector can be finetuned together? (2) how will be the offline history feature be updated when more data are observed? What will be the procedure and the cost?

[2] Clarity. For example, in Sec. 3.1, how are the dense point cloud acquired from combining samples sparse ones? Is there an alignment step involved, or simply shift all points to center at the current location? Extra visualization/example on this would be helpful.

[3] Results. To demonstrate how pedestrian detection benefits most, the paper should include more results and analysis on the pedestrian case. For example, how does improvement in pedestrian detection quantitatively compare to improvement in cars? Will there be any difference between moving people and standing ones, in that fusion will be worse for moving people.
summary_of_the_review: The major strength of the paper is the formulation of the problem and the insights it brings to this classic task of object detection in autonomous driving domain. The models are well designed, experiments are complete and results are convincing. Although more results/ analysis can be added, the paper is self-contained in the current stage.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper aggregates LIDAR scans from past traversals of an AV route to improve 3D object detection. Multiple scans from past traversals are aggregated into a voxellized representation using an accurate GPS-RTK localization system. Each scan is then passed through a state-of-the-art sparse 3D convolution network, SR-UNet, which generates a per-voxel feature that is stored as a representation for that part of the route. Such representations, called SQUASH, are stored periodically along the route, and added to the current LIDAR scan during inference as an additional input to a standard LIDAR based object detection network. The addition of these historical features improves object recognition results by ~20 average-precision points on NuScenes and Lyft Level 5 datasets, without any additional training. 

main_review: This is a well-written paper and a good contribution to the field. AVs often repeat the same route over and over again and this work is a step in the direction of utilizing past information to improve object detection metrics along the route without any additional labeling and training requirement. This will save money, time and effort that would otherwise go into obtaining more data and manual labeling of objects along the route. There are good ablation studies done regarding the effect of localization error in terms of displacement, but what is the effect of bearing error in localization - and it would be good to have that as well. The choice of the Mikowski-net or SR-Unet with sparse 3D convolutions is obviously a significant contributor to the results. An obvious question that arises is how much would such a network contribute to the metrics if used as a backbone for the 3D object detection as well.
There is a typo on page 5: “deconvoluitons”.

summary_of_the_review: This is a well-written paper with a significant contribution towards using historical data from the route to aid in object detection along repeated routes.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper:   The paper tackles the task of object detection in LiDAR
  point clouds, with a strong emphasis on automotive
  applications.

  The proposed approach is orthogonal to model, labeling,
  or training improvements, and consists in retrieving
  unlabeled geo-referenced past data from the area in which
  the car is located at inference time in order to augment
  the present observations. This is shown to have a very
  significant impact in perception quality, especially for
  smaller objects, such as pedestrians, and at longer
  range.

  Chunks of unlabeled data from the same geographic area as
  the input frames are pre-processed by a sparse 3D fully
  convolutional neural net, and the resulting features are
  used as additional inputs to the primary object detector.

  This pre-processor, which is called the "hindsight
  encoder" net, is trained jointly with the detector, as it
  is straightforward to backpropagate through the
  concatenation of "hindsight features" with the original
  point cloud. Since the hindsight features are essentially
  just retrieved by indexing and concatenated with the
  original input point cloud before it is fed into an
  arbitrary object detector, they can be used and trained
  in conjunction with a wide range of detector
  architectures. Once training finishes, they can be
  pre-computed offline to avoid inference-time overhead.

  Given the method's reliance on past (unlabeled)
  experiences through the area being evaluated, the authors
  re-partition the datasets on which they evaluate in order
  to match this setting (e.g., validation contains
  unlabeled data which is just used to compute the
  hindsight features for the labelled validation logs).
  While this makes it a bit difficult to compare against
  existing results, the authors run multiple baselines in
  their setting and promise to release the split
  information to the public. The proposed setting (lots of
  unlabeled data from existing passes through the
  "inference" area) is very common in the self-driving
  industry.

  The paper presents very promising results on Lyft and
  nuScenes, especially for pedestrians, cyclists, and
  distant objects (>50m) in general. The writing is clear
  and most key design choices are evaluated using ablation
  studies.

main_review: ## Strengths

  - [S1] The proposed technique seems to substantially and
    consistently improve perception performance without the
    need for any additional labels, or much additional
    online compute. This has been shown on two datasets
    with different LiDARs - nuScenes (32-beam) and Lyft L5
    (64-beam).
  - [S2] In-depth analysis and detailed discussion of
    Hindsight's performance. The module is shown to help on
    a wide range of base models - PointPillars, PointRCNN,
    SECOND.
  - [S3] The authors mention they plan to open source the
    code upon paper acceptance, which would be very
    beneficial to the research community.
  - [S4] The writing is good and the main ideas of the
    paper are presented clearly. For the few areas where
    clarity could be improved I made corresponding notes in
    the "Suggestions" section.
  - [S5] Detailed ablation studies, including an assessment
    of the impact of localization error on the approach
    (i.e., what happens if the current LiDAR is misaligned
    with the sweeps from the past?).


## Limitations

  - [L1] While HD maps can be more expensive than the
    proposed approach, it is still necessary in my opinion
    to compare against models which use them (I believe at
    least nuScenes does come with HD maps). This can help
    put the benefits of the current paper into perspective.
    Moreover, HD map creation itself can be automated to a
    substantial extent - things like BEV semantic maps,
    lane boundaries, drivable regions, etc. can all be
    computed without human intervention. Furthermore,
    whether or not using HD maps is better than the
    proposed approach would not weaken the paper. If HD
    maps are better, then the proposed method is still
    simpler and less expensive. If HD maps are not as good,
    then that's one more pro for the current method.
  - [L2] The proposed method requires prior passes through
    the same area, even at validation time. Even though
    these prior validation passes do not require labels, it
    still means that the benefits of the model only come
    after one, two, or more journeys have already been
    completed through an area. (Note that this limitation
    holds for HD maps as well.)
  - [L3] (Minor) The proposed approach shares some of the
    limitations of HD maps. For example, past observations
    may become outdated as the world changes - previous
    dangerous hidden driveways could disappear, but they
    would still appear in the "memories" causing confusion,
    potentially. However, the authors do touch on this in
    the "Limitations" section.


## Suggestions (Per-Section)

### Introduction

  - The setting is motivated overall very well, but I would
    adjust the statement on "Current 3D object detectors"
    to hint about HD maps. HD maps are discussed later in
    the section, but I believe already mentioning them in
    this paragraph can make the introduction a bit more
    clear.


### Method

  - In Figure 1, adding colors to the spatial-quantized
    feats and SQuaSH images could make it even clearer that
    you are talking about feature point clouds, not just
    the geometry. (It could be a different color scheme
    from (c) to differentiate the offline features from the
    endowed online point cloud.)
  - Q1: What is the 'e' from $H_e$?


### Experimental Results

  - Thank you for providing brief details on the new split
    as well as a recap of the metrics you are using. This
    makes the paper more self-contained and easier to
    follow.
  - Q2: What is the size of the physical area stored? What
    area does, e.g., the 55MB of SQuaSH in Table 3 cover?
    How does this scale for the entire operational domain
    of an SDV? It would be more informative in my opinion
    to show something like SQuaSH size / km of road, so
    that one may estimate the storage requirements for a
    given city/neighborhood.
  - Q3: Does the performance gap between H-0.2m and 0.3m
    depend on the IoU threshold for the metric? I find it
    odd that the difference for the car class is so large.
    Showing this analysis for different IoU thresholds
    could potentially help shed more light on the topic.
  - Q4: What do you mean by the need to "build SQuaSH"
    every K meters? Aren't the SQuaSH features essentially
    a continuous (sparse) tensor covering the entire
    operational domain? Can't this tensor simply be queried
    using the online detector RoI at arbitrary positions?
  - Q5: Do any of the evaluated detectors take multiple
    LiDAR sweeps as input? If yes, in Section 4.2 is the
    same localization offset applied to all sweeps?
    (Localization errors tend to be highly correlated
    temporally.)
  - (Very Minor) The citation key for OpenPCDet is a bit
    odd - is it possible to override it so that Bibtex does
    not try to split "OpenPCDet Team" into a first and a
    last name?

## Video/Supplementary

  The supplementary contains additional qualitative and
  quantitative results such as $\text{AP}_\text{3D}$ metrics and
  additional ablation studies.

summary_of_the_review:   Apart from a few minor concerns, such as comparisons to
  the use of "traditional" HD maps, I do not see any major
  flaws with the proposed paper. The proposed technique is
  orthogonal to improvements in architecture, data
  augmentation, etc., and it seems to work in a wide range
  of settings. The paper itself is well written and the
  experimental evaluation is thorough.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper:  The paper proposes a method for improving object detection in lidar scans, which works in cases where a lidar scanner visits the same environment multiple times. The paper proposes to take point clouds from previous traversals and put them through a trainable network, before temporally aggregating features from each of the traverals. (It is assumed that camera pose is known for each scan). The aggregated features are then concatenated with the features from the current traversal, before being input into an off-the-shelf object detector. Results show that this method improves object detection rates in almost all cases, regardless of the object classification method used (the authors run experiments with four different object detectors).


main_review: 
# Pros:

+ The overall idea is a good one. Leveraging these past point clouds, which would typically be thrown away, is sensible, and the algorithm is a good and simple approach to using these past traversals, which effectively give a kind of 'peak ahead and behind' into areas of the scene not currently seen well. As SLAM and relocalisation algorithms improve, the algorithm will be more and more suited to different scenarios.

+ I like that fact that the authors used four different object detection algorithms to show the gains are independent of detection method used.
Experiments are generally well thought out and well presented. A good set of experiments in different scenarios are shown, using two sensible datasets.

* Including qualitative figures helps a little to understand the algorithms improvements.

* The paper is very well presented, with clear figures and tables and nice typesettings

* English is good and explanations are clear.

* While related work feels a little short it gives a very clear overview of related works, categorising them nicely.


# Cons:

- My main concern with the paper is a lack of introspection. In the introduction, the authors say that past traversals "...reveal where pedestrians, cars, and cyclists generally tend to be in the scene, and where a stop sign or some unknown background object is persistently present across traversals. With this contextual information at hand, a detector can better recognize, say, a pedestrian heavily occluded by cars on the roadside, since pedestrians have come and gone through these regions before, and it is unlikely that they are persistent background objects." But this is speculation; I was really hoping for (and expecting!) an experiment which confirmed this, shedding some light on what the network is actually learning, and how it is really helping improve scores. For example the 'pedestrian' class undergoes a surprisingly large increase (Table 5, Fig 2). 

- Some ideas for things that might help give introspection include:

1) Looking at the split of improvements on moving vs static objects.

2) Does detection rate only improve in regions where previous traversals actually contained instances of the object being detected? Or does the detection rate also improve where previous traversals contained no cars/pedestrians/cyclists?

3) Some of the following experiments might also help:

- I'd be keen to see an extra row in e.g. Table 5, which uses one 'past traversal', but where that past traversal is in fact just the entire current traversal. So the SQuaSH features would have access to information about the scene captured e.g. 5 or 10 seconds before and after the current scan P_c.

- Similarly, I'd like to see an extra row in Table 5 where the 'past traversal' is a copy of the current scan *but only up to the time at which P_c was captured*.

- It might not be easy, but: I'd have really liked to see a baseline which uses some kind of SOTA recurrent network to make use of the point clouds which immediately came before P_c temporally. And/or merging a short time history of pointclouds together into one single point cloud for input to the detection algorithms. (It feels a little unfair that the baseline only has access to point data at a single timepoint, when models already exist to make use of time series data). See e.g. the LSTM works I list below.

- Related work feels a little short.  

  - There are more 'change detection' works which could be worth mentioning. [City-scale Scene Change Detection using Point Clouds, ICRA 2021] is a recent example I found.

  - In terms of "Perception with historical context" – it might be worth mentioning that *short-term* historical context is very often used where temporal data is present, e.g. in an LSTM-based model. It is just longer-term historical context which is more rarely used. For example: 

    - Liang et al. PnPNet: End-to-End Perception and Prediction with Tracking in the Loop, CVPR 2020
    - Huang et al. An LSTM Approach to Temporal 3D Object Detection in LiDAR Point Clouds. ECCV 2020
    - Yang et al. 3D-MAN: 3D Multi-frame Attention Network for Object Detection. CVPR 2021.


summary_of_the_review: Overall I would like to see this paper in ICLR. 

It is a nice idea, a well-written paper, and I would learn something from it. There are lots of results in the paper, but most of them are simply showing that the algorithm improves scores. I would really like to see a couple more experiments helping to show *why* the scores improve. I would also like to see some stronger baselines using point cloud data immediately preceding P_c as additional input.
review_title: null

Meta-review: The paper proposes a framework for object detection on lidar scans, with query of scene feature extracted offline from previous traversals. Overall there is good agreement among reviewers, with three recommending accepting the paper and one marginally accepting it -- to me the authors satisfactorily addressed most aspect raised in reviewing.
Decision: Accept (Poster)