Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a range image based encoder-decoder architecture to predict future pointclouds. A temporal stack of range images is used to predict another stack of future range images. The experiments evaluate the performance of the architecture in both quality and computational speed. The impact of the different architecture's components is also evaluated.
strengths_and_weaknesses: The proposed method uses common ideas and techniques from image based deep learning with typical range image representations. While the components used are not novel per se, the execution and description of the approach are easy to read and follow, with the method itself being technically sound.

The most prominent aspect that could be improved in the method description is to provide more insight and discussion into why specific aspects are used. For example, the method uses simple convolutions to capture temporal aspects via the "depth" of the image stack instead of LSTM or other mechanisms. What are the expected benefits and drawbacks of this? The encoder-decoder structure is also not exactly the typical reconstruction focused one, as the reconstructed image is not the same as the input. What is the reason this architecture was chosen over other plausible alternatives, such as a pure reconstruction encoder-decoder whose low-dimensional embedding is then used for future prediction via an MLP architecture? One aspect that was not fully clear is whether or not the receptive field in "time" is different from the spatial one.

The losses used seem sensible, and the explanation for their use is clear. One common issue with such compound losses is that the different components require scaling or tuning with respect to each other. Based on the description in the paper, this is not the case here. Is there a particular reason, and were any tests done to see how sensitive the results are regarding such weighting?

The experiments overall are well done and address questions that are pertinent to the method. My biggest complaint would be the lack of proper statistical information, such as standard deviation or better quantiles. Having this information would provide greater insight into the range of outcomes the various methods produce instead of a singular value such as the mean that is easily affected by outliers. The ablation study is welcome and informative, however also somewhat confusing. The methods labelled "CD Loss From Start" and "Ours" are identical based on their feature sets but have different performances. From the text, I could not decipher the difference between them as the names used do not line up with the headings in the table. 

While the paper suggests that the qualitative results shown in Figure 4 showcase the improved performance of the proposed method, I cannot see that. To me, all images look equally "bad" regarding their ability to reconstruct the scan. This is likely due to a 3D point cloud being projected onto a 2D image, making them significantly harder to visualize. I do not doubt that the differences exist, but rather the visualization does not convey this convincingly enough.

Finally, one of the motivations at the beginning of the paper was predicting point clouds for tracking dynamic objects. In light of this, it would have been perfect to evaluate the predictive quality for such a task or at least the quality of the prediction for objects of interest, as opposed to the entire point cloud.
summary_of_recommendation: Overall the paper presents a sound method that achieves the goals outlined. The approach uses well known and understood components to achieve this but could benefit from a more thorough discussion of these choices. The experiments demonstrate the performance of the method adequately. However, they could be improved by providing a proper statistical evaluation of the results.
issues: - Lack of statistical evaluation of the results
- Could benefit from a more thorough discussion of architecture design choices
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper considers the problem of predicting future 3D LiDAR point clouds. It proposes a model that first encodes a single 3D LiDAR scan into a 2D representation, then forms a 3D tensor by concatenating the 2D representations across time, and applies a 3D CNN encoder-decoder architecture. The training does not require any labeled data, but just LiDAR scans across time.
strengths_and_weaknesses: Strengths: The proposed idea is simple and is reported to perform better than existing approaches for point cloud prediction. The proposed model can predict faster than the LiDAR sensor frame rate of 10 Hz, which is a meaningful consideration when it comes to practicability of the proposed solution.

summary_of_recommendation: The paper considers the problem of predicting future 3D LiDAR point clouds and proposes a very simple architecture. The proposed model is reported to outperform some of the methods in the literature. However, the limitations occur on two counts: 

(1) The paper does not discuss how far it can predict. Would the model complexity increase if it is asked to predict for the future 100 frames, instead of just 5? How would that compare with the other models - that for example use LSTM (as LSTM can track and predict over longer time horizons)? How would this impact the time required to predict - would it still be within 10 Hz? 

(2) The proposed model seems like a simple application of an existing 3D convolutional encoder-decoder architecture (with slight modification). There isn't much novelty. 
issues: 1. Figure 1 does not show any perceptible change in the predicted cloud. The authors can think of showing this in another way?

2. The 3D LiDAR scan to 2D representation is a critical part of the architecture and should be included in the main part of the paper. It is not possible to follow the paper without it.

3. The notation P H W is not explaining in the main paper, the first time it is used. 

4. The method is used to predict only the next 5 frames (about 0.5 seconds) from the past 5 LiDAR scans (about 0.5 seconds). This seems like a very short time span for the KITTI dataset. It would have been good if the paper had discussed more the limitations of the model towards extending the prediction horizon.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a self-supervised method for predicting future LiDaR point cloud measurements, using a encoder-decoder type neural network. Data from past point cloud frames is first converted into 2D range images, stacked along the time dimension to obtain a 3D tensor. This tensor is an input to an 3D convolutional encoder network. The latent representation is upsampled by the decoder module to produce a stack of predicted future range images. Skip connections from intermediate layers of the encoder to the decoder are added to maintain spatial and temporal details. The method is pre-trained using fast to compute image-based losses, and fine-tuned using the Chamfer loss on the predicted point clouds. Empirically, the proposed method performs strongly compared to state of the art, with ablation studies backing up the decision choices. Finally, the generalization capabilities of the method to new environments are demonstrated.
strengths_and_weaknesses: Strengths:

- The method is technically solid, with many interesting innovations described in dealing with point cloud data as range images, processing them using 3D CNNs (esp. skip connections and circular padding), and the self-supervised training method with pre-training and fine-tuning phases. Empirically, the outcome is a comparably fast system that is simple to train. I expect that this work will have a significant impact in this area of research.

- The empirical results are stronger than current state of the art, especially for longer prediction steps. The ablation study in Sect. 4.4. clearly shows the usefulness of the different design decisions, in particular that the skip connections and circular padding are very helpful.

- The paper is overall very clearly written and presented, and an enjoyable read.

Limitations:

- Even if prediction accuracy measured by Chamfer distance are better for the proposed method, it is not clear if this necessarily translates to improved performance in downstream tasks. The results of this paper could be further strengthened by demonstrating performance improvements over other methods on some downstream tasks using the predicted point clouds.

- While the ablation studies are sufficient, it could have been nice to explore some further aspects. Longer prediction horizons might be interesting, as the current 5 steps corresponds to 0.5 seconds only, as far as I understand. Sensitivity to noise or other disturbances in the input data could be explored.

- The generalization study is nice, but the results here are not very thorough. This aspect might be interesting to explore more deeply in future work.


Other comments/suggestions/minor:
- Sect. 4.1. could be moved to after the main ablation results. I think this would be clearer, showing qualitative examples after demonstrating the usefulness of the two additions quantitatively.
- Fig 2 caption: rang --> range
summary_of_recommendation: This paper presents a strong contribution for point cloud prediction. The empirical study is thorough, demonstrating all necessary aspects of the method. While there are shortcomings, it can be argued that they will be addressed in follow-up works. My only reason for a weak acceptance recommendation only is that I lack the expertise in this particular area to take a stronger stand.
issues: I would be happy to have comments from the authors on the limitations I mentioned, and hope the minor corrections can be implemented.
review_title: null

Meta-review: The reviews are generally positive. The AC endorses the reviewers' recommendation. The authors are encouraged to address questions and concerns raised in the reviews.
Decision: Accept (Poster)