Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The submission proposes a method for predicting dense traversability maps from LIDAR/pointcloud inputs. The method applies a deep model, various tricks for handling sparse data and is trained via supervised learning. The evaluation is performed on variants of 2 benchmarks (SemanticKITTI and RELLIS-3D) and includes the comparison against one external baseline and a set of variations of the proposed method.
strengths_and_weaknesses: The paper is mostly clearly written and structured. The proposed method outperforms the introduced baseline and additional ablations are useful to understand various parts of the method. In particular, the questions addressed in section 5.2.1 help the reader to better disentangle the individual contributions of various modelling choices.

There are a couple of open questions regarding this paper. A key point in the method seems to be the reduction to 4 traversability categories instead of the original set of categories in existing datasets. It is not clear to me why the presented method could not be applied to the full set of categories and, related, be compared against the full set of methods which have been applied to these datasets. This would not need to replace the main evaluation but would be an important extension for comparing against the existing state of the art in semantic segmentation in these domains. It is even more important as the evaluation only uses a variant of a single external baseline. It would be easy to make space in the paper for an additional comparison by moving the implementation details to the appendix (section 4).

For the method itself there remain some unclear aspects of which I’ve added most under minor issues below. One specific question is how the unknown class is treated. The paper describes its use for evaluation, but couldn’t we also use it for training the network? Since the unknown class depends on what happens in the future, my intuition would be that it is impossible (or at least very hard) to predict as it will depend on driving behaviour. In addition, if predicting the unknown class is never informative for evaluation, it might make sense to not train the model in the first place on cells which are labelled as unknown. 

Another question regarding the evaluation is the test with noisy odometry, a key area where the proposed method outperforms the external baseline. Since you control the noise during training of your model and during evaluation, it would be important to ablate this aspect. In particular if you have defined the noise to be the same across both. Real odometry noise would be out of your control and it would be good to see how good predictions are if your training noise exceeds or is exceeded by the evaluation noise.

Minor issues:
- L35: ‘efficient backbone’ could be described in a more informative way; how does inpainting work here. Short description would suffice. 
- L38: It would help to start with stating that this relies on ground-truth labels in the datasets. There has been a good share of related work which simply uses trajectories (ie odometry) and inverse reinforcement learning to determine traversability maps [1,2]
- Section 4: why do the different datasets require very different numbers of frames and stride? Were these collected at different frequencies? If so it would be good to add that information to justify changes in hyperparameters.
- L176: referring to the subsection would be helpful (instead of just Sec. 5)
- L179: Out of curiosity: Would it be possible to train a single model on both datasets?
- L205: ‘C3D-TA’ Please use the same abbreviations in table and text.

[1] Kris M. Kitani, Brian D. Ziebart, James Andrew Bagnell, and Martial Hebert. Activity  Forecasting. Computer Vision ECCV 2012
[2] Wulfmeier, Markus, Dominic Zeng Wang, and Ingmar Posner. "Watch this: Scalable cost-function learning for path planning in urban environments." 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.

summary_of_recommendation: The submission presents a mostly well-written and intuitively structured paper. The method is new but close to existing approaches and the application is relevant. The overall impact of the paper is likely within limits but could be considerably increased by extending the comparison wrt more baselines (currently only one) and the use of further semantic classes by using the full version of both applied benchmarks.

(rebuttal response: For reasons described during the rebuttal discussion, I will increase the score.)
issues: - extremely limited external baselines.
- not evaluated on complete versions of the applied benchmarks (if there is a good argument for why this is not feasible, I can drop this point).
- further issues in evaluation as described in the main review.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors propose a method for LiDAR semantic information aggregation and completion. The target application of their method is for off-road traversability prediction. 
strengths_and_weaknesses: Strengths:

The authors have constructed a fairly complex system for aggregating semantic information from LiDAR data. The odometry integration is differentiable so that gradients can be backpropagated through time to perform the aggregation. The sparsity of the LiDAR data is overcome partially through this aggregation but also through an in-painting network. The results show that they are able to produce semantic maps over their 4-class ontology that are stable could be useful for off-road robot navigation. 

Weaknesses:

[W1] I feel the largest weakness of the work is that the approach is designed for predicting terrain traversability, but the work never deploys the model to evaluate navigation on a mobile robot. Some of the choices that are made about what classes should be deemed "traversible" and which classes not will be traversible depends heavily on the specific robot. I'm not sure it will always be clear that some classes should be traversible and others not in fact. It would seem that a requirement for demonstration of the value of their traversability prediction is that it is in fact traversible for some specific robot. Without this, the method is just aggregating semantic information and fusing it with odometry. 

[W2] I'm not clear on whether geometry is being used to determine traversability at all. On line "For the remaining points, we take the point with the lowest traversability at each x, y location as the final traversability label" it seems not. It would seem that for traversability we need a mix of geometry and semantic information. For example, if grass is a steep hill then it may not be traversible but if it is flat it will be. 

[W3] I'm somewhat confused about how the aggregation is accounting for odometry error both in the dataset generation and at inference time. On L 211 you claim that "BEVNet-R uses learned recurrency to “fix” the errors in odometry" but I don't understand what this means or even how this is possible. Please clarify/elaborate. 

[W4] I find the 3rd claim of contribution overstated. The claim is " 3) filtering out irrelevant obstacles like overhanging tree branches that do not affect traversability" which is achieved in the methodology by "we filter out overhanging obstacles by remov- ing points that are above the ground plane by a certain threshold because they will not collide with the robot." This doesn't seem to me to warrant a claim of novelty.  

[W5] I would have like to have seen a comparison against a more traditional SLAM-based fusion approach. 
summary_of_recommendation: It is difficult to assess the true usefulness of this approach for real robotics navigation since the model is never shown to work in this setting. The ontology that is used to define traversability is hand-constructed and it is not clear (at least to me) to what degree this output actually maps to traversability of a real robot in a practical setting. It is possible that the work could be re-framed as an approach for semantic fusion and completion for LiDAR data, but as it stands it is hard to evaluate on the use-case that it sets out to achieve, which is traversability.  

-----------------------

Post-rebuttal comments. 

The authors have done an exceptional job trying to address the comments that I raised in the first round. I particularly appreciate the new comparisons with a SLAM-based baseline. I have correspondingly updated my recommendation to "Weak accept". 

A few further suggestions for the camera-ready version if the paper is accepted. I would suggest to move some of the responses to my (and other reviewers') comments out of the supplementary and into the main manuscript. For example, in my opinion, the comparison with the traditional SLAM approach is much more valuable/insightful than the discussion about early/late aggregation in the ConvGRU. I would also suggest that they update the description of the third claim of novelty to include more of the details in the response here since as stated in the manuscript currently it seems kind of trivial, but only now do I understand that it is really not. Additionally, if the authors' have performed experiments on the Warthog (as evidenced by the project page), it would strengthen the paper to include at least some qualitative evidence of this to strengthen this as a truly applicable and battle-tested method for robotics. 


issues: Please address/comment on/rebut the weakness listed above or reframe the work as squarely on semantic information fusion, or demonstrate the applicability of the approach by deploying it on a real robot (probably not possible in the rebuttal time frame).
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper considers traversability estimation for autonomous off-road driving, and develops Bird-eye View Network (BEVNet) which predicts traversability map from sparse Lidar inputs with semantic information (semantic objects are mapped to labels: free, low-cost, med-cost, lethal). The architecture consists of 3d sparse cnn for point cloud processing, ConvGRU for temporal aggregation of scans, and 2d encoder/decoder to produce traversability birds-eye-view map. As parts of the framework, unseen areas of the map can also be predicted. The traversability map can be converted into a cost map for planning purposes. 

The proposed algorithmic aggregates scans, points are semantically mapped to labels, ground plane is estimated, and used as reference for the final projection, before which points of a certain theshold above the ground plane are filtered. For the remaining points, the one with lowest traversability is used as the final 2d projected point label. The vehicle odometry is used to transform and aggregate scans over time. 
The network is also trained using noisy perturbed inputs, which might help explain why it is later more robust to noise from odometry (and hence the map aggregation). 

The authors include comparisons to Cylinder3d network on two standard datasets -- Kitti and Rellis, demonstrating improved performance. 

Other comments:
Would be useful to explain what Cylinder3d does -- it's not clear from the name that it is an appropriate comparison network. 
How is the ground truth constructed in the Figures? It seems noiser (and more incomplete) than BEVNet ?
strengths_and_weaknesses: Strengths:
+ general method for semantics+lidar+odometry-based traversability estimation
+ the qualitative results demonstrate that the method is promising
+ the authors have performed comparisons using standard data-sets

Weaknesses:
- the paper claims to address off-road traversability but by explicitly relying on semantics it is extremely difficult to address offroad navigation where aspects such as slope, surface roughness, tire-surface interaction, etc... are not easy to label semantically; the authors might want to limit the scope and be clear that it only applies to a class of environments that can be semantically labeled
- the dependence on a single ground plane might limit the applications to planar environments
- is Cylinder3d the only/best comparison framework?
summary_of_recommendation: The paper has interesting ideas about combining geometry and semantics for traversability.
issues: Would be useful to limit the scope when referring to "off-road" as noted above. 
Please clarify and address the comments under Weaknesses above.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper proposes a deep learning architecture for off-road terrain traversability. The BEVNet architecture is based on Birds Eye View Network that contains 3 stages; 3D sparse convolutions, a recurrent network and an inpainting network to estimate current and forecast future traversability. The 3D sparse convolutions process voxelized point clouds, the recurrent network accumulates and fuses data over time and the inpatining network completes the traversability map. 
Two main datasets are used to evaluate the proposed network, SemanticKITTI and RELLIS-3D. Quantitative, qualitative and ablation study results are presented and benchmarked with state of the art approaches.
strengths_and_weaknesses: Strengths:
The paper is well written and well structured. The related work seems complete. The methodology is quite clear as well and the results are very well presented.
The authors tackle the problem of semantic mapping and traversability in off-road scenarios, which differs of most of the related research that have focused on on-road developments.
Exploiting geometric and temporal constrains is the key feature of the proposed BEVNet.
The thoroughness of the results is one of the highlights of this paper. The results show very clearly the performance of the network with respect to SOTA and the proposed stages.
The supplementary material helps to a great extent to make results reproducible and to demonstrate the performance.

Weakness
The main weakness in my view is that the network purely rely in data-driven models. Despite the relatively good performance, the authors had to train a separate model per dataset, thus the models have a hard time to generalized. In many circumstances such as data fusion, well established "hand-engineered" models will produce more accurate and robust results in none observed datasets.



summary_of_recommendation: The authors tackle the relevant problem of off-road navigation, which has been largely overlooked by the community. The proposed network is well justified and explained. The methodology is easy to follow. The analysis of the results show the proposed network performs well in the intended off-road scenarios. The thoroughness of the validation benchmarks very well the network with respect to its variations and the SOTA.
issues: Please define "r" of the feature vector (line 138).
The noise added to odometry is realistic, but it would be good to see how the network behaves with more noise.
In the results I am missing the comparison of BEVNet-S with the vanilla Cylinder-3D. Results could be added to Table 1.
Are the results of Table 2 obtained with the network "full"?
Is there any difference with the end-to-end trained network vs the two-stage training?
In the videos, I noticed there is a flickering issue near the borders in the traversability output of the network. It would good to explain this issue in the RELLIS-3D dataset, where is quite evident.
Finally, it would be relevant for the community if the authors make the network open-source.
review_title: null

Meta-review: The paper proposes an approach for traversability estimation from LiDAR point clouds. The reviewers find the paper in general to be clearly written and well structured. The reviewers appreciate the ablation study. Most of the major concerns raised by the reviewers have been addressed in the rebuttal. I thank the authors for incorporating all the suggestions and I recommend acceptance.
Decision: Accept (Poster)