Semantic Terrain Classification for Off-Road Autonomous Driving
Authors: Amirreza Shaban
Xiangyun Meng
JoonHo Lee
Byron Boots
Dieter Fox
Venue: CoRL2021 Poster (2021)
Keywords: Off-road Driving
Autonomous Driving
Deep Learning
Perception

Abstract: Producing dense and accurate traversability maps is crucial for autonomous off-road navigation. In this paper, we focus on the problem of classifying terrains into 4 cost classes (free, low-cost, medium-cost, obstacle) for traversability assessment. This requires a robot to reason about both semantics (what objects are present?) and geometric properties (where are the objects located?) of the environment. To achieve this goal, we develop a novel Bird's Eye View Network (BEVNet), a deep neural network that directly predicts a local map encoding terrain classes from sparse LiDAR inputs. BEVNet processes both geometric and semantic information in a temporally consistent fashion. More importantly, it uses learned prior and history to predict terrain classes in unseen space and into the future, allowing a robot to better appraise its situation. We quantitatively evaluate BEVNet on both on-road and off-road scenarios and show that it outperforms a variety of strong baselines.

Semantic Terrain Classification for Off-Road Autonomous Driving
Amirreza Shaban, Xiangyun Meng", JoonHo Lee", Byron Boots, Dieter Fox
University of Washington
Producing dense and accurate traversability maps is crucial for au- tonomous off-road navigation. In this paper, we focus on the problem of clas- sifying terrains into cost classes (free, low-cost, medium-cost, obstacle) for traversability assessment. This requires a robot to reason about both semantics (what objects are present?) and geometric properties (where are the objects lo- cated?) of the environment. To achieve this goal, we develop novel Bird's Eye View Network (BEVNet), a deep neural network that directly predicts a local map encoding terrain classes from sparse LiDAR inputs. BEVNet processes both geo- metric and semantic information in a temporally consistent fashion. More impor- tantly, it uses learned prior and history to predict terrain classes in unseen space and into the future, allowing a robot to better appraise its situation. We quantita- tively evaluate BEVNet both on-road and off-road scenarios and show that it outperforms a variety of strong baselines. Website:
Abstract:
Keywords: Off-rooa Driving, Autonomous Driving, Deep Learning, Perception
Introduction
While there has been great recent interest in the de- velopment of autonomous vehicles, the vast major- ity of this work has focused on-road and urban driving. However, a wide range of application ar- including defense, agriculture, conservation, and search and rescue could benefit from autonomous off-roud vehicles that can operate in complex, nat- ural terrain. In such environments, understanding the traversability of the terrain surrounding the vehicle is crucial for successful planning and control. Per- ceiving whether the terrain is traversable from sparse LiDAR data can be challenging problem as off-road terrain is often characterized by rapid changes to the ground plane, heavy vegetation, overhanging branches, and negative obstacles. In other words, a successful off-road robot must rea- son about both the geometric and semantic content of its surroundings to determine what terrain is traversable and what is traveerrsr...
Figure 1: Example target scenario. BEVNet can be run on the Clearpath Warthog generate semantic map of offroad envrionments.
In this work, we formulate traversability estimation as a semantic terrain classification problem [1, 2]. The motivation is to unify the semanties (what objects present?) and geometry (where are the objects located?) of the terrain into a single cost ontology. From the semantic perspective, objects such large rocks and tree trunks are non-traversable, whereas gravel, grasses, and bushes are all traversable by an off-road vehicle [3] but with increasing difficulty. From the geometric perspective, overhanging obstacles can be ignored, and objects of the same semantic class may vary in their traversability depending on their heights (e.g., tall bushes vs. short bushes). To this end, we use discrete set of traversability levels for the ease of grouping the semantic classes by their traversability while allowing us to adjust the traversability levels of specific instances based on their geometry.
Equal Contribution. Contact at (ashaban, ziangyun, joon14) washington.ce edu
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Single
Sparss Convolutions
ConVORU
H+1
512
Sparse
M
Sparso sess Input Tonsor
Sparse Feature Tensor Latent Feature Nop 512 012x
512
M
Warp&ffinet flincl (TT1)
M
Figure 2: The network architecture BEVNet. The incoming LiDAR scan first discretized into sparse voxel grid, which is then fed into a sequence of sparse convolution layers to compress the z dimension. The compressed sparse feature tensor is aggregated over time via the COnVGRU unit. We use differentiable affine warping to align the latent feature map with the current odometry frame. Finally, the inpainting network "inpaints" the latent map to output a dense traversability map.
An effective terrain classification system should efficiently 1) aggregate observations over time [1, 4] with noisy odometry [5, 6]. 2) reason about the partially seen or even yet to be seen parts of the environment [7, 8, 9], and 3) detect overhanging structures such tree branches, tunnels, and power-lines [10, 11]. While previous work has addressed each of these issues individually, the aforementioned challenges are related, and solving each one should benefit the others.
In this paper, we propose a Bird's Eye View Network (BEVNet), a recurrent neural network that directly predicts terrain classes in the form of 2D grid around the robot from LiDAR scans. As shown in Figure 2, our model has three main parts: 1) a 3D sparse convolution sub-network to process the voxelized point cloud, 2) a Convolutional Gated Recurrent Unit (Conv GRU) which uses convolutional layers in a gated recurrent unit [12] to aggregate the 3D information, 3) an efficient 2D convolutional encoder-decoder based on [13] that simultaneously inpaints the empty spaces and projects the 3D data into the 2D Bird's Eye View (BEV) map. To train the model, we use both past and future labeled LiDAR scans to build complete 3D semantic point cloud and build the ground-truth 2D traversability map. Previous work [10] utilizes collection of collapsible cube structures with ground/overhang classification remove irrelevant overhangs based on their gaps, but such rule-based filtering lacks generalization when it is difficult to estimate accurate ground levels from sparse LiDAR scans. In comparison, our model trained with ground-truth BEV map built from fully observed and labeled environments, which allows accurate ground-leve estimation for learning. The network then learns detect and remove overhanging obstacles from sparse LiDAR scans without an explicit filtering mechanism.
We make several contributions and empirical observations. We proposed novel framework to build the BEV costmap by simultaneously 1) aggregating observations over time, 2) predicting the unseen areas of the map, and 3) filtering irrelevant obstacles like overhanging tree branches that do not affect traversability. Experimental results on SemanticKITTI [14] and RELLIS-3D [15] show that BEVNet outperforms strong baselines in both on-road and off-road settings.
2 Related Work
Since most prior literature on perception for autonomous driving [16, 17, 18, 19, 20, 21, 22, 23] focus on urban environments and structure inherent in cities and road networks, we compare our system to works that share the mutual components to ours, namely traversability analysis [24, 25, 26, 27], semantic mapping [1, 17, 18, 19, 23, 21), recurrency handling [1, 17, 19], and semantic scene completion [7, 8, 9].
Traversability Analysis and Semantic Mapping. Traversability analysis and semantic mapping are crucial for off-road autonomy [25, 28, 29]. Traversability may be analyzed based on various criteria, including surface roughness [24], negative obstacles [25], and terrain classification [26, 27]. The traversability information is usually projected into Bird's Eye View (BEV) map, which stores local traversability and semantic information in topdown 2D grid [30]. While prior works such as [16, 17, 18] utilize high definition map assumed a priori, such map is expensive to produce,

---

and therefore our system instead constructs one online. Similarly. Casas et al. [19], Wu et al. [23] produces online semantic maps for urban autonomous driving but prioritize mapping of lanes and road objects. In comparison, our system focuses on broader categorization of semantics based on terrain traversability. Additional works generate semantic maps from images, where Philion and Fidler [21] produce BEV maps from RGB images end-to-end, while Maturana et al. [1] apply bayesian filter. Our system instead learns to produce semantic maps from LiDAR scans. We also compare our approach to LiDAR segmentation where the segmented point cloud from networks such as [31, 32] can be projected onto a BEV map. We evaluate this in detail in Sec. 5.
Temporal Aggregation Temporally consistent semantics is crucial for stably mapping the envi- ronment. Recent work [17, 19] directly concatenates the past 10 scans as its input for memory efficiency. In comparison to our system, the author's recurrent neural network (RNN) is specifically designed for semantic occupancy forecasting, whereas our recurrent architecture accumulates fea- tures sequentially to better estimate the traversability of the current surrounding terrain. Maturana et al. [1] accumulate information via Bayes filtering, which in our system is also replaced by the RNN. There are various available architectures for handling recurrency[4, 33, 34, 35, 36, 37, 38], of which our system utilizes ConvGRU to accumulate 2D BEV maps.
Semantic Scene Completion (SSC) The goal of SSC is to generate a complete 3D scene given single LiDAR scan. [7, 8, 9] utilize information from semantic segmentation to complete the scene, whereas our system learns to directly predict the completed scene and therefore does not require segmentation from a secondary network or ground truth labels. In addition, our system performs point cloud projection and scene completion in 2D simultaneously, the main task in our work is to produce a 2D BEV map. We evaluate our system's 's ability to complete scenes by comparing our model against [7], the details can be found in Sec. 5.
3 Method
3.1 Overview
We consider mobile robot with 360 LiDAR mounted at its top. In order for the robot to navigate efficiently and safely in a new environment (either on- on-road or off-road), the robot builds an online traversaiiityy map around itself. The traversability map resembles conventional occupancy map as well as the semantic map from [1], where each cell stores a probability distribution of traversability four levels of traversability: free, low-cost, medium-cost, and lethal.
labels. In this work, The number of traversability levels can be trivially extended if so desired. The traversability map is inside the robot's is odometry frame, so that the robot is always the center, with its heading pointing to the east. The traversability map is converted to a costmap by mapping each traversability level to the corresponding cost value via lookup table. The converted costmap can be easily interfaced with a local planner [39] or a global planner (e.g., A*) for finding the least-cost path to goal.
We adopt supervised-learnnng approach to predict this traversability map. We start with building a traversability dataset from LiDAR segmentation datasets [14, 15] via a traversability- jection procedure. Then, we introduce BEVNet, recurrent neural network that takes the current LiDAR scan and utilizes its history to build a dense traversability map. In the following sections, will describe each component in detail.
pro-
3.2 Building a Traversability Dataset
Recent work [17, 1] focuses on-road driving where reasoning about a large number of fine- grained semantic classes is necessary. Here we consider a more general driving paradigm where we simply care about the traversability of the surrounding terrain. This makes our model applicable to both on-road and off-road driving. Given a dataset with semantically labeled LiDAR scans, we convert it to traversability dataset via the following procedure (illustrated in Figure 3).
Scan Aggregation. For each scan, we aggregate it with the past and the future scans with stride to construct a larger point set. We set / to a large enough number (e.g., 71) to obtain dense traversability information for a large area around the robot. These parameters may be tuned depend- ing on the vehicle speed and density of the LiDAR points.
3

---

Froe
Low cost
Medium
Lethal
Overhanginng
(ramovad)
Traversability
Classes
Traversebilty
binning ground plane estimation
Traversabiliy Projection
Traversability Map
Aggregation
Figure 3: Process for generating the traversability datasct SemanticKITTI Single scan labels are aggre- gated to form complete scenes (scan aggregation), and their semantics are remapped to our 4 class ontology (traversability mapping), legend defined in the figure. The remapped scans are then filtered with ground esti- mation to remove overhanging points (points binning and ground plane estimation). Finally, the filtered points are projected to traversability map (traversability projection).
Traversability Mapping. We map the semantic classes into our 4-level ontology. The general prin- ciple is to map semantic classes with similar costs the same traversability label. For example, cars and buildings are mapped to lethal, whereas mud and grass are mapped to low-cost. Detailed mapping can be found in the supplementary.
Points Binning and Ground Height Estimation. For each point in the aggregated scan, we do a down projection to find its location æ, y on the traversability map. Hence each z, y location of the map contains a pillar of points. We estimate the ground height map by running a mean filter kernel over the lowest 2 coordinates of the points labeled as free and low-cost at each x, y location in the map. This height map is used Traversability Projection. For each pillar of points, we filter out overhanging obstacles by remov- ing points that are above the local ground level by a certain threshold because they will not collide with the robot. Additionally, we adjust the traversability level of certain points based on their height above the ground level and the mobile capabilities of the robot. i.e. points labeled as medium-cost but very close to the local ground level can be deemed negligible for a large off-road vehicle and hence remapped to low-cost other nearby points. Finally, take the class of the least traversable point (i.e., most difficult) at each x, location as the final traversability label.
reference for final traversability projection.
3.3 Feature Extraction via Sparse Convolution with Z Compression
The architecture of BEVNet is shown in Figure 2. An input LiDAR scan is first discretized into 512x511 512x31 grid with a resolution of 0.2m. We perform sparse discretization so that only occupied voxels are preserved. Each voxel contains 4-dimensional feature is the average of the coordinates and remission values of the points inside the voxel. This sparse voxel grid fed into a sequence of sparse convolution layers, which compress the dimension via strided convolutions. We keep x and dimensions unchanged. The output of the sparse convolution layers is sparse feature tensor S of size 512 512 x C, where C' is the feature dimension.
Σ [Tt, Vs. 2;, rz), which
3.4 Temporal Aggregation of Sparse Feature Maps
A single LiDAR scan becomes increasingly sparse as the distance increases, making it difficult to classify the traversability level for areas far away from the robot. Contrary to classical SLAM that aggregates LiDAR measurements over time via hand-engineered Bayesian update rule [40], we let the network learn to aggregate the sparse feature maps from past LiDAR scans via Convolutional Gated Recurrent Unit (ConvGRU). The ConvGRU maintains a 2D latent feature map M that shares the same coordinate system and dimensions as the final traversability map. The latent feature map M is updated as
M++1 = ConGRR((WarAAfiieeeMM, AT++1, St11,,
where AT& the relative transform of the robot's odometry frame from to 1+1. The WarpAffine operation transforms the latent feature map M, from the previous odometry frame to the cur-
4

---

rent odometry frame so that the features from M, and S1+1 are spatially aligned. Note that the WarpAffine operation is differentiable allow the gradients to backpropagate through time.
3.5 Traversability Inpainting
Since the ConvGRU only aggregates sparse feature tensors, M contains little information for areas where there no LiDAR point. Instead treating the no-hit area as unknown, we let the network fill in the empty space by leveraging the local and global contextual cues via the Impainting Net- work. The inpainting network a fully convolutional network inspired by FCHardNet [13] which is originally designed for fast image segmentation. It consists of sequence of downsampling and up- sampling layers with skip connections, making it effective for capturing local and global contextual information for predicting what is missing.
4 Implementation Details
We build the traversability datasets from SemanticKITTI [14] and RELLIS-3D [15] to evaluate BEVNet in both on-road and off-road scenarios. For SemanticKITTI we aggregate 71 frames with stride to generate single traversability map. For RELLIS-3I 33 we aggregate 141 frames with stride 5. Both datasets provide per-frame odometry, which we use for the differential warping layer in the ConvGRU. The traversability maps have size of 102.4m 102.4m with resolution of 0.2m. Note that the traversability maps contain additional "unknown" class marking regions that have never been observed. For additional details on network training and data augmentation please refer to section 4 in the supplementary material.
5 Experiments
We conduct both quantitative and qualitative studies on SemanticKITTI (on-road) and RELLIS-3D (off-road) datasets. We trained a separate model for each dataset. We compare with a variety of baselines, ranging from LiDAR segmentation to scene completion on the validation sequences, We also perform an ablation study better understand the contribution of recurrence, and how our model behaves on the two datasets that have very different characteristics.
5.1 Evaluation Metrics
We use the mean Intersection of Union (mloU) [13], a widely used metric for image segmentation, as the quantitative measure of the prediction accuracy. Note that our model predicts an additional "unknown" class to improve the visual consistency, and we exclude the "unknown" class in the evaluation. To better understand our model's capability of predicting the future, we report mloUs in three modes: seen, unseen, and all. In the "seen" mode, we do not include ground truth labels obtained from future frames, effectively excluding any future predictions. For the "unseen" model we only include the future predictions. the "all" scenario, we evaluate both.
5.2 Comparison with LiDAR Segmentation with Temporal Aggregation
A strong baseline for building traversability map is to perform semantic segmentation of the in- coming LiDAR scan, project it down to obtain a 2D sparse traversability map, and aggregate the traversability maps over time. To compare with this approach, we choose Cylinder3D [31] (fine- tuned on our 4-class ontology) as the LiDAR segmentation network for its strong performance and use the same projection procedure in Sec on the input LiDAR scan obtain the single-frame traversability map. We perform the temporal aggregation by tracking the categorical distribution of traversability via uniform Dirichlet prior. To so, we keep a counter map Mc of size H W x (initialized to zeros). It is of the same size as the traversability map except that the last dimension counts the traversability labels observed so far. We update Mc incrementally. For each incoming single-frame traversability map, we warp Mc to the current odometry frame via bilinear interpo- lation, and increment the counts by adding the one-hot version of the incoming single-frame map. The actual traversability label can be obtained by taking the argmax of the last dimension of Mc-
5

---

SemanticKITTI
RELLIS-3D
All
Seen
Unseen
All
Seen
Unseen
BEVN
0.416
0.465
0.308
0.559
0.518
0.345
Clean Odometry
BEVNet-TA
0.468
0.534
0.335
0.615
0.605
BEVNet-R
0,535
0.625
0.354
0.644
0.621
0.621
Cylinder3D-TA
0.465
0.655
N/A
0.411
0.568
N/A
Cylinder3D- IA-3D wlo tracing
0,482
0,660
N/A
0.408
0,649
NIA
Cylinder3D-TA- TD--3 ray tracing
0.471
0.646
N/A
0.384
0.609
N/A
Nokey Odometry
BEVNet-TA
0,379
0.415
0.310
0.452
0.372
0.560
BEVNet-R R
0.468
0.529
0.343
0.614
0.572
0.616
Cylinder3D-Th
0.342
0.455
N/A
0.318
0.435
N/A
CyDnder3D TA-3D wio ma tracing
0.373
0.479
N/A
0.347
0.517
NIA
Cylinder3D- TA-3D tracing
0.369
0.478
N/A
0.331
0.495
NIA
Table 1: Mcan IoU of different methods on SemanticKITTI and RELLIS
Results on SemanticKITTI In the left half of Table 1 compare the performance of BEVNet- Recurrent (BEVNet-R) with Cylinder3D+ Tempooa Aggregation (C3D-TA) on the SemanticKITTI validation set. When only considering what has been observed so far ("seen") and clean odometry, C3D-TA is better than BEVNet-R. This shows that LiDAR segmentation with accurate temporal aggregation work very well in structured environments such as urban driving. When evaluated on the full groundtruth ("full"), BEVNet-R outperforms C3D-TA because C3D-TA cannot predict the future traversability. When evaluating on noisy odometry, BEVNet-R surpasses C3D-TA for both "seen" and "full" test scenarios. BEVNet-R uses learned recurrency to "fix" small errors in odometry and to adaptively forget history case the error is too large. In comparison, C3D-TA solely uses the provided odometry to aggregate information, which may result in large misalignment as eπтотs compound over time.
The results on RELLIS-3D (right half of Table 1) share similar trend as those in SemanticKITTI, except that BEVNet-R consistently outperforms C3D-TA with larger gap. This suggests that off-road environment is more challenging, where accurate LiDAR segmen- tation hard to obtain due the lack of environmental structure. Indeed, Cylinder3D only achieves 64.1 mloU on RELLIS-3D for LiDAR segmentation, which is lower than the 87.9 mloU on Se- ManticKITTI. Interestingly, noisy odometry has a smaller impact on BEVNet-R R. We hypothesize that it is because the RELLIS-3D dataset contains less clutter and occlusion so BEVNet-R does not rely heavily on the history for traversability prediction.
Results on RELLIS-3D.
Comparison with 3D Semantic Temporal Aggregation. We additionally provide a baseline that aggregates the points in 3D before projection (Cylinde:DD-TA-3D using Octomap [41]. The results included in Table 1. C3D-TA-3D in general works better than C3D-TA and can even outperform BEVNet-R in the seen area. However, since it is not able to predict the future, its mloU on the full scene is poor. It is significantly slower (less than 1 fiss, and is susceptible to odometry noise like C3D-TA. We optionally enable ray tracing in Octomap to remove dynamic obstacles. While quantitatively this does not make large difference, there is clear distinction in the qualitative comparison described next.
5.2.1 Qualitative Results
In left half of Figure 4, we highlight that BEVNet-R can preserve small dynamic objects such as bicyclists better than C3D-TA. Hand-engineered temporal aggregation is prone to treating small dynamics objects as noise and ignoring them. In comparison, BEVNet namic objects, while preserving smoothness static regions. The right half shows the impact of noisy odometry. We can see large misalignment and smear artefacts for C3D-TA, whereas BEVNet- R produces significantly cleaner output. In Figure 5 we provide a comparison between BEVNet-R, C3D-TA, and C3D-TA-3D. C3D-TA-3D can preserve more details in the map, but without ray trac- ing, it fails to clear the residues left by dynamic obstacles such as cars and humans. While turning on ray tracing [41] improves the results, also erroneously clears some of the ground points due to shallow LiDAR incident angles. In comparison, BEVNet-R does not suffer from these short- comings and predicts a more complete area. Finally, in Figure 6 we visualize examples on both SemanticKITTI and RELLIS-3D. In general, BEVNet-R shows strong performance in predicting future traversability. It learns to predict whole cars, alley entrances, and trail paths with extremely sparse LiDAR points.
learn to keep small dy-

---

Cyliddr3D TA
BEWNet-
Ground Truhh
Cylnder
Figure 4: Qualitative comparison of Cylinder3D-TA and BEVNet-R. Left: BEVNet is better at preserving small fast-moving objects such bicyclists (highlighted by the blue circles), which the hand-engineered update rule tends to ignore. (Maps are 50% zoomed in). Right: When noise is injected into the odometry, BEVNet-R still predicts a clean map albeit with fewer details, while Cylinder3D+TA fails to do this, resulting in a blurry, inaccurate map.
BEVNeI-R
Cylinder3D-TA
Cylinder3D-TA-33 wlo ray tracing
Cylinder3D-TA-- with ray tracing
Ground Truth
Figure Comparing BEVNet-R, Cylinder3D-TI and Cylinder3D-TA- While Cylinder3D TA-3D produces more detailed maps, introduces additional artefacts due to dynamic obstacles. See text for more details.
5.2.2 Ablation Study
We conduct our ablation study on three variants of BEVNet: BEVNet- Single (BEVNet-S), BEVNet- Single+Temporal Aggregation (BEVNet-TA), and DEVNet-Recurren (BEVNet-R). We aim to an- swer three questions: 1) is learned recurrence better than temporal aggregation? 2) does history help predict the future? and 3) where should the information be aggregated in the network? We answer these questions through set of experiments on both SemanticKITTI and RELLIS-3) datasets.
Is learned recurrence better than temporal aggregation? When evaluated the full ground truth, we observe that BEVNet-R consistently outperforms BEVNet and BEVNet-TA on both on-road and off-road scenarios (Table 1). Notably, BEVNet-TA also outperforms BEVNet, which shows that any form of recurrence is beneficial. In particular, we observe that the learned recurrence makes the best use of the temporal information in comparison to the sand-engineered TA. When noisy odometry is introduced we observe the same trend as discussed in Sec. 5.2, where BEVNet-R shows robustness to noise and outperforms BEVNet-TA.
Does history help predict the future? In Table 1, we can see that any form of recurrence that accumulates history helps with predicting the unseen area. In particular, BEVNet-R outperforms both BEVNet-S and BEVNet-TA -e,, showing that the learned recurrence can better predict the unseen area than a single-frame model or hand-engineered temporal aggregation approach.
Where to put ConyGRU? Recurrence may be applied right after the sparse convolutions (early aggregation) or may be applied af- ter the 2D inpainting network (late aggregation). We compare the two options the SemanticKITTI dataset with clean odometry in Table 2. Note that our model trained with clean odometry. Early aggregation yields better results than late aggregation. This because when early aggregation is applied the inpainting network has access to temporally fused information and therefore is given more information to complete the scene and maintain temporal consistency across scans. Furthermore, we may infer that if late aggregation is applied, it is more given completed scenes
mloU
Early Aggregation
0.535
Late Aggregation
0.479
Table 2: Effect of GRU location in the network.
difficult for the recurrent network learn correct the odometry with potentially noisy information instead of the sparse feature maps.
5.3 Comparison and Discussion with other Related Work
Several recent works have focused on semantic understanding of scenes from sparse LiDAR scans [7, 9]. They solve related, but slightly different problems. They produce smaller maps, and do not
7

---

Input scan
Cylinder3D
CylinderD--TA
BEVNet-S
BEVNet-R
Ground truth
Figure Qualitative results of our method on SemanticKITTI [14] (top) and RELLIS [15] (bottom).
Learned recurrency in our end-to-end network can preserve previously observed information while predicting future ohservations for the occluded areas.
Canal Road
Weeds
Input scan
BEVNet-R
Camera View
Input scan
BEVNet-R
Camera View
Figure 7: Qualitative results BEVNet on collected on Clearpath Warthog. Each item shows LiDAR scan input, BEVNet-R's 's output, and frontal camera view (for reference), respectively.
perform temporal aggregation. Due to space limits we refer the reader to Sec.4 of the supplementary for detailed discussion.
5.4 Real Robot Experiments
Figure 7, we show that BEVNet-R trained on SemanticKITTI and RELLIS-3D can generalize to novel environments which fed into BEVNet-R to classify the terrains. The first environment Canal Road, is a dirt trail with light vegetation, whereas the second environment Weeds, is scattered with grass and tree branches and is uneven. In both scenarios, BEVNet-R is able to predict a complete traversability map with sparse lidar inputs, and reason about the traversability of surroundings using the semantic and geometric features. More details can be found on the website.
ClearPath Warthog robot. The robot is equipped with OS1-64 LiDAR
6 Conclusion
We propose BEVNet, a framework that classifies terrain traversability in local region around mobile robot with the aim of helping the robot navigate in novel on-road or off-road environment. BEVNet addresses a number of challenges sparse LiDAR information over time, 2) it learns reason about traversability that involves both ge- ometric and semantic understanding of the environment, and 3) it learns fill in the unknown space where there are no LiDAR hits, and thus provides the robot with a more complete understanding of its surroundings. Most notably, BEVNet can leverage past information to better predict the future. We believe BEVNet provides important step towards robot autonomy complex terrains where a prior map is unavailable.
unified architecture, namely: 1) learns to aggregate
Limitations and Future Work. Since BEVNet is data-driven, it may overfit to certain types of environments if the training data is not diverse enough. Moreover, due the fact that the publicly available datasets do not contain rough terrains, we do not take the roughness of terrains into ac- count at the moment. However, BEVNet can extended to predict additional quantities such as an elevation map to be fused with the current semantic costmap for more accurate traversability estima- tion. the future, we would like to extend the capability of BEVNet and test it in more challenging off- off-road scenarios.

---

Acknowledgments
This work was supported in part by ARL SARA CRA W911NFF20-2-0095.
References
[1] D. Maturana, P.-W. Chou, M. Uenoyama, and S. Scherer. Real-time semantic mapping for autonomous off-road navigation. In Proceedings of 11th International Conference on Field and Service Robotics (FSR '17), pages 335 350, September 2017.
[2] F. Schilling. X. Chen, J. Folkesson, and P. Jensfelt. Geometric and visual terrain classification for autonomous mobile navigation. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages
2017.
[3] Warthog unmanned ground vehicle robot clearpath, URL httpps//clearrpahhrobbii.. com/warthog unmanned -un-ann---rouu---eeeeie rround-vehicle-rote
[4] Y. Xiang and D. Fox. DA-RNN: semantic mapping with data associated recurrent neural
networks. Proceedings of Robotics: Science and Systems, Cambridge, Massachusetts, July 2017. doi: 15517//55.2001.... X1I1.01..
[5] D. S. Chaplot, D. Gandhi, S. Gupta, A. Gupta, and R. Salakhutdinov. Learning to explore using active neural slam. International Conference Learning Representattonss 2020.
[6] S. A. Mohamed, M.-H. Haghbayan, T. Westerlund, J. Heikkonen, H. Tenhunen, and J. Plosila. survey on odometry for autonomous navigation systems. IEEE Access, 7:97466-97486, 2019.
[7] X. Yan, J. Gao, J. Li, R. Zhang, Z. Li, R. Huang, and S. Cui. Sparse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. In AAAI, 2021.
[8] R. Cheng, C. Agia, Y. Ren, X. Li, and B. Liu. S3enet: A sparse semantic scene completion network for lidar point clouds. In CoRL, 2020.
[9] Y. Han, J. Banfi, and M. Campbell. Planning paths through unknown space by imagining what lies therein. In CoRL, 2020.
[10] A. J. Reina, Martínez, Mandow, Morales, and A. Gareía-Cerezo. Collapsible cubes: Removing overhangs from 3d point clouds to build local navigable elevation maps. In 2014 IEEE/ASME International Conference on Advanced Intelligent Mechatronics, pages 1012- 1017. IEEE, 2014.
[11] J. Larson, M. Trivedi, and M. Bruch, Off-road terrain traversability analysis and hazard avoid- ance for ugvs. Technical report, CALIFORNIA UNIV SAN DIEGO DEPT OF ELECTRICAL ENGINEERING, 2011.
[12] K. Cho, B. van Merrienboer, Gulcehre, Bougares, Schwenk, and Bengio. Learn- ing phrase representations using mn encoder-decoder for statistical machine translation. In Conference on Empirical Methods in Natural Language Processing (EMNLP 2014), 2014.
[13] P. Chao, C.-Y. Kao, Y. Ruan, C.-H. Huang, and Y.-L. Lin. Hardnet: A low memory traffic network. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages doi: 10. 1111100C
3551-3560, 2019.
[14] J. Behley, M. Garbade, A. Milioto, J. Quenzel, S. Behnke, C. Stachniss, and J. Gall, Se- manticKITTI: Dataset for Semantic Scene Understanding of LiDAR Sequences. In Proc. of the IEEE/CVF International Conf. on Computer Vision (ICCV), 2019.
[15] Jiang, Osteen, M. Wigness, and Saripalli. Rellis-3d dataset: Data, benchmarks and analysis, 2020.

---

[16] M. Bansal, A. Krizhevsky, and A. Ogale. Chauffeurnet: Learning to drive by imitating the best and synthesizing the worst. In Proceedings of Robotics: Science and Systems. Freiburgim- Breisgau, Germany, June 2019. doi: 201.5500///SS2299....
[17] A. Sadat, S. Casas, M. Ren, X. Wu, P. Dhawan, and R. Urtasun. Perceive, predict, and plan: Safe motion planning through interpretable semantic representations. In ECCV, 2020.
[18] W. Zeng, W. Luo, S. Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end inter- pretable neural motion planner. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8652-8661, 2019.
[19] S. Casas, A. Sadat, and R. Urtasun. Mp3: A unified model map, perceive, predict and plan, 2021.
[20] M. Bojarski, D. Testa, D. Dworakowski, B. Firner, B. Flepp, P. Goyal, L. Jackel, M. Monfort, U. Muller, J. Zhang, X. Zhang, J. Zhao, and K. Zieba. End to end learning for self self-driving cars. 04 2016.
[21] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In Proceedings of the European Conference Computer Vision, 2020.
[22] S. Hoermann, M. Bach, and K. Dietmayer. Dynamic occupancy grid prediction for urban autonomous driving: A deep learning approach with fully automatic labeling. 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 2056-2063, 2018.
[23] Wu, S. Chen, and D. N. Metaxas. Motionnet: Joint perception and motion prediction for autonomous driving based bird's eye view maps. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 13322-13992 2020.
[24] D. Stavens and Thrun. A self-supervised terrain roughness estimator for off-road au- tonomous driving. UAI'06, page 469-476. AUAI Press, 2006. ISBN 0974903922.
[25] J. Larson and M. Trivedi. Lidar based off-road negative obstacle detection and analysis. 2011 14th International IEEE Conference on Intelligent Transportation Systems (ITSC), pages 192- 197, 2011.
[26] B. Suger, B. Steder, and W. Burgard. Traversability analysis for mobile robots in outdoor environments: A semi-supervised learning approach based on 3d-lidar data. 2015 IEEE Inter- national Conference on Robotics and Automation (ICRA), pages 3941-3946, 2015.
[27] D. Barnes, W. P. Maddern, and I. Posner. Find your own way: Weakly segmentation of path proposals for urban autonomy. 2017 IEEE International Conference on Roboties and Automation (ICRA), pages 203-210. 2017.
[28] G. Reina, A. Milella, and R. Rouveure. Traversability analysis for off-road vehicles using stereo and radar data. 2015 IEEE International Conference on Industrial Technology (ICIT), pages 540-546, 2015.
[29] J. Larson, M. Trivedi, and M. Bruch, Off-road terrain traversability analysis and hazard avoid- ance for ugvs. 2011.
[30] A. Elfes. Using occupancy grids for mobile robot perception and navigation. Computer, 22: 46-57, 1989.
[31] X. Zhu, H. Zhou, T. Wang, F. Hong, Y. Ma, W. Li, H. Li, and D. Lin. Cylindrical and asym- metrical 3d convolution networks for lidar segmentation. arXiv preprint arXiv:2011.10033, 2020.
[32] T. Cortinhal, G. Tzelepis, and E, Aksoy. Salsanext: Fast, uncertainty-awarc semantic segmen- tation of lidar point clouds. In ISVC, 2020.
[33] C. Choy, J. Gwak, and S. Savarese. 4d spatio-temporal convnets: Minkowski convolutional neural networks. 2019 IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3070-3079, 2019.
10

---

[34] H. Fan and Y. Yang. Pointrnn: Point recurrent neural network for moving point cloud process-
ing. arXiv, 1910.08287, 2019.
[35] X. Shi, Z. Chen, H. Wang, D. Yeung, W. Wong, and W. Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In NIPS, 2015.
[36] S. Hochreiter and J. Schmidhuber. Long short-term memory. Neural Computation, 9:1735- 1780, 1997.
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, I.. u. Kaiser, and I. Polosukhin. Attention is all you need. I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 30. Curran Associates, Inc., 2017. URL https: /proceeddss. neurips. http://pppp............o////aaa.
Paper pdf.
[38] J. Carreira and A. Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4724-4733, 2017.
[39] G. Williams, N. Wagener, B. Goldfain, P. Drews, J. M. Rehg. B. Boots, and E. A. Theodorou. Information theoretic MPC for model-based reinforcement learning. In 2017 IEEE Interna- fional Conference Robotics and Automation (ICRA), pages 1714-1721, IEEE, 2017.
[40] S. Thrun, W. Burgard, and D. Fox. Probabilistic robotics (intelligent robotics and autonomous agents). 2005.
[41] A. Hornung, K. M. Wurm, M. Bennewitz, C. Stachniss, and W. Burgard. Octomap: An efficient probabilistic 3d mapping framework based on octrees. Autonomous robots, 34(3):189 206, 2013.


Reviews:
[{"review_id": "nSAy7-d965C", "reviewer": "Reviewer_u6cx", "summary": "The submission proposes a method for predicting dense traversability maps from LIDAR/pointcloud inputs. The method applies a deep model, various tricks for handling sparse data and is trained via supervised learning. The evaluation is performed on variants of 2 benchmarks (SemanticKITTI and RELLIS-3D) and includes the comparison against one external baseline and a set of variations of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is mostly clearly written and structured. The proposed method outperforms the introduced baseline and additional ablations are useful to understand various parts of the method. In particular, the questions addressed in section 5.2.1 help the reader to better disentangle the individual contributions of various modelling choices.\n\nThere are a couple of open questions regarding this paper. A key point in the method seems to be the reduction to 4 traversability categories instead of the original set of categories in existing datasets. It is not clear to me why the presented method could not be applied to the full set of categories and, related, be compared against the full set of methods which have been applied to these datasets. This would not need to replace the main evaluation but would be an important extension for comparing against the existing state of the art in semantic segmentation in these domains. It is even more important as the evaluation only uses a variant of a single external baseline. It would be easy to make space in the paper for an additional comparison by moving the implementation details to the appendix (section 4).\n\nFor the method itself there remain some unclear aspects of which I’ve added most under minor issues below. One specific question is how the unknown class is treated. The paper describes its use for evaluation, but couldn’t we also use it for training the network? Since the unknown class depends on what happens in the future, my intuition would be that it is impossible (or at least very hard) to predict as it will depend on driving behaviour. In addition, if predicting the unknown class is never informative for evaluation, it might make sense to not train the model in the first place on cells which are labelled as unknown. \n\nAnother question regarding the evaluation is the test with noisy odometry, a key area where the proposed method outperforms the external baseline. Since you control the noise during training of your model and during evaluation, it would be important to ablate this aspect. In particular if you have defined the noise to be the same across both. Real odometry noise would be out of your control and it would be good to see how good predictions are if your training noise exceeds or is exceeded by the evaluation noise.\n\nMinor issues:\n- L35: ‘efficient backbone’ could be described in a more informative way; how does inpainting work here. Short description would suffice. \n- L38: It would help to start with stating that this relies on ground-truth labels in the datasets. There has been a good share of related work which simply uses trajectories (ie odometry) and inverse reinforcement learning to determine traversability maps [1,2]\n- Section 4: why do the different datasets require very different numbers of frames and stride? Were these collected at different frequencies? If so it would be good to add that information to justify changes in hyperparameters.\n- L176: referring to the subsection would be helpful (instead of just Sec. 5)\n- L179: Out of curiosity: Would it be possible to train a single model on both datasets?\n- L205: ‘C3D-TA’ Please use the same abbreviations in table and text.\n\n[1] Kris M. Kitani, Brian D. Ziebart, James Andrew Bagnell, and Martial Hebert. Activity  Forecasting. Computer Vision ECCV 2012\n[2] Wulfmeier, Markus, Dominic Zeng Wang, and Ingmar Posner. \"Watch this: Scalable cost-function learning for path planning in urban environments.\" 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2016.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The submission presents a mostly well-written and intuitively structured paper. The method is new but close to existing approaches and the application is relevant. The overall impact of the paper is likely within limits but could be considerably increased by extending the comparison wrt more baselines (currently only one) and the use of further semantic classes by using the full version of both applied benchmarks.\n\n(rebuttal response: For reasons described during the rebuttal discussion, I will increase the score.)", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- extremely limited external baselines.\n- not evaluated on complete versions of the applied benchmarks (if there is a good argument for why this is not feasible, I can drop this point).\n- further issues in evaluation as described in the main review.\n"}}, {"review_id": "Yi6MqUQoPY", "reviewer": "Reviewer_vtKE", "summary": "This paper proposes a deep learning architecture for off-road terrain traversability. The BEVNet architecture is based on Birds Eye View Network that contains 3 stages; 3D sparse convolutions, a recurrent network and an inpainting network to estimate current and forecast future traversability. The 3D sparse convolutions process voxelized point clouds, the recurrent network accumulates and fuses data over time and the inpatining network completes the traversability map. \nTwo main datasets are used to evaluate the proposed network, SemanticKITTI and RELLIS-3D. Quantitative, qualitative and ablation study results are presented and benchmarked with state of the art approaches.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe paper is well written and well structured. The related work seems complete. The methodology is quite clear as well and the results are very well presented.\nThe authors tackle the problem of semantic mapping and traversability in off-road scenarios, which differs of most of the related research that have focused on on-road developments.\nExploiting geometric and temporal constrains is the key feature of the proposed BEVNet.\nThe thoroughness of the results is one of the highlights of this paper. The results show very clearly the performance of the network with respect to SOTA and the proposed stages.\nThe supplementary material helps to a great extent to make results reproducible and to demonstrate the performance.\n\nWeakness\nThe main weakness in my view is that the network purely rely in data-driven models. Despite the relatively good performance, the authors had to train a separate model per dataset, thus the models have a hard time to generalized. In many circumstances such as data fusion, well established \"hand-engineered\" models will produce more accurate and robust results in none observed datasets.\n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The authors tackle the relevant problem of off-road navigation, which has been largely overlooked by the community. The proposed network is well justified and explained. The methodology is easy to follow. The analysis of the results show the proposed network performs well in the intended off-road scenarios. The thoroughness of the validation benchmarks very well the network with respect to its variations and the SOTA.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Please define \"r\" of the feature vector (line 138).\nThe noise added to odometry is realistic, but it would be good to see how the network behaves with more noise.\nIn the results I am missing the comparison of BEVNet-S with the vanilla Cylinder-3D. Results could be added to Table 1.\nAre the results of Table 2 obtained with the network \"full\"?\nIs there any difference with the end-to-end trained network vs the two-stage training?\nIn the videos, I noticed there is a flickering issue near the borders in the traversability output of the network. It would good to explain this issue in the RELLIS-3D dataset, where is quite evident.\nFinally, it would be relevant for the community if the authors make the network open-source."}}, {"review_id": "N3irB-ibsOC", "reviewer": "Reviewer_dzpe", "summary": "The authors propose a method for LiDAR semantic information aggregation and completion. The target application of their method is for off-road traversability prediction. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nThe authors have constructed a fairly complex system for aggregating semantic information from LiDAR data. The odometry integration is differentiable so that gradients can be backpropagated through time to perform the aggregation. The sparsity of the LiDAR data is overcome partially through this aggregation but also through an in-painting network. The results show that they are able to produce semantic maps over their 4-class ontology that are stable could be useful for off-road robot navigation. \n\nWeaknesses:\n\n[W1] I feel the largest weakness of the work is that the approach is designed for predicting terrain traversability, but the work never deploys the model to evaluate navigation on a mobile robot. Some of the choices that are made about what classes should be deemed \"traversible\" and which classes not will be traversible depends heavily on the specific robot. I'm not sure it will always be clear that some classes should be traversible and others not in fact. It would seem that a requirement for demonstration of the value of their traversability prediction is that it is in fact traversible for some specific robot. Without this, the method is just aggregating semantic information and fusing it with odometry. \n\n[W2] I'm not clear on whether geometry is being used to determine traversability at all. On line \"For the remaining points, we take the point with the lowest traversability at each x, y location as the final traversability label\" it seems not. It would seem that for traversability we need a mix of geometry and semantic information. For example, if grass is a steep hill then it may not be traversible but if it is flat it will be. \n\n[W3] I'm somewhat confused about how the aggregation is accounting for odometry error both in the dataset generation and at inference time. On L 211 you claim that \"BEVNet-R uses learned recurrency to “fix” the errors in odometry\" but I don't understand what this means or even how this is possible. Please clarify/elaborate. \n\n[W4] I find the 3rd claim of contribution overstated. The claim is \" 3) filtering out irrelevant obstacles like overhanging tree branches that do not affect traversability\" which is achieved in the methodology by \"we filter out overhanging obstacles by remov- ing points that are above the ground plane by a certain threshold because they will not collide with the robot.\" This doesn't seem to me to warrant a claim of novelty.  \n\n[W5] I would have like to have seen a comparison against a more traditional SLAM-based fusion approach. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "It is difficult to assess the true usefulness of this approach for real robotics navigation since the model is never shown to work in this setting. The ontology that is used to define traversability is hand-constructed and it is not clear (at least to me) to what degree this output actually maps to traversability of a real robot in a practical setting. It is possible that the work could be re-framed as an approach for semantic fusion and completion for LiDAR data, but as it stands it is hard to evaluate on the use-case that it sets out to achieve, which is traversability.  \n\n-----------------------\n\nPost-rebuttal comments. \n\nThe authors have done an exceptional job trying to address the comments that I raised in the first round. I particularly appreciate the new comparisons with a SLAM-based baseline. I have correspondingly updated my recommendation to \"Weak accept\". \n\nA few further suggestions for the camera-ready version if the paper is accepted. I would suggest to move some of the responses to my (and other reviewers') comments out of the supplementary and into the main manuscript. For example, in my opinion, the comparison with the traditional SLAM approach is much more valuable/insightful than the discussion about early/late aggregation in the ConvGRU. I would also suggest that they update the description of the third claim of novelty to include more of the details in the response here since as stated in the manuscript currently it seems kind of trivial, but only now do I understand that it is really not. Additionally, if the authors' have performed experiments on the Warthog (as evidenced by the project page), it would strengthen the paper to include at least some qualitative evidence of this to strengthen this as a truly applicable and battle-tested method for robotics. \n\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Please address/comment on/rebut the weakness listed above or reframe the work as squarely on semantic information fusion, or demonstrate the applicability of the approach by deploying it on a real robot (probably not possible in the rebuttal time frame).  "}}, {"review_id": "7OKJYW4J8Y4", "reviewer": "Reviewer_2yv4", "summary": "The paper considers traversability estimation for autonomous off-road driving, and develops Bird-eye View Network (BEVNet) which predicts traversability map from sparse Lidar inputs with semantic information (semantic objects are mapped to labels: free, low-cost, med-cost, lethal). The architecture consists of 3d sparse cnn for point cloud processing, ConvGRU for temporal aggregation of scans, and 2d encoder/decoder to produce traversability birds-eye-view map. As parts of the framework, unseen areas of the map can also be predicted. The traversability map can be converted into a cost map for planning purposes. \n\nThe proposed algorithmic aggregates scans, points are semantically mapped to labels, ground plane is estimated, and used as reference for the final projection, before which points of a certain theshold above the ground plane are filtered. For the remaining points, the one with lowest traversability is used as the final 2d projected point label. The vehicle odometry is used to transform and aggregate scans over time. \nThe network is also trained using noisy perturbed inputs, which might help explain why it is later more robust to noise from odometry (and hence the map aggregation). \n\nThe authors include comparisons to Cylinder3d network on two standard datasets -- Kitti and Rellis, demonstrating improved performance. \n\nOther comments:\nWould be useful to explain what Cylinder3d does -- it's not clear from the name that it is an appropriate comparison network. \nHow is the ground truth constructed in the Figures? It seems noiser (and more incomplete) than BEVNet ?", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ general method for semantics+lidar+odometry-based traversability estimation\n+ the qualitative results demonstrate that the method is promising\n+ the authors have performed comparisons using standard data-sets\n\nWeaknesses:\n- the paper claims to address off-road traversability but by explicitly relying on semantics it is extremely difficult to address offroad navigation where aspects such as slope, surface roughness, tire-surface interaction, etc... are not easy to label semantically; the authors might want to limit the scope and be clear that it only applies to a class of environments that can be semantically labeled\n- the dependence on a single ground plane might limit the applications to planar environments\n- is Cylinder3d the only/best comparison framework?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper has interesting ideas about combining geometry and semantics for traversability.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Would be useful to limit the scope when referring to \"off-road\" as noted above. \nPlease clarify and address the comments under Weaknesses above. ", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for traversability estimation from LiDAR point clouds. The reviewers find the paper in general to be clearly written and well structured. The reviewers appreciate the ablation study. Most of the major concerns raised by the reviewers have been addressed in the rebuttal. I thank the authors for incorporating all the suggestions and I recommend acceptance.
Decision: Accept (Poster)