Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks
Authors: Benedikt Mersch
Xieyuanli Chen
Jens Behley
Cyrill Stachniss
Venue: CoRL2021 Poster (2021)
Keywords: Point Cloud Prediction
3D LiDAR
Temporal Convolutional Networks

Abstract: Exploiting past 3D LiDAR scans to predict future point clouds is a promising method for autonomous mobile systems to realize foresighted state estimation, collision avoidance, and planning. In this paper, we address the problem of predicting future 3D LiDAR point clouds given a sequence of past LiDAR scans. Estimating the future scene on the sensor level does not require any preceding steps as in localization or tracking systems and can be trained self-supervised. We propose an end-to-end approach that exploits a 2D range image representation of each 3D LiDAR scan and concatenates a sequence of range images to obtain a 3D tensor. Based on such tensors, we develop an encoder-decoder architecture using 3D convolutions to jointly aggregate spatial and temporal information of the scene and to predict the future 3D point clouds. We evaluate our method on multiple datasets and the experimental results suggest that our method outperforms existing point cloud prediction architectures and generalizes well to new, unseen environments without additional fine-tuning. Our method operates online and is faster than the common LiDAR frame rate of 10 Hz.

Self-supervised Point Cloud Prediction Using 3D Spatio-temporal Convolutional Networks
Benedikt Mersch
Xieyuanli Chen
Jens Behley
Cyrill Stachniss
University of Bonn
(mersch, xieyuanli chen, jens behley, cyrill. strill..tachhiaaaeei.. uni uni-bonn,co de
Exploiting past 3D LiDAR scans to predict future point clouds is a promising method for autonomous mobile systems realize foresighted state es- timation, collision avoidance, and planning. In this paper, we address the prob- lem of predicting future 3D) LiDAR point clouds given sequence of past LiDAR scans. Estimating the future scene on the sensor level does not require any preced- ing steps in localization or tracking systems and can be trained self-supervised. We propose an end-to-end approach that exploits a 2D range image representation of each 3D LiDAR scan and concatenates a sequence of range images obtain architecture
Abstract:
a 3D tensor. Based on such tensors, we develop an using 3D convolutions to jointly aggregate spatial and temporal information of the scene and to predict the future 3D point clouds. We evaluate our method on mul- tiple datasets and the experimental results suggest that our method outperforms existing point cloud prediction architectures and generalizes well to new, unseen environments without additional fine-ture ane-tuning. Our method operates online and is faster than the common LiDAR frame rate of Hz.
1 Introduction
Sequential 3D point clouds obtained from Light Detection And Ranging (LiDAR) sensors are widely used for numerous tasks in robotics and autonomous driving, like localization [1], detection [2, 3], mapping [4], segmentation [5, 6], and trajectory prediction [7]. The ability to forecast what the sensor is likely to in the future can enhance decision-making for autonomous vehicle. A promising application
to use the predicted point clouds for path planning tasks like collision avoidance. In contrast to most approaches, which predict for example future 3D bounding boxes of traffic agents, point cloud prediction does not need any preceding inference steps such as localiza- tion, detection, tracking predict future scene. Running an off- off-the-shelf detection and tracking system on the predicted point clouds yields future 3D object bounding boxes as demonstrated by recent approaches for point cloud forecasting [8, 9]. From machine learning perspective, point cloud prediction is an interesting problem since the ground truth data is always given by the next incoming LiDAR scans. Thus, need for expensive labeling and evaluate its performance online in unknown environments.
train point cloud prediction self-supervised without the
In this paper, we address the problem of predicting large and unordered future point clouds from given sequence of past scans as shown in Fig. 1. High dimensional and sparse 3D point cloud data render point cloud prediction challenging problem that not yet fully explored in the literature. A future point cloud can estimated by applying a predicted future scene flow to the last received scan set of future points. In this paper, focus on the generation of new point clouds to predict the future scene. In contrast to existing approaches [8, 9], which exploit recurrent neural networks for modeling temporal correspondences, the spatial and temporal information. Our proposed approach takes a new 3D representation based on concatenated range images as input and jointly estimates future range image and per-point scores for being a valid or an invalid point for multiple future time steps. This allows us to predict detailed future point clouds of varying sizes with reduced number of parameters to optimize resulting in faster training and inference times. Furthermore, our approach is also fully self self-supervised and does not require any manual labeling of the data.
or by generating
use 3D convolutions to jointly encode
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

T-P+1
T
T+1
T+F
Pre-
diction
Figure 1:
Given a sequence of P' past point clouds (left in red) at time T, the goal is to predict the F' future scans (right in blue). Note that all point clouds are the sensor's coordinate system. Best viewed in color.
The main contribution of this paper is a novel range manage-basede encoder-decoded neural network to jointly process spatio-temporal information from points clouds using 3D convolutions. Our method can obtain structural details of the environment by using skip connections and horizontal consis- tency using circular padding, and the end, provides more accurate predictions than other state- of-the-art approaches for point cloud prediction. We make three main key claims: Our approach able to (i) predict a sequence of detailed future 3D point clouds from a given input sequence by fast joint spatio-temporal point cloud processing using temporal 3D convolutional networks (CNNs), (ii) outperform state-of-the-ar point cloud prediction approaches, (iii) generalize well to unseen environments, and operate online faster than typical rotating 3D LiDAR sensor frame rate Hz). These claims are backed up by the paper and our experimental evaluation. The open-source code and the trained models available at https: httpbbtthhh..... com/ PRBon// point-clous cloud- clooud-redii
2 Related Work
Extracting spatio-temporal information from point cloud sequences has been exploited in the lit- erature for different tasks related to point cloud prediction. Liu et al. [10] extend PointNet++ [6]. which extracts spatial features from single point cloud, and propose FlowNet3D for estimating the 3D motion field between two point clouds. FlowNet3D is used by Lu et al. [11] to interpolate intermediate frames between two LiDAR scans. Other deep learning-based scene flow estimation methods can be found in the literature [12, 13]. In contrast to scene flow between two past frames, point cloud prediction estimates future point clouds. In addition to that, scene flow requires labels point-wise wise correspondences or segmentation masks. This makes learning costly, In contrast to that, we focus on the self self-supervised generation of new, future point clouds.
like point-
Predicting future trajectories of traffic participants based on their past motion is one possible ap- plication of LiDAR-based future scene prediction. Given LiDAR scans only, these methods rely on preceding detection and tracking modules to infer past trajectories. For example, Luo et al. [7] develop 3D convolutional architecture based on bird's eye view (BEV) of voxelized LiDAR scans transformed by the sensor's ego-motion. Inspired by this, Casas et al. [14] increase the predic- tion horizon with IntentNet and also estimate high-level driver behavior from HD maps. Liang et al. [15] integrate a tracking module into the end-to-end pipeline to improve the temporal consis- tency of the predictions while Meyer et [3] and Laddha representation to predict future 3D bounding boxes. A second method to estimate future scene representation without the need for prior object detections is prediction performed
[16] explore the range image
the sensor level. Hoermann et al. [17] directly predict a future occupancy grid map from a sequence of past 3D LiDAR scans and Song et al. [18] predict motion flow for indoor 2D LiDAR maps. Wu et al. [19] use spatio-temporal pyramid network to estimate voxelized BEV classification and fu- ture motion maps. Recently, Toyungyernsub et al. [20] differentiate between dynamic and static cells for grid map prediction but require tracking labels to segment moving cells. In contrast to the aforementioned approaches, our method operates on the full-size point clouds without the need for voxelization. As shown by Weng et al. [8], reversed trajectory prediction pipeline can take the point cloud predictions and detect and track objects in future scans. This makes possible to first train the prediction of point clouds self-supervised and then use off-the-sher detection and tracking modules to get future trajectories.
There has been large interest in vision-based prediction. Srivastava et al. [21] propose an L.STM [22] encoder-decoder model that takes an input sequence of image patches or feature vectors and outputs the sequence of future patches or feature vectors, respectively. Shi et al. [23]

---

ST+1 Mask and Re-project
Spherical Projection
Spatio- Temporal CNN
ST-1
ST+2 Mask and Re-project
Spherical
Projection
ST-P+1
ST
Spherical Projection
Mask and Re-project
Range
Valid Point Mask
Range
Figure 2: Overview of our approach, At time T, the past point clouds are first projected into 2D range image representation and then concatenated. After passing through proposed ppatio-tempora 3D CNN network, the combined predicted mask and range tensors are re-projected obtain the future 3D point cloud predictions.
propose ConvLSTMs that employ convolutional structures in the temporal transitions imposed by the LSTM. This makes it possible to jointly encode temporal and spatial correlations in image se- quences. The integration of 3D convolutions into L.STMs has been researched by Wang et al. [24]- Aigner et al. [25] omit the use of recurrent structures and develop a video prediction approach called FutureGAN based on 3D convolutions combined with a progressively growing GAN. Using the range image representation of LiDAR scans, we transfer the techniques from vision-based methods for solving point cloud prediction. However, there are significant differences between video prediction and range image-basee based point cloud prediction. The main difference is that range images contain 3D point information rather than RGB values. In addition to that, range images have a 360 degrees field of view, which causes strong correlation between the left and right image borders because of the principle of a typical rotating LiDAR sensor, c.g., a Velodyne or an Ouster scanner.
Though vision-based and voxel-based prediction methods have been well studied, predicting full- scale future point clouds still challenging and has not yet been fully explored in existing works. To predict future sequence of unordered and unstructured point clouds, Fan al. [26] propose point recurrent neural network (PoinRNN). Their method models spatio-temporal local correlations of point features and states. Lu et al. [9] introduce MoNet, which integrates point-based context and motion encoder network and processes these features novel recurrent neural network. A 3D scene flow-basee point cloud prediction approach has been recently developed by Deng and Zakhor [27]. All three methods are 3D point-based but only consider a limited number of points. deeoddr network that takes highly compressed feature repre- sentation of a full-scale LiDAR sean and models the temporal correlations with a recurrent auto- encoder. The feature vectors obtained either by point-based architecture 2D convolutional neural networks processing 2D range image representations. Deep learning- hased generation of Li- DAR scans from 2D range image has been investigated by Caccia et al. [28] well. In contrast to existing approaches, we propose a technique that concatenates the projected 2D range images to spatio-temporal 3D representation, which makes possible use 3D convolutions without the need for recurrent architectures for temporal modeling. As discussed by Bai et al. [29] for general sequence modeling, using convolutional networks results an architecture that is easier to train and faster at inference since the input range images are no longer processed sequentially. Compared to MoNet [9], this allows us to predict more future point clouds in shorter amount of time.
Weng et al. [8] propose
3 Our Approach
The goal of our approach is predict future sequence of full-scale LiDAR points given sequence of past scans. As illustrated in the overview of our method in Fig. 2, we first project each past 3D point cloud into range image representation and concatenate the 2D images obtain 3D spatio-temporal tensor, see also Appendix A for details. Second, we pass this tensor to our encoder- decoder architecture to extract the spatial and temporal correlations with 3D convolutional kernels as described in Sec. 3.1. We use skip connections and circular padding to maintain details and horizontal consistency of the predicted range images, see Sec. 3.2. Details on the training procedure are provided in Sec. 3.3.
3

---

3.1 Spatio-Temporal Encoder-Decoder Architecture
Our approach first converts the 3D LiDAR points into spherical coordinates and maps them image coordinates resulting in a dense 2D range image from which 3D information can still be recovered. If no point is projected into a specific pixel, its value is set zero. If multiple points are mapped the same pixel, we keep the closest point to focus on the prediction of foreground objects. For technical details on the range image projection see Appendix A. The main task of the encoder-decoder ar- chitecture illustrated in Fig. 3 is to jointly extract spatio-temporal features from the input sequence and output future range image predictions. Compared to a multilayer perceptron architecture that disregards spatial and temporal dimensions, the use of convolutions imposes an inductive bias of having local correlations along these dimensions. Convolutional networks usually require less trainable parameters and are less prone to overfitting, When using range images for point cloud pre- diction with 3D convolutions, sufficient receptive fields along the spatial and temporal dimensions between encoder and decoder are required to capture the motion of points in the past frames and to propagate their locations into the future range images.
The encoder takes the 3D input tensor of size (P,, H, W), containing P' range images with height H and width W. and first standardizes the range values based on mean and standard deviation com- puted from the training data. We pass the standardized tensor through a 3D convolutional input layer with C' kernels to get a C--caanel feature representation size (C, P. H, W). Inspired by Fu- tureGAN [25], the encoder block receives tensor of size (Ci, Pt. Hi, Wi) at each encoder stage I, applies combination of 3D convolution, 3D batch normalization (BN) [30]. and leaky ReLU ac- tivation function [31] while maintaining the size of the tensor. A strided 3D convolution follows Hi Wi where h1 and iy are predefined resulting in a downsampled tensor of size (CI+1, Pt hy wy
downsampling factors for the spatial feature size and pt reduces the temporal feature dimension. We batch normalize the resulting tensor and apply leaky ReLU activation function. The kernel size is (Pt, h4, wi) and the stride (1, h4. wy) to achieve the desired downsampling. The downsam- pling compresses the sequential point cloud feature representation and forces the network to learn features.
meaningful spatio-tempoll
To predict future range images for F future time steps, the decoder subsequently upsamples the feature tensor to the final output size (2, F, H. W). Note that the number of future range images is fixed in the architecture, but longer prediction horizons can be achieved in an auto-regressive manner. This is done by sequentially feeding back the predicted range images as the input tensor. In this work, we will only focus on the prediction of a fixed number of future point clouds. The decoder architecture is a mirrored version of the encoder. First, a transposed 3D convolutional layer with kernel size (pt, hy, wy) and stride (1, hy, wy) increase the feature map. We insert 3D batch normalization and leaky ReLU activation layer before a second 3D CNN, 3D BN, and leaky ReLU follows. Finally, we pass the output tensor of size (C, F, H, W) through a 3D CNN output layer with two kernels of size (1, 1, 1) and stride 11 1) and apply the final Sigmoid function to get normalized values between 0 and 1. The first output channel maps to a predefined range interval resulting in the future range predictions. As done by Weng et al. [8], the second channel contains probability for each range image point to be a valid point for re-projection. The re- re-projection mask keeps all points with a probability above 0.5. This makes it possible to mask out e.g. the sky since there are no ground truth points available.
3.2 Skip Connections and Horizontal Circular Padding
We use strided 3D convolutions and downsample the range images as described in See. 3.1. The reduced size of the feature space however causes a loss of details in the predicted range images. We address this problem by adding skip connections [32] between the encoder and decoder to maintain details from the input scene. As shown in Fig. 3, the feature maps bypass the remaining encoding steps and the mirrored decoder stage concatenates them with the previously upsampled feature vol- ume along the channel dimension. Concatenation enables the network to account for the temporal offset between encoder and decoder feature maps. A combination of 3D convolutions, 3D batch nor- malization, and leaky ReLU follows to merge the features back to the original number of channels while maintaining the temporal and spatial dimensions. We investigate the effect of skip connections in Sec. 4.1 and Sec. 4.4 and show that they maintain details in the predicted point clouds.

---

Predicted Mask Logits
Stacked Range Images
Predicted Range
(16.5 616,5.64.2 (6.5.64.2048)) 2048)
(32,4,22,12)
(64.3, 16, 256
(128,2.8,32 32)
1256, 1,488
(256. 1-488)
(256 1.4.8
(128.2,8,32 3.33)
(128,2,8 0,8,2,8,322 32)
(64.3 169,6692266
(64,3,169,250
(32,4,2,5122
(32,4,22552))
(16.5 6,.64.44 2048)
(512.1.2,2)
CNN K(3.3.3) I(I. P(L.I)) BN Leu
Concat. 3D CNN K(3.3.3) S(1.1.1) P(I...) ReUU
Standardization
+ 3D K(LJ1) SCL.D)
3D CNN (((1.2).2.4) S(1.2.4) 3D BN LeukyReLU
ConvTransposeie ( (1.2).2,44 S(1.2,4) 3D BN LeakyReLLU
3D CNN K(L.I) S(L.II)
Siggmodd
Figure 3: Our detailed spatio-temporal architecture using 3D convolutional neural networks with feature map sizes (C., Ps, H., W.). Solid and dashed arrows indicate information flow and skip connections, respectively. Details of each block are in the colorized boxes with kernel sizes K, stride S and padding P. The feature map not padded in case is specified. To be viewed in color.
Another challenge when using range images for 3D point cloud prediction to maintain spatial con- sistency the horizontal borders of the range images. Range images obtained by rotating LiDAR sensors such as a Velodyne or an Ouster sensor, are panoramic images and have strong horizontal correlations between the image borders. If the ego vehicle is rotating along the vertical axis, object passing the left border of the range image will appear on the right border. To take this prop- erty into account, introduce circular padding for the width dimension. We pad the left side of each feature map with its right side and vice versa. The vertical dimension is padded with zeros. An experiment the effectiveness of this padding is provided in Sec. 4.1.
3.3 Training
When training the network, we project the ground truth point clouds into the range images of size (H, W), which allows for the computation of 2D image-based losses. We slice the data into samples of sequences consisting of P past and F future frames, where subsequent samples are one frame apart. Our architecture is trained with combination of multiple losses. The average range loss LR. at time step t between a predicted range image (frz), E k and a ground truth range image (ri.), E TR. HWW is given by
LR.1
1 HW
Σ
Arijo with Arts =
|l rarll rf >0 0
otherwise
such that the range loss is only computed for valid ground truth points. We train the mask output at time step with predicted probabilities (mi,), E R with binary cross-entropy loss
CM.E = HW
Σ 1
Yo. log(m), - (1 yo.) log(1 mi;,))
(2)
where Vi.j is 1 in case the ground truth point is valid and 0 otherwise. Both losses only consider the predicted 2D range images and not the re- re-projected point clouds to compute the loss faster. For comparison 3D-based methods like MoNet [9], we fine-tune our model 3D point-based loss. A common metric for comparing 3D point clouds is the Chamfer distance [33]. At time I, we re- project the masked range image into 3D point cloud S; = { E 333, | = N, and compare it with the ground truth point cloud S, = {p 3R [S|] M,, by computing
LCD.
1
Σ
min PES,
pll
Σ min 1 pll- PES,
AES.
The computation the Chamfer distance is in general slow due to the search for nearest neighbors. A fast to compute image-based loss using LR, and LM is about 2.5 times faster compared to loss
5

---

including the 3D Chamfer distance. We propose training scheme with only range and mask loss for pre-training. This results in good initialization for fine-tuning including a Chamfer distance loss. We provide an experiment on the training scheme in Sec. 4.4. Given a current time step / = T, our total loss function for F future time steps is given by
T+F
C=
Σ T=
LR. + aMLMI acDLeDa.
(4)
with tunable weighting parameters ay and aCD. During the experiments, we experienced that setting OM = 1.0 and aCD = 0.0 for pre-training and oM = 1.0 and aCD = 1.0 for fine-tuning worked best. We train our network parameters with the Adam optimizer [34], a learning rate of 10-3, and
accumulate the gradients for 16 samples. We use an exponential decaying learning rate schedule. All models require less than 50 epochs to converge.
4 Experimental Evaluation
In this work, we present an approach for self supervseee point cloud prediction using apatio-temporal 3D convolutional neural networks. We present our experiments to show the capabilities of our method and to support our key claims, which are: (i) predicting a sequence of detailed future 31) point clouds from a given input sequence by a fast joint spatio-temporal point cloud processing using temporal 3D convolutional networks, (ii) outperforming state-of-the- art point cloud prediction approaches, (iii) generalizing well to unseen environments, and operate online faster than a typical rotating 3D LiDAR sensor frame rate of Hz We follow the experimental setting of MoNet [9] with the KITTI Odometry dataset [35] and predict the next 5 frames from the past 5 LiDAR scans captured at 10 Hz. Please see Appendix for further details on the experimental setting.
4.1 Qualitative Results
The first experiment supports our claim that our model can predict a sequence of detailed future 3D point clouds using a temporal convolutional network and demonstrate the effectiveness of our proposed skip connections and circular padding. The situation in Fig. 4 from test sequence 08 shows an ego-vehicle finishing a left turn while passing a close vehicle. This is challenging scenario because of the different scales of close and far objects and the ego-motion of the car. Our approach pre-trained on range and mask loss only is capable of predicting the future point clouds across 5 time steps and maintains details in the scene. Without skip connections, small objects are lost due to the compression (green circle). With typical zero padding, the points belonging to the car in the back of the ego vehicle (black circle) are not consistently predicted since they span across the range image borders, while our approach can solve this problem by using the proposed circular padding.
4.2 Quantitative Results
To support the claim that our method outperforms state-of-the-ar point cloud prediction approaches, compare the quantitative results to multiple baselines. Note that Lu et al. [9] operate and also evaluate on sab-sampled point clouds, whereas our approach can predict full-size point clouds. To account for this, Tab. (Lefi) shows the performance of our method on sampled point clouds with two different sampling rates used in MoNet [9]. For both samplings, our method achieves a better result for larger prediction steps. However, it can be seen that sampling points from the original point clouds during evaluation strongly affects the Chamfer distance of our full-sized prediction. Thus, we provide an evaluation on full-scale point clouds demonstrate the true performance of our approach, see Tab. (Right). We compare our results to an Identity bascline that takes the last received scan as a constant prediction for all future time steps. The Constant Velocity and Ray Tracing baselines estimate the transformation between the last two received scans with SLAM approach [36] as a constant velocity prediction of the sensor. At each prediction step, the Constant Velocity baseline transforms the last received point cloud according to the predicted motion. The Ray Tracing baseline aggregates all five transformed past point clouds to render a more dense range image. Since a lot of points in the scene are static, these are very strong baselines that work well for constant ego- motion. Note that contrast to our method, all three baselines do not consider points belonging to moving objects. The results in Tab. show that our method outperforms all baselines at larger

---

Ours (Pre-trained)
No Circular Padding
No Skip Connections
T+1
T+1
T+1
T+5
T+5
T+5
Figure 4: Qualitative comparison of our pre-trained method and two ablated models with predicted (bluc) and ground truth (red) point clouds from test sequence 08. Given past sequence of point clouds time the upper row shows the first predicted frame at T+1 and the lower row the last predicted frame T+5. The green circle encloses an area where details are lost without skip connections and the points in the black circle demonstrate that circular padding maintains spatial consistency at the range image border. Best viewed in color.
Sampled Point Clouds
Full-scale Point Clouds
Prediction Step
Point [26]
Scene
MoNet
Ours
Ours
Identity Bascline
Const. Vel.
Ray Tracing
[9]
[9]
32,768 pts
65,536
Baseline
Baseline
0.332
0.503
0.278
0.367
0.288
0.271 (0.144)
0.105 (0.067) 0.158 (0.095) 0.254 (0. 124)
0.561
0.854
0.409
0.446
0.352
0.719 (0.448)
0.217 (0.221)
0,253 (0.246)
0.310 188)
0.810
(0.798)
0.385
0,388 (0.460)
0.378 (0.262)
1.054
1.913
0.692
0,638
0.509
1.727 (1.169)
0.604 (0.711)
0.556 (0.719)
0.448 (0.370)
1,299
2.610
0.842
0.763
0.615
2.240 (1.156)
0.852 (0.968)
0.751 (0.971) 0.547 (0.487)
Mcan (Sid)
0.811
1.440
0.554
0.552
0.439
1.235 (1.192)
0.433 (0.641)
0.421 (0.627) 0.387 (0.331)
Table 1: Chamfer distance results on KITTI Odometry test sequences 08 to 10 in [m²] with sampled (leff) and full-scale (right) points clouds. We first train our model range and mask loss and then fine-tune including Chamfer distance loss for 10 epochs described in Sec. 3.3. Best mean results in hold for sampled and full-scale point clouds, respectively. For full-scale point clouds, standard deviation of Chamfer distance is given in parentheses. Please see Appendix E for a more detailed statistical analysis.
prediction steps and that it can reason about the sensor's ego-motion and the motion of moving points. The standard deviation the Chamfer distance indicates that the results of our approach are more consistent throughout the test set whereas the baseline results suffer from outliers in case of moving objects or non-linear ego-motion.
4.3 Runtime
For the runtime, our approach takes average 11 ms Hz) for predicting 5 future point clouds with up to 131,072 points each on a system with an Intel i7-6850K CPU and an Nvidia GeForce RTX 2080 Ti GPU, which is faster than the sensor frame rate, i.e. 10 of typical rotating 3D LiDAR sensor and faster than existing approaches like MoNet [9] using the same GPU. Compared to runtime of 280 reported by Lu al. [9] for MoNet for predicting 5 future point clouds with 65,536 points each, our approach is 25 times faster for twice as many points.
4.4 Ablation Study
In this experiment, we provide a more detailed analysis of the effect of the model architecture and the proposed training scheme. We average the metrics over all validation samples and time steps as shown in Tab. 2. Note that we always report the Chamfer distance but it is only used during training if specified. First, we modify the size of the network by decreasing (Small) or increas- ing (Large) the number of channels of our model by factor of 2. The larger model predicts slightly better masked range images but also requires 4 times more parameters. All losses increase without skip connections (No Skip Connections). This is because a lot of objects are not predicted due to the large feature compression. The effect of circular padding (No Circulur Pudding) is not visible quantitatively because of the small number of affected points and we refer to Sec. 4.1 for a quali- tative evaluation. Finally, we show that our fine-tuned model (Ours) achieves similar performance
7

---

Small
Large
No Skip Connections
No Circular Padding
CD Loss
Ours (Pre-trained)
Ours
Property
4.M Paramelers
X
17 Parameters
X
X
X
68 Parameters
X
Skip Connections
X
X
X
X
Circular Padding
x
x
x
X
Loss Weight cico
0.0
0.0
0.0
0.0
1.0
0.0
0.0 - 1.0
CR [m]] Extsk
0.867
0.741
1,258
0.801
0.805
0.798
0,858
0.309
0.288
0.337
0.297
0.300
Ccu [m*]
1.110
0.714
2.759
0.885
0.487
0.985
0.480
Table 2: Evaluation of models with varying architectures, padding, and training schemes the KITTI Odom- etry validation sequences 06 and 07. Each variation (columns) is given by combination of different design decisions (rows) indicated by crosses (X), e.g., variation "Large" is model with M parameters using skip connections and circular padding. The arrow (-> indicates a pre-training and fine-tuning with two different loss weights cep as discussed in Sec. 3.3.
Figure 5: Predicted (blue) and ground truth (red) future point clouds on the Apollo-SouthBay Columbia- Park [37] test set at prediction step 5. Left: Model only trained on KITTI odometry data. Right: After training epoch the Apollo training set.
on the validation set compared to model optimized with the Chamfer distance loss from the be- ginning (CD Loss From Start). Our proposed training scheme with pre-training on fast-to-compute image-based losses gives good pre- pre-trained model for fine-tuning with the 3D Chamfer distance loss while reducing training time.
4.5 Generalization
Finally, we aim at supporting our claim that our method generalizes well new, unseen environ- ments. We first evaluate a model fine-tuned on KITTI Odometry on the Apollo-SouthBay Col- umbiaPark [37] test set. Note that this data is from an unseen, different environment (collected in the U.S.) with different type of car setup, comparing to the training data (collected in Germany). Our model achieves a mean Chamfer distance of 0.934m² m² on the Apollo test set. The performance gap compared to the results in Tab. 1 is due to the fact that the training data has maximum range of 85 m but the Apollo test data contains points up to a range of 110 m. Thus, we fine-tune the model on the mean Chamfer distance of 0.426 m² and demonstrates the generalization capability of our approach and the advantage of self-supervised training. We provide additional experiment the effect of using more training data Appendix C. Fig. 5 shows that the fine-tuned model reduces Chamfer distance by predicting more accurate shapes and more distant points. Note that compared to Fig. 4, the different sensor mount causes a changed pattern of invalid points around the car. Our approach can correctly predict the re-projection mask.
ColumbiaPark [37] training set for single epoch. This yields
5 Conclusion
In this paper, we presented novel approach to self- self-supervised point cloud prediction with 3D con- volutional neural networks. Our method exploits the range image representation to jointly extract spatio-temporal features from the input point clouds sequence. This allows to successfully predict detailed, full-scale, future 3D point clouds during online operation. We implemented and evaluated approach on different datasets and provided comparisons to other existing techniques and sup- ported all claims made in this work. The experiments suggest that our method outperforms existing baselines and generalizes well to different sensor mounts in unseen environments.
8

---

Acknowledgments
We thank Lu et al. [9] for providing implementation details and experimental results.
This work has partially been funded by the Deutsche Forschungsgemmiinccaatt (DFG, Ger- man Research Foundation) under Germany's Excellence Strategy, EXC-2070 - 390732324 PhenoRob, by the European Union's Horizon 2020 research and innovation programme under grant agreement No 101017008 (Harmony), and by the Federal Ministry of Food and Agricul- ture (BMEL) based on decision of the Parliament of the Federal Republic of Germany via the Federal Office for Agriculture and Food (BLE) under the innovation support programme under funding no 28DK108B20 (RegisTer).
References
X. Chen, Vizzo, T. Läbe, Behley, and C. Stachniss. Range Image-based LiDAR Localization for Autonomous Vehicles. Proc. of the IEEE Inl. Conf. on Robotics AAttomation (ICRA), 2021.
[2] T. Yin, X. Zhou, and Krahenbuhl. Center-Based 3D Object Detection and Tracking. In Proc. of the IFFF/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2021.
[3] G. P. Meyer, J. Charland, S. Pandey, A. Laddha, S. Gautam, C. Vallespi- and C. K. Wellington. LaserFlow: Efficient and Probabilistic Object Detection and Motion Forecasting. In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), volume 6, pages 526-533, 2021.
[4] Vizzo, X. Chen, N. Chebrolu, Behley, and C. Stachniss. Poisson Surface Reconstruction for LiDAR Odometry and Mapping. In Proc. of the IEEE Inil. Conf. on Robotics & Automation (ICRA), 2021.
[5] X. Chen, S. Li, B. Mersch, L. Wiesmann, Gall, J. Behley, and C. Stachniss. Moving Object Seg- mentation in 3D LiDAR Data: A Learning-haseed Approach Exploiting Sequential Data. arXiv preprint, abs/2105.08971, 2021.
[6] C. Qi, K. Yi, H. Su, and L. J. Guibas. PointNet++: Deep Hierarchical Feature Learning on Point Sets in Metric Space. In Proc. of the Conference on Neural Information Processing Systems (NeurIPS), 2017.
(7) W. Luo, B. Yang, and R. Urtasun. Fast and Furious: Real Time End-to-End 3D Detection, Tracking and Motion Forecasting with Single Convolutional Net. In Proc. of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), pages 3569-3577, 2018.
[8] X. Weng, J. Wang. S. Levine, K. Kitani, and N. Rhinehart. Inverting the Pose Forecasting Pipeline with SPF2: Sequential Pointcloud Forecasting for Sequential Pose Forecasting. In Proc. of the Conf. on Robot Learning (CoRI.), 2020.
F. Lu, G. Chen, Liu, Z. Li, Qu, and T. Zou. MoNet: Motion-based Point Cloud Prediction Network. arXiv preprint, abs/201 10812, 2020.
[10] X. Liu, C. Qi, and I.. J. Guibas. FlowNet3D: Learning Scene Flow in 3D Point Clouds. In Proc. of the JEEF/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2019.
[11] F. Lu, G. Chen, Qu, Li, Y. Liu, Knoll. PointlNet: Point Cloud Frame Interpolation Network. In Proc. of the Conference on Advancements of Artificial Intelligence (AAAI), 2021.
[12] A. K. Ushani and M. Eustice. Feature Learning for Scene Flow Estimation from LIDAR. In Proc. of the Conf. on Robot Learning (CoRL), pages 283-292, 2018.
[13] Zhai, X. Xuezhi, L. Ning, X. Kong. Optical Flow and Scene Flow Estimation: A Survey, Pattern Recognàtion, 114:107861, 02 2021.
[14] S. Casas, W. Luo, and Urtasun. IntentNet: Learning Predict Intention from Raw Sensor Data. In Proc. of the Conf. on Robot Learning (CoRL), pages 947-956, 2018.
[15] Liang, Yang. Zeng, Chen, Hu. Casas, and R. Urtasun. PnPNet: End-to-End Perception and Prediction With Tracking in the Loop. In Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2020.
[16] A. Laddha, S. Gautam, G. P. Meyer, and C. Vallespi-Gonzalee RV-FuseNet: Range View based Fusion of Time-Serie LiDAR Data for Joint 3D Object Detection and Motion Forecasting. arXiv preprint, abs/2005. 10863, 2020.

---

[17] S. Hoermann, M. Bach, and K. Dietmayer: Dynamic occupancy grid prediction for urban autonomous driving: A deep learning approach with fully automatic labeling. In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), 2018
[18] Y. Song, Y. Tian, G. Wang, and M. Li. 2D LiDAR Map Prediction via Estimating Motion Flow with GRU. In Proc. of the IEEE Intl. Conf. on Robotics & &Automation (ICRA), 2019.
[19] P. Wu, S. Chen, and D. N. Metaxas. MotionNet: Joint Perception and Motion Prediction for Autonomous Driving Based on Bird's Eye View Maps. In Proc. the IEEE/CVF Conf. on Computer Vision and Pottern Recognition (CVPR), 2020.
[20] Toyungyernsub, M. Itkina, R. Senanayake, and M. Kochenderfer. Double-Prond ConvLSTM for Spatiotemporal Occupancy Prediction in Dynamic Environments. In Proc. of the IEEE Intl. Conf. on Robotics & Automation (ICRA), 2021.
[21] N. Srivastava, E. Mansimov, and R. Salakhutdinov. Unsupervised Learning of Video Representations using LSTMs. In Proc. of the Int. Conf. on Muchine Learning (ICML), 2015.
[22] S. Hochreiter and Schmidhuber. Long Short-Term Memory. Neural Computation, 98)):1735-778,, 1997.
[23] X. Shi, 7. Chen, H. Wang. D.-Y. Yeung, W.KK. Wong, and W.-C. Woo. Convolutional L.TTM Network: Machine Learning Approach for Precipitation Nowcasting. In Proc. of the Advances in Neural Informa- tion Processing Systems (NIPS), 2015.
[24] Y. Wang, L. Jiang. M.-H. Yang. L.-J. Li, M. Long, and L. Fei-Fei. Eidetic 3D LSTM: A Model for Vidco Prediction and Beyond. Proc. of the Int. Conf. Learning Representations (ICLR), 2019.
[25] S. Aigner and M. Korner. FutureGAN: Anticipating Future Frames of Video Sequences using Spatio- Temporal 3d Convolutions in Progressively Growing GANs. In Proc. of the IEEE/CVF Conf. on Computer Vision and Puttern Recognition (CVPR), 2018.
[26] H. Fan and Y. Yang. PoimRNN: Point Recurrent Neural Network for Moving Point Cloud Processing. arXiv preprint, abs/1910.08287, 2019.
[27] D. Deng and A. Zakhor. Temporal LiDAR Frame Prediction for Autonomous Driving. In Proc. of the Intl. Conf. on 3D Vision (3DV), pages 829-837, 2020.
[28] L. Caccia, H. Van Hoof, A. Courville, and Pincau. Deep Generative Modeling of Lidar Data, Proc. of the JEEE/RSJ Ina. Conf. on Intelligent Robots and Systems (IROS), 2019.
[29] S. Bai, J. Kolter, and V. Koltun. An Empirical Evaluation of Generic Convolutional and Recurrent Net- works for Sequence Modeling, arXiv preprint, abs/1803.01171, 2018
[30] S. loffe and C. Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift, arXiv preprint, abs/1502.03167, 2015.
[31] A. Maas, A. Hannun, and A. Ng. Rectifier Nonlinearitics Improve Neural Network Acoustic Models. In Proc. of the Int. Conf. on Machine Learning (ICML), 2013.
[32] O. Ronneberger, P.Fischer. and T. Brox. U-Net: Convolutional Networks for Biomedical Image Segmen- tation. In Medical Image Computing and Computer-Assisted Intervention, volume 9351 of LNCS, pages 234-241. Springer, 2015.
[33] H. Fan, H. Su, and L. Guibas. A point set generation network for object reconstruction from a single image. In Proc. of the IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), pages 605 613, 2017.
[34] D. Kingma and J. Ba. Adam: A Method for Stochastic Optimization. In Proc. the the Int. Conf. on Learning Representations (ICLR), 2015.
[35] A. Geiger, P. Lenz, and Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Bench- mark Suite. In Proc. of the IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3354-3361, 2012.
[36] J. Behley and C. Stachniss. Efficient Surfel-Base SLAM using 3D Laser Range Data in Urban Environ- ments. In Proc. of Robotics: Science and Systems (RSS), 2018.
10

---

[37] W. Lu, Y. Zhou, G. Wan, S. Hou, and S. Song. L3-Net: Towards Learning Based LiDAR Localization for Autonomous Driving, In Proc. of the IEEE/CVF Conf. Computer Vision and Puttern Recognition (CVPR), 2019.
[38] A. Milioto. 1. Vizzo, Behley, and C. Stachniss. RangeNet++: Fast and Accurate LiDAR Semantic Segmentation. In Proceedings of the IEEE/RSJ Int. Conj. on Intelligent Robots and Systems (IROS), 2019.
[39] Chen, Milioto, Palazzolo, Giguère, J. Behley, and Stachniss. SuMa++: Efficient LiDAR- based Semantic SLAM. In Proc. of the JEEE/RST Intl. Conf. on Intelligent Robots and Systems (IROS), 2019.


Reviews:
[{"review_id": "pdD_FubCnw", "reviewer": "Reviewer_puBM", "summary": "The paper proposes a range image based encoder-decoder architecture to predict future pointclouds. A temporal stack of range images is used to predict another stack of future range images. The experiments evaluate the performance of the architecture in both quality and computational speed. The impact of the different architecture's components is also evaluated.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The proposed method uses common ideas and techniques from image based deep learning with typical range image representations. While the components used are not novel per se, the execution and description of the approach are easy to read and follow, with the method itself being technically sound.\n\nThe most prominent aspect that could be improved in the method description is to provide more insight and discussion into why specific aspects are used. For example, the method uses simple convolutions to capture temporal aspects via the \"depth\" of the image stack instead of LSTM or other mechanisms. What are the expected benefits and drawbacks of this? The encoder-decoder structure is also not exactly the typical reconstruction focused one, as the reconstructed image is not the same as the input. What is the reason this architecture was chosen over other plausible alternatives, such as a pure reconstruction encoder-decoder whose low-dimensional embedding is then used for future prediction via an MLP architecture? One aspect that was not fully clear is whether or not the receptive field in \"time\" is different from the spatial one.\n\nThe losses used seem sensible, and the explanation for their use is clear. One common issue with such compound losses is that the different components require scaling or tuning with respect to each other. Based on the description in the paper, this is not the case here. Is there a particular reason, and were any tests done to see how sensitive the results are regarding such weighting?\n\nThe experiments overall are well done and address questions that are pertinent to the method. My biggest complaint would be the lack of proper statistical information, such as standard deviation or better quantiles. Having this information would provide greater insight into the range of outcomes the various methods produce instead of a singular value such as the mean that is easily affected by outliers. The ablation study is welcome and informative, however also somewhat confusing. The methods labelled \"CD Loss From Start\" and \"Ours\" are identical based on their feature sets but have different performances. From the text, I could not decipher the difference between them as the names used do not line up with the headings in the table. \n\nWhile the paper suggests that the qualitative results shown in Figure 4 showcase the improved performance of the proposed method, I cannot see that. To me, all images look equally \"bad\" regarding their ability to reconstruct the scan. This is likely due to a 3D point cloud being projected onto a 2D image, making them significantly harder to visualize. I do not doubt that the differences exist, but rather the visualization does not convey this convincingly enough.\n\nFinally, one of the motivations at the beginning of the paper was predicting point clouds for tracking dynamic objects. In light of this, it would have been perfect to evaluate the predictive quality for such a task or at least the quality of the prediction for objects of interest, as opposed to the entire point cloud.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall the paper presents a sound method that achieves the goals outlined. The approach uses well known and understood components to achieve this but could benefit from a more thorough discussion of these choices. The experiments demonstrate the performance of the method adequately. However, they could be improved by providing a proper statistical evaluation of the results.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "- Lack of statistical evaluation of the results\n- Could benefit from a more thorough discussion of architecture design choices", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "pJ8Ly1K2Gz8", "reviewer": "Reviewer_2bTD", "summary": "The paper considers the problem of predicting future 3D LiDAR point clouds. It proposes a model that first encodes a single 3D LiDAR scan into a 2D representation, then forms a 3D tensor by concatenating the 2D representations across time, and applies a 3D CNN encoder-decoder architecture. The training does not require any labeled data, but just LiDAR scans across time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: The proposed idea is simple and is reported to perform better than existing approaches for point cloud prediction. The proposed model can predict faster than the LiDAR sensor frame rate of 10 Hz, which is a meaningful consideration when it comes to practicability of the proposed solution.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper considers the problem of predicting future 3D LiDAR point clouds and proposes a very simple architecture. The proposed model is reported to outperform some of the methods in the literature. However, the limitations occur on two counts: \n\n(1) The paper does not discuss how far it can predict. Would the model complexity increase if it is asked to predict for the future 100 frames, instead of just 5? How would that compare with the other models - that for example use LSTM (as LSTM can track and predict over longer time horizons)? How would this impact the time required to predict - would it still be within 10 Hz? \n\n(2) The proposed model seems like a simple application of an existing 3D convolutional encoder-decoder architecture (with slight modification). There isn't much novelty. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "1. Figure 1 does not show any perceptible change in the predicted cloud. The authors can think of showing this in another way?\n\n2. The 3D LiDAR scan to 2D representation is a critical part of the architecture and should be included in the main part of the paper. It is not possible to follow the paper without it.\n\n3. The notation P H W is not explaining in the main paper, the first time it is used. \n\n4. The method is used to predict only the next 5 frames (about 0.5 seconds) from the past 5 LiDAR scans (about 0.5 seconds). This seems like a very short time span for the KITTI dataset. It would have been good if the paper had discussed more the limitations of the model towards extending the prediction horizon.\n"}}, {"review_id": "DWY884RUSm", "reviewer": "Reviewer_n5qa", "summary": "The paper proposes a self-supervised method for predicting future LiDaR point cloud measurements, using a encoder-decoder type neural network. Data from past point cloud frames is first converted into 2D range images, stacked along the time dimension to obtain a 3D tensor. This tensor is an input to an 3D convolutional encoder network. The latent representation is upsampled by the decoder module to produce a stack of predicted future range images. Skip connections from intermediate layers of the encoder to the decoder are added to maintain spatial and temporal details. The method is pre-trained using fast to compute image-based losses, and fine-tuned using the Chamfer loss on the predicted point clouds. Empirically, the proposed method performs strongly compared to state of the art, with ablation studies backing up the decision choices. Finally, the generalization capabilities of the method to new environments are demonstrated.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The method is technically solid, with many interesting innovations described in dealing with point cloud data as range images, processing them using 3D CNNs (esp. skip connections and circular padding), and the self-supervised training method with pre-training and fine-tuning phases. Empirically, the outcome is a comparably fast system that is simple to train. I expect that this work will have a significant impact in this area of research.\n\n- The empirical results are stronger than current state of the art, especially for longer prediction steps. The ablation study in Sect. 4.4. clearly shows the usefulness of the different design decisions, in particular that the skip connections and circular padding are very helpful.\n\n- The paper is overall very clearly written and presented, and an enjoyable read.\n\nLimitations:\n\n- Even if prediction accuracy measured by Chamfer distance are better for the proposed method, it is not clear if this necessarily translates to improved performance in downstream tasks. The results of this paper could be further strengthened by demonstrating performance improvements over other methods on some downstream tasks using the predicted point clouds.\n\n- While the ablation studies are sufficient, it could have been nice to explore some further aspects. Longer prediction horizons might be interesting, as the current 5 steps corresponds to 0.5 seconds only, as far as I understand. Sensitivity to noise or other disturbances in the input data could be explored.\n\n- The generalization study is nice, but the results here are not very thorough. This aspect might be interesting to explore more deeply in future work.\n\n\nOther comments/suggestions/minor:\n- Sect. 4.1. could be moved to after the main ablation results. I think this would be clearer, showing qualitative examples after demonstrating the usefulness of the two additions quantitatively.\n- Fig 2 caption: rang --> range", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper presents a strong contribution for point cloud prediction. The empirical study is thorough, demonstrating all necessary aspects of the method. While there are shortcomings, it can be argued that they will be addressed in follow-up works. My only reason for a weak acceptance recommendation only is that I lack the expertise in this particular area to take a stronger stand.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "I would be happy to have comments from the authors on the limitations I mentioned, and hope the minor corrections can be implemented.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews are generally positive. The AC endorses the reviewers' recommendation. The authors are encouraged to address questions and concerns raised in the reviews.
Decision: Accept (Poster)