TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories
Authors: Jannik Zürn
Sebastian Weber
Wolfram Burgard
Venue: CoRL 2022 Poster (2022)
Keywords: Knowledge Distillation
Semantic Segmentation
Navigation

Abstract: Robustly classifying ground infrastructure such as roads and street crossings is an essential task for mobile robots operating alongside pedestrians. While many semantic segmentation datasets are available for autonomous vehicles, models trained on such datasets exhibit a large domain gap when deployed on robots operating in pedestrian spaces. Manually annotating images recorded from pedestrian viewpoints is both expensive and time-consuming. To overcome this challenge, we propose \textit{TrackletMapper}, a framework for annotating ground surface types such as sidewalks, roads, and street crossings from object tracklets without requiring human-annotated data. To this end, we project the robot ego-trajectory and the paths of other traffic participants into the ego-view camera images, creating sparse semantic annotations for multiple types of ground surfaces from which a ground segmentation model can be trained. We further show that the model can be self-distilled for additional performance benefits by aggregating a ground surface map and projecting it into the camera images, creating a denser set of training annotations compared to the sparse tracklet annotations. We qualitatively and quantitatively attest our findings on a novel large-scale dataset for mobile robots operating in pedestrian areas. Code and dataset will be made available upon acceptance of the manuscript.

TrackletMapper: Ground Surface Segmentation and Mapping from Traffic Participant Trajectories
Jannik Zürn 1 Sebastian Weber
Wolfram Burgard?
University of Freiburg
University of Technology Nuremberg
zuern0ca uni wwr.effreiibrdd.e de, wolfram. burgarddutn.de de
Abstract: Robustly classifying ground infrastructure such as roads and street crossings is an essential task for mobile robots operating alongside pedestrians. While many semantic segmentation datasets are available for autonomous vehi- cles, models trained on such datasets exhibit a large domain gap when deployed on robots operating in pedestrian spaces. Manually annotating images recorded from pedestrian viewpoints is both expensive and time-consuming. To overcome this challenge, we propose TracklerMapper, a framework for annotating ground surface types such as sidewalks, roads, and street crossings from object track- lets without requiring human-annotated data. To this end, we project the robot ego-trajectory and the paths of other traffic participants into the ego-view camera images, creating sparse semantic annotations for multiple types of ground sur- faces from which ground segmentation model can be trained. We further show that the model can self-distilled for additional performance benefits by aggre- gating ground surface map and projecting it into the camera images, creating denser set of training annotations compared to the sparse tracklet annotations. We qualitatively and quantitatively attest our findings a novel large-scale dataset for mobile robots operating pedestrian areas. Code and dataset will be made available at
cs. umi-freiburg de.
Keywords: Knowledge Distillation, Semantic Segmentation, Navigation
Figure I: We present TrapkletMapper, a novel approach for ground surface segmentation that lever- ages observed traffic participant tracklets to supervise a surface segmentation model. Our pipeline automatically annotates obstacles (red), pedestrian areas/sidewalks (yellow), street crossings (green) and roads (bluc) based on the observed trajectories of pedestrians and vehicles.
"Equal contribution
6th Conference on Robot Learning (CoRL 2022), Auckland, New Zealand.

---

Introduction
Mobile ground robots operating in urban areas encounter wide range of environments. It is sential for autonomous robots to robustly navigate through such environments even without access to
map data. Urban environments feature different types of ground surfaces stricted for use only by particular traffic participants. While vehicles are mostly permitted to operate generally only allowed on sidewalks and in pedestrian areas. Street cross-
on roads, pedestrians ings are permitted to be crossed both by vehicles and by pedestrians. To allow for robust and safe navigation, autonomously operating robots in urban environments are required to localize nearby traffic participants accurately [1, 2, 3] and classify ground surfaces robustly. While autonomous vehicles typically require a binary distinction between road and non-road surfaces, mobile robots operating in pedestrian spaces must erucially be able to distinguish between sidewalks, roads, and road crossings in order to navigate urban environments safely [4, 6].
In recent years, multitude of semantic segmentation datasets for urban autonomous driving has been proposed [7, 8, 9]. These datasets, however, are recorded from the vantage point of street vehi- cles. Therefore, models trained on these datasets exhibit strong bias toward the camera viewpoint, introducing a significant domain gap when deployed in areas intended for non-vehicle usages such as pedestrian areas sidewalks. While it is possible to manually annotate images obtained from the pedestrian viewpoint, this is an expensive and time-consuming task. Automatic annotation of images offers promising alternative to manual annotations made by human annotators. Previously proposed automatic annotation approaches typically leverage the ego-motion of data collection platform to obtain spatially sparse image-level labels of traversable ground surfaces [10, 11] or are based on proprioceptive sensors such as sound and vibration [12, 13, 14]. In contrast to exist- ing work, we additionally leverage the trajectories of other traffic participants such as vehicles and pedestrians, and project them into the camera images. This enables us to label multiple types of ground surfaces, including roads, sidewalks or pedestrian areas, and street crossings based on the type of tracked objects. Hereby, we leverage the fact that under most circumstances, pedestrians walk in areas reserved for them and vehicles drive on roads or through street crossings but not on the sidewalk. The object detector used to generate the pedestrian- and vehicle trajectories does not suf- fer from the viewpoint-inducas domain gap present in segmentation models. To further boost model performance, build a ground surface map from these predictions by spatially aggregating the predictions. Aggregation of semantic segmentation predictions has been previously proposed [15], however, the generated maps have previously not been used as an annotation source for semantic seg- mentation models. We show that it is possible to self-distill the segmentation model by re-projecting the aggregated surface map back into the camera images and using them as annotations, boosting the model performance.
In summary, this work offers the following key contributions: (i) A novel automatic annotation approach that leverages trajectories of traffic participants such as vehicles and pedestrians for gen- erating sparse multi-class semantic pixel annotations. (ii) A segmentation model self-distillation pipeline to generate training annotations from projections of an aggregated surface map. (iii) The Freiburg Pedestrian Scenes dataset recorded with a robot platform navigating through a wide range of urban pedestrian environments.
2 Related Works
Self-supervised methods for visual terrain segmentation in off-road driving applications were inves- tigated in [12. 16, 17. 14, 13J. In these works, labels obtained from proprioceptive sensor modality (i.e. vibration, sound) are used to partially annotate exteroceptive sensor modalities (i.c. RGB vi- sion). Other non-learning approaches leverage geometric features in LiDAR point clouds to classify vertical and horizontal surfaces [18, 19, 20].
One of the first works to consider auto-generated annotations for semantic image segmentation in the context of autonomous driving was Barnes et al. [10]. The authors propose self-supervised approach for generating drivable paths in monocular RGB images from projected ego-trajectories of the recording vehicle on popular urban driving datasets. Mayr al. [21] propose self- lebe-labeling pipeline for drivable road area segmentation. Based on stereo disparity maps and ground plane fitting, they extract drivable road areas from images and use the annotated RGB images to train binary segmentation model. Cho et al. [22] estimate drivable space and surface normal vectors from stereo images, which are used as secudd--rrund-oee -trth to train segmentation model. Bruls et al. [23]

---

leverage weakly-labelee annotations for urban road markings based on LiDAR reflectance values and potentials from Conditional Random Field. Wang et al. [24] propose self-supervised drivable area and road anomaly segmentation approach from RGB-D data. They leverage stereo depth image obtain weak labels for obstacles sticking out from the ground level. Wellhausen er [14] propose self-supervised weak image labeling scheme based on a proprioceptive vibration-basee terrain classifier. Labels predicted by the proprioceptive classifier are projected into the robot camera ego-view. Zür et al. [13] propose self-suuervised labeling scheme based on an unsupervised audio clustering approach, where the cluster indices serve weak labels and are projected into the robot camera images. Most recently, Onozuka et al. [1] propose a traversable area segmentation approach for personal mobility systems such intelligent wheelchairs.
To summarize, existing methods for automatic annotation or self- saffssueervieed approaches do not leverage the additional data provided by the trajectories of other traffic participants, thus, ignor- ing relevant information. In addition, our work makes use of the aggregated surface map as an additional annotation source, further boosting the segmentation model performance by increasing the number of annotated pixels.
3 Technical Approach
Our goal is to label the surface classes Pedestrian, Road, Crossing, and Obstacle. The classes Pedes- trian and Road contain surface areas, where either of the two classes is exclusively permitted. Areas intended for pedestrian use include sidewalks, pedestrian zones, and footpaths while vehicle areas include all road sections without crossings. The class Crossing is intended to annotate asphalt sur- faces at street crossings (zebra crossings or signaled pedestrian crossings). Both pedestrians and vehicles are permitted to cross these areas. Pixels labeled from the ego-trajectory and those obtained from pedestrian trajectories are jointly used to provide annotations for the class Pedestrian since we assume that the robot is teleoperated to only traverse pedestrian surfaces. The class Obstacle annotates different kinds of non-traversable surfaces such as buildings, moving or static objects ex- tending over ground or vegetation. The class Unknown serves as a filler class for all pixels where no annotation is provided. In the following, we will first discuss the automatic generation of image annotations from the robot ego-trajectory and traffic participant tracklets (Subsec. 3.1) ) and subse- quently the generation and projection of the semantic surface map for additional model performance gains (Subsec. 3.2).
3.1 Surface Annotations from Tracklets
We first perform LiDAR-SLAM [25], generating a list of poses E SB(3) for the robot base for each data collection run. In the following, we will discuss the projection of the ego-trajectory into image coordinates. In order to project the robot trajectory into the viewpoint of the onboard camera, associate time-synchronized robot pose with each of the camera images. Assuming static transform T between the robot base and the camera mounting position relative to the base, the robot trajectory in homogeneous pixel coordinates u [u, u, 1]T can be expressed as
KTÖTH D.
where TW is the time-dependent transformation between the world coordinates and the current robot base position, obtained from p,, K E 33x3 denotes the intrinsic camera matrix, and P g denotes the ego-trajectory projected onto the ground surface. For brevity, we omit the superscript t for dime-dependent variables. Note that we dilate the robot trajectory laterally by half its base width in order to label all pixels within the robot footprint.
To obtain the trajectories of other traffic participants such vehicles and pedestrians, leverage the ByteTrrack [26] object tracker with an EfficientDet [27] object detector pre-trained on the MS- COCO dataset [28]. The object trajectory in 3D world coordinates is obtained by projecting the tracklet bounding box center point coordinates into 3D world coordinates. To perform this trans- formation, interpolate sparse depth images obtained from the LiDAR points, which provides an accurate depth estimation for a given object bounding box. Formally, the projection of tracklets into 3D world coordinates x E I follows the inverse projection equation:
3

---

I: Surface Annotations from Tracklets
RGB
Tracker
LIDAR
Projector
Do
Ego- Pas-ss
Araoklet- Images
Training
Segmentation Model
Il: Model Self-Distillation via Map Aggregation
Segmemator
Surfaco Map
D
Map rsgator
Projector
LDAR
Map-Annotated
Pows
Images
Training
Segmentation Model
Figure 2: Visualization of our automatic annotation pipeline. In step I, we leverage RGB images, LiDAR point clouds, ego-poses, and object tracker project the ego-poses and the observed tracklets into camera images, generating a sparsely-annotane In step II, aggregate them into a global semantic surface map. Finally, we project this map into the camera images and obtain denser and more consistent annotations, denoted as D2.
semantic segmentation dataset Do- frozen segmentation model trained on Do to obtain semantic annotations and
X T T K 'u
(2)
where we follow the naming convention in Eq. and the scalar d E R denotes the depth scaling factor. Similar to the projection of the ego-trajectory, we assign labels to image pixels according the 3-D world tracklet projection into image coordinates, according to Eq. Similar to the ego-trajectory, we laterally dilate the tracklet line segments by a fixed object width, which is set to be 0.5 m for pedestrians and 2 2 for vehicles. Street crossings defined to be traversable by both pedestrian and motorized traffic participants. We, therefore, define set of all pixels indicating street crossing Sc as the intersection of pixels indicating pedestrian usage Sp and vehicle usage Sy. More formally, we define Sç == Sp n Sv. Obstacles are defined as objects extending substantially above the ground plane. To detect the ground plane, we segment the LiDAR point cloud using the pre-trained ground plane estimation network GroundNet [29]. After projecting the segmented point cloud into each RGB image, we label each RGB image pixel located more than 20 em above the ground plane as Obstacle, following existing stixel-based approaches [30]. We denote the set of so-produced annotations for the surface classes Pedestrian, Road, Crossing, and Obstacle as dataset Do-
3.2 Surface Mapping and Self-Distillation via Aggregation and Reprojection
In addition to the aforementioned annotation procedure, we propose novel self-distillation method for the semantic segmentation model. We argue that the inherent class prediction uncertainty the segmentation model can be reduced by aggregating multiple predictions for a given patch of ground and re-training the model with these aggregated predictions. Prior works [31, 32] have shown how model self-distillation can help improve model performance. In this work we perform model self- distillation by spatially aggregating predictions in order re-train the model on these aggregated predictions. Consider a surface patch St. Following similar formulations by [33] and [34], we associate belief hy E K with St, containing the log odds vector of S being of class k. We denote K as the total number of considered classes. We collect all model predictions p. e E that

---

Figure 3: trajectajetsssies (left, blue color) and camera images from Freiburg Pedestrian Scenes dataset. The dataset features a wide range of urban environments including busy streets, pedestrian areas, and road crossings with varying weather conditions.
contain that patch of ground. In the beginning, the vector is initialized with uniform distribution hg over the classes and is updated according to the update rule h h + (I) hh), where 1 denotes the inverse observation model log odds:
2)(==2) 18=10m log - (k= log 1 1-p((k=2)
1k==)
Pi K) 1- pk(k -))
=
log
(3)
and (k) denotes the model prediction for ground surface patch S at time step t. After all be- lief updates have been executed, we transform the log-odds vector into class probabilities using the softmax function. We take the arg max over the probability vector to obtain the most likely surface class and annotate patch St with that class. In order obtain dense surface representation suitable for training a segmentation model, we triangulate all surface patch center points and create a trian- gular mesh of ground surfaces. As a post-processing step, we smooth the surface mesh using the Taubin filter [35]. To generate training data for the segmentation model, we again use the 3D poses of the camera and project the semantic surface mesh back into the camera RGB images semantic annotations. Due to the larger spatial extent the surface map compared to the tracklets, significantly increase the number of annotated pixels in each image. We denote the set of so-produced annotations as dataset D1.
dense
3.3
Model Training
The aforementioned annotation scheme labels pixels that are associated with obstacles have been traversed either by the robot or by other traffic participants. All other pixels in the images are assigned the label Unknown. We pose the ground segmentation task segmentation task with sparse label supervision, where only a subset of the pixels in each image has annotations available. As the model architecture, we use the Lav3++ model architecture [36]. We use standard cross-entropy loss for all non- non-Unknown image pixels. Unknown pixels are ignored during training.
Dataset
We present the Freiburg Pedestrian Scenes dataset recorded with our robot platform. During each data collection run, the robot is telcoperated through semi-structured urban environments and moves alongside pedestrians sidewalks, pedestrian areas, and street crossings. Each data collection run consists of time-synchronized sensor measurements from a Bumblebee Stereo RGB camera, a Velo- dyne HDL 32-beam rotating LiDAR scanner, an IMU, and GPS/GNSS receiver. Furthermore, provide Graph-SLAM poses [37]. In total, the dataset comprises 15 highly diverse and challeng- ing urban scenes. The data collection conditions, and structural diversity. Figure illustrates exemplary RGB images and the record- ing locations. The dataset Germany. The dataset key statistics are listed in Tab. 1. To evaluate our approach, we manually annotated 50 view RGB images from five data collection runs not included in the training set.
cover wide range of illumination conditions, weather
recorded over the course of multiple years in the city of Freiburg,
5

---

Table 2: Model performances when trained on the Freiburg Pedestrian Scenes / Vistas datasets and
evaluated on the Freiburg Pedestrian Scenes dataset. We denote the IoU values in %.
Annotation Source
Road
Pedestrian
Crossing
Obstacle
Mean
Mapillary Vistas [7]
12.1
20.0
0.5
89.2
30.4
Ego
37.3
85.8
30.8
Ego Tracklets
35.9
67.5
43.4
88.3
58.8
Map Reprojection
38.4
69.2
48.2
85.9
60.4
We also hand-annotated large sections of the traversed areas with semantic BEV map in order to be able to compare aggregated and ground-truth maps. Exemplary visualizations of this map are visualized in Fig. 5.
Experimental Results
We compare our automatic annota- tion approach with several baseline ap- proaches. All results are listed in Tab 2.
Table I: Freiburg Pedestrian Scenes dataset details
Modality
Quantity
Frequency [Hz]
We first evaluate model trained on the Vistas dataset [7]. To perform quanti- tative comparison, we re-map the Vis- tas classes to the Freiburg Pedestrian Scenes class labels. While we obtain a high test mean Intersection-over- Union (mloU) of 62.9% on the Vistas test split, obtain relatively low mIoU values of 30.4% on our dataset. This can be attributed to the domain gap between the two datasets due to the inconsistent camera viewpoints. Leveraging the robot ego-trajectory yields greatly improved results for the Pedestrian class but cannot account for any other semantic class. Our tracklet-based method, in contrast, shows better performance than the baseline model in all classes but the Obstacle class. What's more, our experiments indicate an improvement IoU values when leveraging the aggregated semantic surface map (constituting dataset D1) for model training. This is most likely due to the larger number of annotated pixels and the increased annotation consistency due to predic- tion aggregation. We illustrate qualitative results in Fig. 4. Generally speaking, the model trained saadfelassifiaations due to the camera viewpoint bias present
Stereo RGB
260k
LiDAR IMU
490k
4.2M
100
GPS
490k
SLAM poses
49k
Map annotations
112523 m²
on Vistas shows many false-positive in the Vistas dataset. We also observe that segmentation masks of our persfrrerooring model are well aligned with the ground-truth annotations. However, due to the challenging visual similarity between ground classes, not all areas are predicted correctly. Most incorrect predictions are pro- duced in crossing regions and in places where sidewalks and streets not easily distinguishable (see Fig. 4, failure cases).
5.1 Evaluation of Semantic Surface Maps
We qualitatively evaluate the semantic surface maps obtained with our approach. To generate the maps, we use our segmentation model and aggregate its predictions as described in Subsec. 3.2. Figure 5 illustrates maps produced with our approach and the respective ground truth maps. We observe that the generated maps exhibit more consistent class assignments compared to the ego- view image predictions due the prediction aggregation procedure for map generation. We observe that in most areas, the predicted ground class equals the actual ground class. In particular, the classes Pedestrian and Road align well with the ground truth areas. Challenging street crossings are accounted for in all regions. However, we note that the spatial extent of some crossing regions due Crossing/Roaa and Crossing/Pedestrian misclassifications leaves further room for improvement. For more experimental evaluations, please refer to Suppl. Material Sec. and I.
5.2 Limitations
Despite the fact that the aggregated maps are mostly well-aligned with the ground truth maps, not all annotations are correct, leading to partial bleeding of classes into each other. The class Crossing is particularly challenging for two reasons: Firstly, the annotations produced by our approach are not always consistent since not all street crossings are covered by observed trajectories. Secondly,

---

Trained Vistas[7]
Trained on Fgo-Trajectory
Trained Map Reprojection
RGB Input
Ground Truth
Figure 4: Qualitative results of models trained on different datasets and evaluated the test split of our Freiburg Pedestrian Scenes dataset. We visualize RGB input images, ground truth annotations, and model predictions. Images below the horizontal gray line illustrate failure cases. Color-codes for the semantic classes are: Road, Pedestrian, Cressing, Obstacle.

---

20mm
100
7Om
Figure 5: Exemplary illustrations of maps produced with our approach (best viewed zoomed in). The top row shows the respective ground-truth map and the bottom row shows the aligned semantic surface maps obtained with aggregation approach. Color-codes for the semantic classes identical to Fig. 4. Black color denotes not annotated / unobseevedd areas.
the Crossing pixels have substantial overlap in terms of texture with pixels of classes Road and Pedestrian, requiring the model to rely contextual information such line markings, which is not present in all scenes. Furthermore, our approach requires highly accurate localization, sensor calibration, and object tracker performance in order to generate correct annotations. Finally, the annotation quality depends on the behavior of traffic in accordance with traffic rules. If pedestrians jaywalk to streets vehicles drive in pedestrian areas, the annotations be inconsistent, leading to reduced model performance.
6 Conclusion
In this work, we showed how a semantic segmentation model for urban surface segmentation can be trained from projections of the ego-trajectory and projections of tracklets of other traffic partici- pants. We also showed that the segmentation model can be further improved via self-distillation by spatially aggregating the model predictions into semantic map. Regarding possible future work, there is room for improvement terms overall segmentation quality. Furthermore, future work might include the extension of the approach to more types of traffic participants such bicycles and railways to accommodate more urban environments and annotating higher-level map attributes compared to surface types such as road graphs and lane graphs.
Acknowledgments
We would like to thank our former colleagues at AIS for providing parts the datasets. This includes in particular Noha Radwan, ... We would also like to thank the reviewers for their helpful comments and suggestions. Finally we would like to thank the DFG for providing funding for the project under grant DFG BU 865/10-2- Autonomous Street Crossing with Pedestrian Assistant Robots.
References
[1] N. Radwan, W. Winterhalter, C. Dornhege, and W. Burgard. Why did the robot cross the load?-lerniing from multi-modal sensor data for autonomous road crossing. In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4737- 4742. IEEE. 2017.
[2] N. Radwan, W. Burgard, and A. Valada. Multimodal interaction-awart motion prediction for autonomous street crossing. The International Journal of Robotics Research, 39(13):1567- 1598, 2020.
[3] Zür and W. Burgard. Self-supervised moving vehicle detection from audio-visual cues. arXiv preprint arXiv:2201. 2022.

---

[4] R. Kümmerle. M. Ruhnke, B. Steder, C. Stachniss, and W. Burgard. Autonomous robot navi- gation in highly populated pedestrian zones. Jourmal of Field Robotics, 32(4):565-589 2015.
[5] N. Radwan and W. Burgard. Effective interactionaware trajectory prediction using temporal convolutional neural networks. In Workshop on Crowd Navigation: Current Challenges and New Paradigms for Safe Robot Navigation in Dense Crowds at IEEE/RSJ International Con- ference on Intelligent Robots and Systems (IROS), 2018.
[6] K. Petek, K. Sirohi, D. Büscher, and W. Burgard. Robust monocular localization in sparse hd maps leveraging multi-task uncertainty estimation. arXiv preprint arXiv:21 10563,,,,. 2021.
[7] G. Neuhold, T. Ollmann, S. Rota Bulo, and P. Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the IEEE international conference on computer vision, pages 4990-4999, 2017.
[8] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. The cityscapes dataset for semantic urban scene understanding, P Proceedings of the IEEE conference on computer vision and pattern recognition. pages 3213-3223, 2016.
[9] W. Maddem,, G. Pascoe, C. Linegar, and P. Newman. year, 1000 km: The oxford robotcar dataset. The International Journal of Robotics Research, 36(1):3-15, 2017.
[10] D. Barnes, W. Maddern, and I. Posner. Find your own way: Weakly-supervisee segmentation of path proposals for urban autonomy. In 2017 IEEE International Conference on Robotics and Automation (ICRA), pages 203-210. IEEE, 2017.
[11] Y. Onozuka, R. Matsumi, and M. Shino. Weakly recommended traversable area segmentation using automatically labeled images for autonomous driving in pedestrian envi- ronment with no edges. Sensors, 21(2):437, 2021.
[12] C. A. Brooks and K. lagnemma. supe-supe terrain classification for planetary surface exploration rovers. Journal of Field Robotics, 29(3):445 468, 2012.
[13] J. Zürm, W. Burgard, and A. Valada. Self-supervised visual terrain classification from unsuper- vised acoustic feature learning. IEEE Transactions on Robotics, 37(2):466-45 2020.
[14] L. Wellhausen, A. Dosovitskiy, R. Ranftl, K. Walas, C. Cadena, and M. Hutter. Where should i walk? predicting terrain properties from images via self-supervised learning. IEEE Robotics and Automation Letters, (12::1509-15) 2019.
[15] D. Paz, H. Zhang, Q. Li, H. Xiang, and H. I. Christensen. Probabilistic semantic mapping for urban autonomous driving applications. 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2059-2064. IEEE, 2020.
[16] K. Otsu, M. Ono, T. J. Fuchs, I. Baldwin, and T. Kubota. Autonomous terrain classification with co-and self-training approach. IEEE Roboties and Automation Letters, 1(2):814-819, 2016.
[17] A. Valada and W. Burgard. Deep spatiotemporal models for robust proprioceptive terrain clas- sification. The International Journal of Robotics Research, (4): 15:1:115521115333 2017.
[18] F. Moosmann, O. Pink, and C. Stiller. Segmentation of 3d lidar data in non-flat urban environ- ments using local convexity criterion. In 2009 IEEE Intelligent Vehicle Symposium, pages 215-220, IEEE, 2009.
[19] B. Douillard, J. Underwood, N. Melkumyan, S. Singh, S. Vasudevan, C. Brunner, and A. Quadros. Hybrid elevation maps: 3d surface models for segmentation. In 2010 IEEE/RS/ International Conference on Intelligent Robots and Systems, pages 1532-1538. IEEE, 2010.
[20] A. K. Aijazi, P. Checchin, and L. Trassoudaine. Segmentation based classification of 3d urban point clouds: A super-voxel based approach with evaluation. Remote Sensing, 5(4): 2013.
[21] J. Mayr, C. Unger, and F. Tombari. Self-supervised learning of the drivable area for au- tonomous vehicles. In 2018 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 362-369. IEEE, 2018.

---

[22] J. Cho, Kim, H. Jung, C. Oh, J. Youn, and K. Sohn. Multi-task self-supervised visual repre- sentation learning for monocular road segmentation. In 2018 IEEE International Conference on Multimedia and Expo (ICME), pages 1-6, IEEE, 2018.
[23] T. Bruls, W. Maddemn, A. A. Morye, and P. Newman, Mark yourself: Road marking segmen- tation via weakly woakly-supervised annotations from multimodal data. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1863-18700 IEEE, 2018.
[24] H. Wang, Y. Sun, and M. Liu. Self-supervised drivable area and road anomaly segmentation using rgb-d data for robotic wheelchairs. IEEE Robotics and Automation Letters, 4(4):4386- 4393, 2019.
[25] R. Kümmerle, G. Grisetti, H. Strasdat, K. Konolige, and W. Burgard. 2 o: A general frame- work for graph optimization. In 2011 IEEE International Conference on Robotics and Au- tomation, pages 3607-3613. IEEE, 2011.
[26] Y. Zhang, P. Sun, Y. Jiang, D. Yu, Z. Yuan, P. Luo, W. Liu, and X. Wang. Bytetrack: Multi- object tracking by associating every detection box. arXiv preprint arXiv:211000664, 2021.
[27] M. Tan, R. Pang, and Q. Le Efficientdet: Scalable and efficient object detection. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10781- 10790, 2020.
[28] T.-Y. Lin, M. Maire, Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and L. Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[29] A. Paigwar, Ö. Erkent, Sierra-Gonzalee Gonzalez, and C. Laugier. Gndnet: Fast ground plane estima- tion and point cloud segmentation for autonomous vehicles. In 2020 JEEE/RSJJ International Conference on Intelligent Robots and Systems (IROS), pages 2150-2156. IEEE, 2020.
[30] S. Ramos, S. Gehrig, P. Pinggera, U. Franke, and C. Rother. Detecting unexpected obstacles for self-driving cars: Fusing deep learning and geometric modeling. In 2017 IEEE Intelligent Vehicles Symposium (IV), pages 1032. IEEE, 2017.
[31] L. Zhang, J. Song, A. Gao, Chen, C. Bao, and K. Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3713-3722, 2019.
[32] Gou, B. Yu, S. J. Maybank, and D. Tao. Knowledge distillation: A survey, International Journal of Computer Vision, 12966):789-1888 2021.
[33] A. Asgharivaskasi and N. Atanasov. Active bayesian multi-class mapping from range and semantic segmentation observations. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1-7. IEEE, 2021.
[34] A. Dewan and W. Burgard. Deeptemporalseg: Temporally consistent semantic segmentation of 3d lidar scans. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 2624-2630. IEFF, 2020.
[35] G. Taubin. Curve and surface smoothing without shrinkage. In Proceedings of IEEE interna- lional conference on computer vision, pages 852-857. IEEE, 1995.
[36] L.-C. Chen, Y. Zhu, G. Papandreou. F. Schroff, and H. Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European con- ference on computer vision (ECCV), pages 801-818. 2018
[37] G. Grisetti, R. Kümmerle, C. Stachniss, and W. Burgard. tutorial on graph- basedd slam. IEEE Intelligent Transportation Systems Magazine, 2(4):3 2010.
10


Reviews:
[{"review_id": "xZx1uzaiQka", "reviewer": "Reviewer_vAHU", "summary": "This paper focuses on an automated data labeling pipeline where camera poses are found, per image detection and tracking is done and then everything is fused to the world frame. The labeling is aggregated in the world frame and projected back to each image. They use the pipeline to annotate images and train a semantic segmentation model with deeplab.", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Weaknesses:\nThe idea of labeling images in 3D and projecting them back to the images is not new and has been done before: https://arxiv.org/pdf/1702.07836.pdf\nThe paper does not present a novel idea and rather uses bits and pieces of prior work.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is more like a systems paper and does not present new perspective/insight.\n\nPost Rebuttal: I still think the paper is very incremental. It is not clear what this dataset provides beyond all the other large scale datasets such as the ones released by Waymo and Lyft. Having a paper about how to collect new dataset without really justifying what is wrong with other datasets is not a good motivation. In order for this dataset to be on the next level, it should have significantly more diversity in locations and size of dataset.", "technical_quality": "Poor: Lack of required evidence to support the claims.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Irrelevant to robotics", "quality_of_limitations": "Limitations section not present", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "See Weaknesses.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "sEtSXojvR-B", "reviewer": "Reviewer_a8Do", "summary": "This paper presents a two-stage approach to automatically label ground surface segmentation (roads, pedestrian, crossing, obstacle). In the first stage, the paper leverages pedestrian and vehicle tracking as well as the robot's trajectories to produce sparse, trajectory-based annotations. In the second stage, the page leverages a learning-based segmentation model to dilate the labels to fill more pixels. Using this automatic annotation pipeline, the paper also introduces an annotated dataset.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper overall is clearly written with its approach and evaluation clearly documented.\n\nI am very excited about the paper's proposed self-distillation procedures. This is a great idea for weak supervision. The presence of pedestrians and vehicles on the roads are light in most environments and their behavior can be somewhat stochastic, but segmentation labels contain very rich information. These procedures can be promising in filling the gaps among the sparse annotations to achieve the same level of information richness as full segmentation annotations much like filling in the missing pixels of an image.\n\nI also like the evaluation performed in comparison with the Vistas dataset. It really shows the domain gap between datasets collected from different perspectives. The dataset also appears to be very large in scale.\n\nUnfortunately, I think the positioning of the paper is a bit off or unfinished. This paper aims to produce labels for a dataset. Although from the experiments the paper shows that its dataset is more suitable for pedestrian-oriented tasks than autonomous driving datasets, the generated labels are still far from perfect as seen from figures 4, 5 and the video in the supplementary materials. Perhaps a good question to ask is: what task do the authors think this dataset is useful for? If it is for navigation, this dataset can be dangerous to rely on. For example, I observe ground surfaces that should be roads or even obstacles to be labelled as pedestrian and we certainly don't want to see a robot wandering in those places. I also observe 'holes' in areas that are predominantly 'pedestrian' or 'road'. The paper's proposed method outputs labels that already have a discounted quality from the human-labelled data. The models that are trained on the paper's dataset will surely only perform worse in their respective tasks than the models trained on more accurately labelled datasets.\n\nTherefore, I think the paper is unfinished and has limited impact in its current stage. It would be very helpful if the authors can identify a task that may benefit from this dataset (or maybe just the segmentation task). Despite the imperfect labels, the dataset is very large in scale. If the scale of the dataset can compensate the inaccuracies in labels and the authors can demonstrate this in an experiment. This would highlight the utility of their dataset.\n\nAlternatively, I see this as an opportunity for a learning-based robot navigation task. The outputted segmentation maps, though dangerous to rely on, can still be useful in providing contextual information for certain downstream robot navigation tasks. If the authors can integrate their approach in a navigation model and demonstrate its success, I think it will be a strong paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper has limited impact because the quality of the automatically produced labels are questionable and more experiments are needed to support the utility of the dataset. It is unclear what downstream tasks the authors have in mind that would benefit from this dataset. It is also unclear whether the scale of the dataset might help in reducing the negative impact of the low-quality labels. There is novelty in the paper's approaches, so I rate this paper a weak reject.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "- Figure 1: why are there no obstacle labels on the top-left area of the image?\n- The author's definition of 'crossing' can be more abstract. Semantically, crossing are specified patch of areas where the pedestrians and vehicles both may appear on it. Defining it as intersections of 'pedestrian' and 'road' may result in noisy labelling.\n- The authors frequently mention 'model performance'. What is this performance? What metric are the authors using to define this 'performance'?\n- $^W\\bf{T}_B$ and $^B\\bf{T}_C$ look confusing when put into the equations. Consider $\\bf{T}^W_B$ and $\\bf{T}^B_C$\n- Unclear what surface patches are defined in section 3.2\n- It is very confusing how 'unclear' is handled in section 3.3. Consider refining this part.\n- In section 5.1, \"We observe that in most areas, the predicted ground class equals the actual ground class.\" I don't think this is the case. Can you show this with better illustrations? Maybe define some metric and conduct some quantitative evaluation (such as IoU). "}}, {"review_id": "nSXo9uf9j_P", "reviewer": "Reviewer_sqbj", "summary": "In this paper, the authors propose a method to automatically label the roads according to their possible traversability by vehicles, pedestrians or none. They use the tracklets of vehicles, pedestrians and segmentation results of DeeplabV3+ for this purpose. My comments are as follows.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Is the surface considered to be flat? What would happen if there were slopes/irregularities on the road surface?\n\nDo you perform a manual post-process to check the correctness of the final labels such as tracklet continuation, etc.?\n\nIf an HD map was possible, would this increase the accuracy?\n\nIt can be interesting if you can report all the results of the trained mIoU's on related datasets for Tab.2.\n\nHow long did it take approximately to label a ground truth image?\n\nDid you consider multi-class labels for pixels that may belong to more than one class?\n\nDid the robot platform visit the same locations more than once? If so, did it change/improve the labeling of the road? This point should be discussed further.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Segmentation of the roads according to their traversability is a hard work, both from the point of view of data collection and platform maintenance. The limitations are also clearly addressed. If the data can be made public, it will be good addition to the community.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {"issues": "- The number of revisits and its effect on the improvement of the segmentation should be discussed.\n- The possibility of a SLAM algorithm for creating a semantic map for the collected data should be discussed."}}, {"review_id": "ZAgqrt82PA6", "reviewer": "Reviewer_x2BQ", "summary": "The paper proposes a framework for ground surface segmentation and mapping in RGB images. The main motivation is to facilitate the labeling effort for training learning-based ground segmentation methods. The proposed approach does not require any human labeling and infers the ground classes sidewalk, road, street crossing, and obstacles from projected ego-motion and object tracklets estimated with an object detection and tracking approach. To increase the number of annotations, a ground segmentation network is trained using this data and its predictions are used in a second iteration to generate a semantic map. The authors provide a dataset with ground segmentation labels estimated by their method as well as a labeled test set. The experiments show that training with additional labels generated from object tracklets and with the surface model increases the segmentation performance on the test set.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is very well written and easy to follow.\n- The experimental section supports the claim that supervision signals from object tracklets boost the ground segmentation performance.\n- The authors record a diverse dataset and plan to release it along with the code which will be beneficial for the community.\n- The limitations are well discussed.\n\nWeaknesses:\n- There are just some minor unclarities that are listed in the Issues section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper is very well written and technically sound. The authors clearly motivate the problem and describe their annotation procedure in detail. I acknowledge the recording and annotation of a public dataset which makes it possible to assess the quality of the annotation when using them for training a segmentation network. The experimental section shows that using tracked objects and an aggregated surface map leading to more annotation labels improves the training of the segmentation network. There are just a few unclarities listed below that should be discussed and possibly revised in the manuscript.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Minor issues:\n\n- It is not clear to me how the approach selects tracklets to be projected into the camera image. If trajectories are occluded by other objects or just very far away, how are these excluded from the projection? Looking at Fig 1., it seems like the past trajectory of both pedestrians on the right is not projected into the image since is occluded by them. How is this achieved?\n\n- Are the surface map-based annotations in D1 solely extracted from the aggregated map? If yes, I wonder if they could have been combined with the previously generated ego-motion and tracklet-based annotations? This could increase the robustness in case wrong semantic predictions lead to a corrupted aggregated map.\n\n- How are the surface patches in Section 3.2 extracted?  \n\n- Since a surface patch contains multiple pixels, what if the per-pixel predictions differ? How is the belief h in Equation (3) computed for a surface patch given contradicting pixel-wise predictions?\n\n- The caption in Table 2 suggests that all models are trained on the proposed dataset. Is this a mistake, since the Mapillary Vistas is a different annotation source?\n\n- Do the authors have an explanation for why the map reprojection decreases the segmentation performance for obstacles? Are the obstacles derived from bounding boxes as mentioned in Supplementary D also included in the map reprojection?\n\n- It would be very nice to see the corresponding annotation results for the images in Figure 4. I could imagine that the human-annotated ground truth masks differ from the automatically generated supervision signals, therefore it is harder to assess how good the corresponding prediction is."}}]

Author rebuttal:

Decision: Accept (poster)