Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a two-stage approach to automatically label ground surface segmentation (roads, pedestrian, crossing, obstacle). In the first stage, the paper leverages pedestrian and vehicle tracking as well as the robot's trajectories to produce sparse, trajectory-based annotations. In the second stage, the page leverages a learning-based segmentation model to dilate the labels to fill more pixels. Using this automatic annotation pipeline, the paper also introduces an annotated dataset.
strengths_and_weaknesses: The paper overall is clearly written with its approach and evaluation clearly documented.

I am very excited about the paper's proposed self-distillation procedures. This is a great idea for weak supervision. The presence of pedestrians and vehicles on the roads are light in most environments and their behavior can be somewhat stochastic, but segmentation labels contain very rich information. These procedures can be promising in filling the gaps among the sparse annotations to achieve the same level of information richness as full segmentation annotations much like filling in the missing pixels of an image.

I also like the evaluation performed in comparison with the Vistas dataset. It really shows the domain gap between datasets collected from different perspectives. The dataset also appears to be very large in scale.

Unfortunately, I think the positioning of the paper is a bit off or unfinished. This paper aims to produce labels for a dataset. Although from the experiments the paper shows that its dataset is more suitable for pedestrian-oriented tasks than autonomous driving datasets, the generated labels are still far from perfect as seen from figures 4, 5 and the video in the supplementary materials. Perhaps a good question to ask is: what task do the authors think this dataset is useful for? If it is for navigation, this dataset can be dangerous to rely on. For example, I observe ground surfaces that should be roads or even obstacles to be labelled as pedestrian and we certainly don't want to see a robot wandering in those places. I also observe 'holes' in areas that are predominantly 'pedestrian' or 'road'. The paper's proposed method outputs labels that already have a discounted quality from the human-labelled data. The models that are trained on the paper's dataset will surely only perform worse in their respective tasks than the models trained on more accurately labelled datasets.

Therefore, I think the paper is unfinished and has limited impact in its current stage. It would be very helpful if the authors can identify a task that may benefit from this dataset (or maybe just the segmentation task). Despite the imperfect labels, the dataset is very large in scale. If the scale of the dataset can compensate the inaccuracies in labels and the authors can demonstrate this in an experiment. This would highlight the utility of their dataset.

Alternatively, I see this as an opportunity for a learning-based robot navigation task. The outputted segmentation maps, though dangerous to rely on, can still be useful in providing contextual information for certain downstream robot navigation tasks. If the authors can integrate their approach in a navigation model and demonstrate its success, I think it will be a strong paper.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper has limited impact because the quality of the automatically produced labels are questionable and more experiments are needed to support the utility of the dataset. It is unclear what downstream tasks the authors have in mind that would benefit from this dataset. It is also unclear whether the scale of the dataset might help in reducing the negative impact of the low-quality labels. There is novelty in the paper's approaches, so I rate this paper a weak reject.
issues: - Figure 1: why are there no obstacle labels on the top-left area of the image?
- The author's definition of 'crossing' can be more abstract. Semantically, crossing are specified patch of areas where the pedestrians and vehicles both may appear on it. Defining it as intersections of 'pedestrian' and 'road' may result in noisy labelling.
- The authors frequently mention 'model performance'. What is this performance? What metric are the authors using to define this 'performance'?
- $^W\bf{T}_B$ and $^B\bf{T}_C$ look confusing when put into the equations. Consider $\bf{T}^W_B$ and $\bf{T}^B_C$
- Unclear what surface patches are defined in section 3.2
- It is very confusing how 'unclear' is handled in section 3.3. Consider refining this part.
- In section 5.1, "We observe that in most areas, the predicted ground class equals the actual ground class." I don't think this is the case. Can you show this with better illustrations? Maybe define some metric and conduct some quantitative evaluation (such as IoU).
review_title: null

final_score: rating: Strong Reject: I recommend rejecting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: This paper focuses on an automated data labeling pipeline where camera poses are found, per image detection and tracking is done and then everything is fused to the world frame. The labeling is aggregated in the world frame and projected back to each image. They use the pipeline to annotate images and train a semantic segmentation model with deeplab.
strengths_and_weaknesses: Weaknesses:
The idea of labeling images in 3D and projecting them back to the images is not new and has been done before: https://arxiv.org/pdf/1702.07836.pdf
The paper does not present a novel idea and rather uses bits and pieces of prior work.

robotics_focus: Irrelevant to robotics
quality_of_the_limitations_section: Limitations section not present
summary_of_recommendation: The paper is more like a systems paper and does not present new perspective/insight.

Post Rebuttal: I still think the paper is very incremental. It is not clear what this dataset provides beyond all the other large scale datasets such as the ones released by Waymo and Lyft. Having a paper about how to collect new dataset without really justifying what is wrong with other datasets is not a good motivation. In order for this dataset to be on the next level, it should have significantly more diversity in locations and size of dataset.
issues: See Weaknesses.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: In this paper, the authors propose a method to automatically label the roads according to their possible traversability by vehicles, pedestrians or none. They use the tracklets of vehicles, pedestrians and segmentation results of DeeplabV3+ for this purpose. My comments are as follows.
strengths_and_weaknesses: Is the surface considered to be flat? What would happen if there were slopes/irregularities on the road surface?

Do you perform a manual post-process to check the correctness of the final labels such as tracklet continuation, etc.?

If an HD map was possible, would this increase the accuracy?

It can be interesting if you can report all the results of the trained mIoU's on related datasets for Tab.2.

How long did it take approximately to label a ground truth image?

Did you consider multi-class labels for pixels that may belong to more than one class?

Did the robot platform visit the same locations more than once? If so, did it change/improve the labeling of the road? This point should be discussed further.
robotics_focus: Highly relevant to robotics but no hardware experiments
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Segmentation of the roads according to their traversability is a hard work, both from the point of view of data collection and platform maintenance. The limitations are also clearly addressed. If the data can be made public, it will be good addition to the community.
issues: - The number of revisits and its effect on the improvement of the segmentation should be discussed.
- The possibility of a SLAM algorithm for creating a semantic map for the collected data should be discussed.
review_title: null

final_score: rating: Strong Accept: I recommend accepting the paper and will argue for my recommendation even if other reviewers hold a different opinion.
review_content: summary: The paper proposes a framework for ground surface segmentation and mapping in RGB images. The main motivation is to facilitate the labeling effort for training learning-based ground segmentation methods. The proposed approach does not require any human labeling and infers the ground classes sidewalk, road, street crossing, and obstacles from projected ego-motion and object tracklets estimated with an object detection and tracking approach. To increase the number of annotations, a ground segmentation network is trained using this data and its predictions are used in a second iteration to generate a semantic map. The authors provide a dataset with ground segmentation labels estimated by their method as well as a labeled test set. The experiments show that training with additional labels generated from object tracklets and with the surface model increases the segmentation performance on the test set.
strengths_and_weaknesses: Strengths:
- The paper is very well written and easy to follow.
- The experimental section supports the claim that supervision signals from object tracklets boost the ground segmentation performance.
- The authors record a diverse dataset and plan to release it along with the code which will be beneficial for the community.
- The limitations are well discussed.

Weaknesses:
- There are just some minor unclarities that are listed in the Issues section.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: The paper is very well written and technically sound. The authors clearly motivate the problem and describe their annotation procedure in detail. I acknowledge the recording and annotation of a public dataset which makes it possible to assess the quality of the annotation when using them for training a segmentation network. The experimental section shows that using tracked objects and an aggregated surface map leading to more annotation labels improves the training of the segmentation network. There are just a few unclarities listed below that should be discussed and possibly revised in the manuscript.
issues: Minor issues:

- It is not clear to me how the approach selects tracklets to be projected into the camera image. If trajectories are occluded by other objects or just very far away, how are these excluded from the projection? Looking at Fig 1., it seems like the past trajectory of both pedestrians on the right is not projected into the image since is occluded by them. How is this achieved?

- Are the surface map-based annotations in D1 solely extracted from the aggregated map? If yes, I wonder if they could have been combined with the previously generated ego-motion and tracklet-based annotations? This could increase the robustness in case wrong semantic predictions lead to a corrupted aggregated map.

- How are the surface patches in Section 3.2 extracted?  

- Since a surface patch contains multiple pixels, what if the per-pixel predictions differ? How is the belief h in Equation (3) computed for a surface patch given contradicting pixel-wise predictions?

- The caption in Table 2 suggests that all models are trained on the proposed dataset. Is this a mistake, since the Mapillary Vistas is a different annotation source?

- Do the authors have an explanation for why the map reprojection decreases the segmentation performance for obstacles? Are the obstacles derived from bounding boxes as mentioned in Supplementary D also included in the map reprojection?

- It would be very nice to see the corresponding annotation results for the images in Figure 4. I could imagine that the human-annotated ground truth masks differ from the automatically generated supervision signals, therefore it is harder to assess how good the corresponding prediction is.
review_title: null

Meta-review: metareview: The paper proposes a method for annotating ground surface types such as sidewalks, roads, and street crossings, using a both the egocentric view of the robot and the projections of other traffic participants to provide sparse semantic annotations with which a ground segmentation model can be trained. 

The paper's **strengths** are: 
- the idea is interesting and the part of self-distillation is a promising way for handling sparsely annotated datasets;
- the evaluation results regarding the domain gap among datasets is insightful.

The paper's **weaknesses** are mainly:
- that the contributions seems to be fragmented and the overall method seems to not be complete;
- discussion of the contribution of this method compared to state-of-the-art;
- a sufficient connection with the robot-learning community is missing.

The authors should provide a more complete and improved version of their work, and adequately address the issues raised by the reviewers during the rebuttal.

**Post-rebuttal assessment:** After inspecting reviews and the paper, with the updated material [here](https://openreview.net/forum?id=qr0wqg8NqkL&noteId=0tIX7QHVSJB) -- please next time post updates on a general comment, reviewers were not able to discover it in chains of comments-- I believe that the paper address a challenging topic for semantic segmentation during navigation, and has interesting ideas for overcoming sparsely-annotated data via distillation. Overall, I think that the paper can be presented in a conference, given the overall improvements made by the authors.
confidence: 3: The area chair is somewhat confident
Decision: Accept (Poster)