Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes a labeling method for active learning for self-driving applications. In particular, the paper focuses on the percpetion and prediction problem. Unlike traditional active learning frameworks that assume each example in the dataset can be labeled for the same cost, the proposed method explicitly model the cost to label each example. The motivation is, given a fixed labeling budget, more examples (lower cost examples) can be labeled. The authors proposed to include the cost to label in the scoring function, the function used in active learning frameworks to pick examles that are believed to be most likely to improve the model performance. 
Furthermore, the paper introduces partial labeling, i.e., labeling only particular regions of an example. The authors show that given a fixed labeling budget, the performance will improve by training with more examples but equal number of labels. 
strengths_and_weaknesses: Strengths
* The paper is well-structured and the results support the claims. 
* The idea is quite simple but seems to have a considerable effect on the results 

Weaknesses 
* The numbers/results provided lack measures like confidence intervals to ensure statistical significance. 
* One of the core contributions of the paper, i.e., predicting the cost of labeling for each example, is not clearly described. 
summary_of_recommendation: It is not really clear how partial labeling would work for the prediction problem. An actor can leave the region in subsequent frames, therefore, the labeling region can change across consecutive frames. 


issues: * An important contribution of the paper is the "cost-aware" part of it. However, this component is not described well and it is not really clear how the cost of each example is predicted. 

* For all experimental results, a confidence interval measure is required to ensure statistical significance. 

* Line 117: NMS is not defined. 
* Line 166: The sentence is incomplete
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents a way to introduce active learning in the Perception and Prediction (P&P) setting: it chooses to label the scenes partially instead of label the entire scene for a higher cost. It trains with partial supervision and only apply the loss on the labeled regions; essentially it will select a large number of scenes, each labelled very sparsely. The methods has been shown effective in P&P tasks and outperform prior baselines.
strengths_and_weaknesses: Strengths:
- It is a novel idea to explicitly model the cost and incorporate it in the active learning selection process. 
- Comprehensive literature review
- Clear presentation of methods and algorithms


One concern I have on visualization is that in Figure 3: Qualitative Examples, the graphics do not seem to be very clear in presentation. The red boxes are quite small and I'm not sure are the white boxes meaningful? There could be more semantically meaningful explanations for the figures (although I like the idea of using qualitative examples).

Another question I have is in Figure 2 (Right): the experimental results of Fine-Grained is not too different from that using Random Regions, especially when using small amount of labels. Do the authors have an explanation on the similarity of results?
summary_of_recommendation: I would recommend this paper since it presents an interesting active learning method in P&P setting that involves labelling cost, and introduces the idea of modularity in training samples. The main idea could potentially inspire active learning in other fields, for example image classification, robot learning etc.
issues: Please refer to the Strengths and Weaknesses section for more details.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: Self-driving vehicles require extensive data supervision in order to generalize in the real world. However, much of this labelling is wasted effort due to the "long-tail problem" - common situations (e.g. parked car) are easy to predict and make up the bulk of the dataset while rare (but crucial) situations make up a tiny fraction of the data-points. This paper proposes an active learning technique to make maximal use of fixed labelling budgets. The utility $S(r)$ and the cost $C(r)$ of each un-labelled region $r$ are explicitly modeled, and data points are chosen for labelling if they are expected to improve prediction performance most efficiently. Furthermore, the training loss is modified to model partially labelled scenes. These modifications enable improved performance v.s. baselines and strongly indicate that fully labelling scenes is a wasteful process.
strengths_and_weaknesses: The paper is well written and the big ideas behind the work are effectively conveyed. The supplementary video is very helpful for understanding the paper and the qualitative examples help contextualize the work's contributions. The general idea of using active learning and partial labelling seems like a sensible way to attack the long-tail problem. Furthermore, the ablations do a good job of justifying hyper-parameter choices (e.g. scoring function parameterization and selection policy) in the method.

However, the results section could use comparisons to real active learning baselines. The authors mention *LearnLoss* and *Core-Set* but show no results on those baselines, other than vague references of similarity to *Coarse-Grained* in the supplementary material. Real comparisons to prior active learning/labelling work will help us judge this paper's contributions fairly. 
summary_of_recommendation: This paper is well executed and thorough investigation of partial labelling and active learning in P&P modeling context. However, it is missing comparisons against real baselines that are important to understand significance.
issues: * Please show comparisons v.s. baselines you mentioned as well as any other active learning baselines that have been previously tried in this space.
* Please do a better job documenting the dataset. Did you collect it or did someone else? If you collected it, what sort of data distribution should we expect?
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors present a method for fine-grained active learning in the self-driving domain.  At each round of training, a learning agent scores unlabelled examples that seem most informative, and an oracle provides labels for them.  The agent then trains on all labeled examples seen so far and repeats.  The authors argue that performing this active selection on a per-instance basis rather than a per-scene basis improves performance.
strengths_and_weaknesses: Strengths:
The paper is very well written and easy to follow.  Nothing is confusing, and the material is well presented.

The experiments are thorough and the authors have presented good baselines to disentangle the effects of active learning vs. fine-grained/instance-based labelling.

The results seem to provide useful insight for practitioners in this space.

Weaknesses:
There is not a great deal of technical novelty here.  The key idea is to do active learning over instances rather than full scenes, which leads to a fairly straightforward implementation.

The authors heavily promote their approach based on active learning, but the most important result of the paper seems to be that if you have a limited labeling budget, it’s much better to partially label many scenes than densely label a few, regardless of whether you use active labeling or just random selection.  The difference between active fine-grained and random fine-grained (random regions) is quite small in the immediate perception/planning results (Figure 2.)  There is more of a gap for the downstream planning results (Table 1.) but here the gaps between all methods seem relatively small.  In general it would be very helpful if the authors provided some guidance as to what constitutes a significant improvement in this domain.  The authors comment that the difference of 0.1 “Progress” where their model is weaker than the random baseline is not significant, but do not provide information on what threshold would be considered significant here.  I understand that with self-driving cars every collision counts, but I also have a limited understanding of how well performance on these held-out test sets relates to real-world performance.  More explanation here would be helpful.

Along these lines, please provide units for the various error rates in Table 1 to give a better idea of what these numbers mean.

summary_of_recommendation: I am very much on the fence with regards to this paper.  It is well presented, and well argued, but the minimal technical novelty and results which are marginally better, but not dominating other methods makes it hard to strongly argue in favor of acceptance.  Part of this stems from my own lack of familiarity with what constitutes a significant improvement in this domain.  I would improve my score if the authors can convincingly argue that these results are significant and useful and/or other reviewers can vouch for their significance.

Also, if there was an option for Impact between 3 and 4 (something like "This work is likely to have some impact within a limited domain") I would apply it here.  It is reasonable to see this paper having some impact within the self-driving community.
issues: It would be great if you can provide more clarity on what constitutes significant improvement in this domain, and make a strong argument that the improvements here will have a strong impact on real-world performance.

Label/explain the units in Table 1 please.
review_title: null

Meta-review: All reviewers agree that the idea proposed in the paper is interesting and that the paper is well presented. Three of four reviewers rate the paper's impact to be limited, although I can see this to generalize to other domains as well. There are two issues with the paper in its current form though: I agree with reviewer TDyJ who criticizes the lack of confidence intervals to understand the statistical significance of the results. Second, reviewer KVfT recommends a comparison with prior active learning work - something I would love to see as well. If these issues were to be addressed I would recommend acceptance. 
Decision: Accept (Poster)