Package: planning-centric-metrics
Learning to Evaluate Perception Models Using Planner-Centric MetricsPyTorch code for PKL (CVPR 2020).Learning to Evaluate Perception Models Using Planner-Centric MetricsJonah Philion,Amlan Kar,Sanja FidlerCVPR, 2020 (Poster)[Paper] [Project Page]Abstract:Variants of accuracy and precision are the gold-standard by which the computer vision community measures progress of perception algorithms. One reason for the ubiquity of these metrics is that they are largely task-agnostic; we in general seek to detect zero false negatives or positives. The downside of these metrics is that, at worst, they penalize all incorrect detections equally without conditioning on the task or scene, and at best, heuristics need to be chosen to ensure that different mistakes count differently. In this paper, we propose a principled metric for 3D object detection specifically for the task of self-driving. The core idea behind our metric is to isolate the task of object detection and measure the impact the produced detections would induce on the downstream task of driving. Without hand-designing it to, we find that our metric penalizes many of the mistakes that other metrics penalize by design. In addition, our metric downweighs detections based on additional factors such as distance from a detection to the ego car and the speed of the detection in intuitive ways that other detection metrics do not. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time.Note:This repository builds on thenuscenes-devkit librarywhich also uses the Apache 2.0 License.CitationIf you found this codebase useful in your research, please consider citing@InProceedings{Philion_2020_CVPR,
author = {Philion, Jonah and Kar, Amlan and Fidler, Sanja},
title = {Learning to Evaluate Perception Models Using Planner-Centric Metrics},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}Quick StartWe show how to evaluate PKL for a synthetic detection submission using a pretrained planner. Instructions for training models yourself can be found in the "Full Tutorial" section. We assume thatnuScenesis located atNUSCENES_ROOTand nuScenes maps are located atNUSCENES_MAP_ROOT.Install theplanning-centric-metricspackage (or install locally withpip install -e .).pip install planning-centric-metricsTheplanning-centric-metricspackage has one exposed function (calculate_pkl) and the API is shown below.nusc_eval = DetectionEval(nusc, config=cfg,
                          result_path='./example_submission.json',
                          eval_set='mini_train',
                          output_dir='./res', verbose=True)
info = planning_centric_metrics.calculate_pkl(nusc_eval.gt_boxes, nusc_eval.pred_boxes,
                                              nusc_eval.sample_tokens, nusc_eval.nusc,
                                              nusc_maps, device,
                                              nworkers, bsz=128,
                                              plot_kextremes=5,
                                              verbose=True)An example script that uses theplanning_centric_metricspackage to calculate the PKL for a synthetic detector can be found inexamples/synthetic.py. Set--gpuid=-1to use the CPU.python examples/synthetic.py quick_test --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOT --gpuid=0 --nworkers=10

-> {'min': tensor(0.2776), 'max': tensor(256.7013), 'mean': tensor(54.6309), 'median': tensor(13.1194), 'std': tensor(71.5134)}Full TutorialPreparationDownload nuscenes data and maps fromhttps://www.nuscenes.org/.Install dependencies.pip install nuscenes-devkit tensorboardY(optional) Visualize the dataVisualization of inputs and outputs of the "planner".python main.py viz_data mini --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOTGenerate masksDetermine the locations where a car can exist after dt seconds from data. These masks are used both during training and during PKL calculation.python main.py scrape_masks trainval --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOTVisualize the masks.python main.py viz_masks --out_name=masks.json --imname=masks.jpgTrain a plannerEval accuracy generally peaks after 100k steps (~24 hours to train on a Titan V).python main.py train trainval --ego_only=False --flip_aug=True --lr=0.002 --weight_decay=1e-08  --logdir=./runs --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOT --num_workers=NWORKERS --mask_json=masks.jsonMonitor tensorboard if you'd like to.tensorboard --logdir=./runs --bind_allVisualize the predictions of a model.python main.py eval_viz mini --modelpath=MODELPATH --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOTGenerate a "submission"To demonstrate how to evaluate PKL, we first generate a synthetic detection submission. For instance, generate a submission that drops detections 0% of the time (a perfect submission)python main.py generate_drop_noise mini mini_train --drop_p=0.0 --dataroot=NUSCENES_ROOT...or generate a submission that drops all detections.python main.py generate_drop_noise mini mini_train --drop_p=1.0 --dataroot=NUSCENES_ROOTScore a submission with PKLScore the synthetic submissions generated above. Fordrop_p=0.0the PKL should be 0 no matter what planner is used. Fordrop_p=1.0on themini_trainsplit using the pre-trained planner, the mean PKL should be 54.6310.python main.py eval_test mini mini_train --result_path=SUBMISSION --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOT --modelpath=MODELPATH --plot_kextremes=5Fordrop_p=1.0, the 5 timestamps with the highest PKL (high is bad) under the pre-trained planner are shown below.PKL DistributionAnalyze the distribution of PKL values for a given submission. Image shown below analyzes the PKL ofMEGVII val detections.python main.py pkl_distribution_plot trainval --pkl_result_path=SUBMISSION --dataroot=NUSCENES_ROOT(optional) Sensitivity visualizationTo investigate what PKL is sensitive to, visualize the importance of each ground-truth object by removing it from the scene and computing the PKL.python main.py false_neg_viz mini --modelpath=MODELPATH --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOTVisualize the importance of correctly not predicting a false positive at each location in a grid about the ego.python main.py false_pos_viz mini --modelpath=MODELPATH --dataroot=NUSCENES_ROOT --map_folder=NUSCENES_MAP_ROOTAcknowledgementsThank you to Amlan Kar, Sanja Fidler, Karan Sapra, the NVIDIA AV Team, and NVIDIA Research for their help in making this research possible.