Package: coco-froc-analysis
COCO FROC analysisFROC analysis for COCO annotations and Detectron(2) detection results. The COCO annotation style is definedhere.Installationpipinstallcoco-froc-analysisAboutA single annotation record in the ground-truth file might look like this:{"area":2120,"iscrowd":0,"bbox":[111,24,53,40],"category_id":3,"ignore":0,"segmentation":[],"image_id":407,"id":945}While the prediction (here for bounding box) given by the region detection framework is such:{"image_id":407,"category_id":3,"score":0.9990422129631042,"bbox":[110.72555541992188,13.9161834716797,49.4566650390625,36.65155029296875]}The FROC analysis counts the number of images, number of lesions in the ground truth file for all categories and then counts the lesion localization predictions and the non-lesion localization predictions. A lesion is localized by default if its center is inside any ground truth box and the categories match or if you wish to use IoU you should provide threshold upon which you can define the 'close enough' relation.Usagefromcoco_froc_analysis.countimportgenerate_bootstrap_count_curvesfromcoco_froc_analysis.countimportgenerate_count_curvefromcoco_froc_analysis.frocimportgenerate_bootstrap_froc_curvesfromcoco_froc_analysis.frocimportgenerate_froc_curve# For single FROC curvegenerate_froc_curve(gt_ann=args.gt_ann,pr_ann=args.pr_ann,use_iou=args.use_iou,iou_thres=args.iou_thres,n_sample_points=args.n_sample_points,plot_title='FROC'ifargs.plot_titleisNoneelseargs.plot_title,plot_output_path='froc.png'ifargs.plot_output_pathisNoneelseargs.plot_output_path,test_ann=args.test_ann,)# For bootstrapped curvesgenerate_bootstrap_froc_curves(gt_ann=args.gt_ann,pr_ann=args.pr_ann,n_bootstrap_samples=args.bootstrap,use_iou=args.use_iou,iou_thres=args.iou_thres,n_sample_points=args.n_sample_points,plot_title='FROC (bootstrap)'ifargs.plot_titleisNoneelseargs.plot_title,plot_output_path='froc_bootstrap.png'ifargs.plot_output_pathisNoneelseargs.plot_output_path,test_ann=args.test_ann,)Please checkrun.pyfor more details. TheIoUpart of this code is not reliable and currently the codebase only works for binary evaluation, but any multiclass problem could be chunked up to work with it.Description ofrun.pyarguments:usage:run.py[-h][--bootstrapBOOTSTRAP]--gt_annGT_ANN--pr_annPR_ANN[--use_iou][--iou_thresIOU_THRES][--n_sample_pointsN_SAMPLE_POINTS][--plot_titlePLOT_TITLE][--plot_output_pathPLOT_OUTPUT_PATH][--test_annTEST_ANN][--counts][--weighted]optionalarguments:-h,--helpshowthishelpmessageandexit--bootstrapBOOTSTRAPWhethertodoasingleorbootstrapruns.--gt_annGT_ANN--pr_annPR_ANN--use_iouUseIoUscoretodecidebasedon`proximity`--iou_thresIOU_THRESIfIoUscoreisusedthedefaultthresholdissetto.5--n_sample_pointsN_SAMPLE_POINTSNumberofpointstoevaluatetheFROCcurveat.--plot_titlePLOT_TITLE--plot_output_pathPLOT_OUTPUT_PATH--test_annTEST_ANNExtraground-truthlikeannotations--counts--weightedCLI Usagepython-mcoco_froc_analysis[-h][--bootstrapN_BOOTSTRAP_ROUNDS]--gt_annGT_ANN--pred_annPRED_ANN[--use_iou][--iou_thresIOU_THRES][--n_sample_pointsN_SAMPLE_POINTS][--plot_titlePLOT_TITLE][--plot_output_pathPLOT_OUTPUT_PATH]optionalarguments:-h,--helpshowthishelpmessageandexit--bootstrapN_ROUNDSWhethertodoasingleorbootstrapruns.--gt_annGT_ANN--pred_annPRED_ANN--use_iouUseIoUscoretodecideon`proximity`ratherthenusingcenterpixelinsideGTbox.--iou_thresIOU_THRESIfIoUscoreisusedthedefaultthresholdisarbitrarilysetto.5--n_sample_pointsN_SAMPLE_POINTSNumberofpointstoevaluatetheFROCcurveat.--plot_titlePLOT_TITLE--plot_output_pathPLOT_OUTPUT_PATHBy default centroid closeness is used, if the--use_iouflag is set,--iou_thresdefaults to.75while the--score_thresscore defaults to.5. The code outputs the FROC curve on the given detection results and GT dataset.For developersRunning testspython-mcoveragerun-munittestdiscover--pattern"*_test.py"-v
python-mcoveragereport-mBuilding and publishing (reminder)act# for local CI pipelinepdoc-dgooglecoco_froc_analysis-odocs# build docspoetryversionprerelease/patch# test or actual releasepoetrypublish--build-rtest-pypi# or without -r test-pypi for publishing to pypi@Regards, Alex@misc{qbeer,
  author       = {Alex Olar},
  title        = {FROC analysis for COCO-like file format},
  howpublished = {GitHub repository},
  month        = {September},
  year         = {2022},
  url          = {https://github.com/qbeer/coco-froc-analysis}
}