Package: simple-cocotools
simple-cocotoolsA simple, modern alternative topycocotools.AboutWhy not just usePycocotools?Code is more readable and hackable.Metrics are more transparent and understandable.Evaluation is fast.Only dependencies arenumpyandscipy. Nocythonextensions.Code is more modern (type annotations, linting, etc).InstallFrom PyPIpipinstallsimple-cocotoolsFrom Repopipinstall"simple-cocotools @ git+ssh://git@github.com/fkodom/simple-cocotools.git"For Contributors# Clone this repositoryghrepoclonefkodom/simple-cocotoolscdsimple-cocotools# Install all dev dependencies (tests etc.)pipinstall-e.[all]# Setup pre-commit hookspre-commitinstallUsageExpects target annotations to have the same format as model predictions. (The format used by alltorchvisiondetection models.)  You may already have code to convert annotations into this format, since it's required to train many detection models.  If not, use'AnnotationsToDetectionFormat' from this repoas an example for how to do that.A minimal example:fromtorchvision.detection.modelsimportmaskrcnn_resnet50_fpnfromsimple_cocotoolsimportCocoEvaluatorevaluator=CocoEvaluator()model=maskrcnn_resnet50_fpn(pretrained=True).eval()forimages,targetsindata_loader:predictions=model(images)evaluator.update(predictions,targets)metrics=evaluator.summarize()metricswill be a dictionary with format:{"box":{"mAP":0.40,"mAR":0.41,"class_AP":{"cat":0.39,"dog":0.42,...},"class_AR":{#Sameas'class_AP'above.}}"mask":{#Sameas'box'above.}}For a more complete example, seescripts/mask_rcnn_example.py.BenchmarksI benchmarked against severaltorchvisiondetection models, which havemAP scores reported on the PyTorch website.Using a default score threshold of 0.5:ModelBackbonebox mAP(official)box mAPbox mARmask mAP(official)mask mAPmask mARMask R-CNNResNet5037.936.943.234.634.140.0Faster R-CNNResNet5037.036.342.0---Faster R-CNNMobileNetV3-Large32.839.935.0---Notice that the mAP forMobileNetV3-Largeis artificially high, since it has a much lower mAR at that score threshold.  After tuning the score threshold, so that mAP and mAR are more balanced:ModelBackboneThresholdbox mAPbox mARmask mAPmask mARMask R-CNNResNet500.641.141.338.238.5Faster R-CNNResNet500.640.840.4--Faster R-CNNMobileNetV3-Large0.42536.236.2--These scores are more reflective of model performance, in my opinion.  Mask R-CNN slightly outperforms Faster R-CNN, and there is a noticeable (but not horrible) gap between ResNet50 and MobileNetV3 backbones.  PyTorch docs don't mention what score thresholds were used for each model benchmark. ¯\(ツ)/¯Ignoring the time spent getting predictions from the model, evaluation is very fast.Bbox:~400 samples/secondBbox + mask:~100 samples/secondUsing a Google Cloudn1-standard-4VM (4 vCPUs, 16 GB RAM).Note:Speeds are dependent on the number of detections per image, and therefore dependent on the model and score threshold.Keypoints UsageKeypoint mAP and mAR normally use pre-computed "sigmas" to determin the "correctness" of each keypoint prediction.  Unfortunately, those sigmas are tailored specifically for human pose (as in the COCO dataset), and not applicable to other keypoint datasets.NOTE:Sigmas are actually computed using the predictions of a specific model trained on COCO.  To make this applicable to other datasets, you would need to train a model on that dataset, and then use the sigmas from that model.  The logic is somewhat circular -- you need to train a model to get the sigmas, but you need the sigmas to compute mAP / mAR.There's no way around this, unless a large body of pretrained models are already available for the dataset you're using.  For most real-world problems, that is not the case.  So, the open-source mAP / mAR keypoints metrics are not generally extensible to other datasets.simple-cocotoolsdoes not use sigmas, and instead computes the average distance between each keypoint prediction and ground truth.  This is a much simpler approach, and is more applicable to other datasets.  It's roughly how thesigmas for COCO were originally computed.  The downside is that it's not directly comparable to the official COCO keypoints mAP / mAR.Some keypoints are more ambiguous than others.  For example, "left hip" is much more ambiguous than "left eye" -- the exact location of "left eye" should be obvious, while "left hip" is hidden by the torso and clothing.  The average distance for "left hip" will be much larger than "left eye", even if the predictions are correct. (This is how sigmas were used in the official COCO keypoints mAP / mAR.)  For that reason, keypoint distances should be interpreted with some knowledge about the specific dataset at hand.metricswill be a dictionary with format:{"box":{"mAP":0.40,"mAR":0.41,"class_AP":{"cat":0.39,"dog":0.42,...},"class_AR":{#Sameas'class_AP'above.}}"keypoints":{"distance":0.10,"class_distance":{"cat":{"distance":0.11,"keypoint_distance":{"left_eye":0.12,"right_eye":0.13,...}},...}}}How It WorksTODO:Blog post on howsimple-cocotoolsworks.Match the predictions/labels together, maximizing the IoU between pairs with the same object class. SciPy'slinear_sum_assignmentmethod does most of the heavy lifting here.For each IoU threshold, determine the number of "correct" predictions from the assignments above.  Pairs with IoU < threshold are incorrect.For each image, count the number of total predictions, correct predictions, and ground truth labels for each object class and IoU threshold.Compute AP/AR for each class from the prediction counts above.  Then compute mAP and mAR by averaging over all object classes.