Package: visionmetrics
visionmetricsThis repo contains evaluation metrics for vision tasks such as classification, object detection, image caption, and image matting. It usestorchmetricsas a base library and extends it to support custom vision tasks as necessary.Available MetricsImage Classification:Accuracy: computes the top-k accuracy for a classification problem. A prediction is considered correct, if the ground truth label is within the labels with top k confidences.PrecisionEvaluator: computes precision.RecallEvaluator: computes recall.AveragePrecisionEvaluator: computes the average precision, i.e., precision averaged across different confidence thresholds.AUCROC: computes Area under the Receiver Operating Characteristic Curve.F1Score: computes f1-score.CalibrationLoss**: computes theECE loss, i.e., the expected calibration error, given the model confidence and true labels for a set of data points.ConfusionMatrix: computes the confusion matrix of a classification. By definition a confusion matrix C is such that Cij is equal to the number of observations known to be in group i and predicted to be in group j (https://en.wikipedia.org/wiki/Confusion_matrix).ExactMatch: computes the exact match score, i.e., the percentage of samples where the predicted label is exactly the same as the ground truth label.The above metrics are available for Binary, Multiclass, and Multilabel classification tasks. For example,BinaryAccuracyis the binary version ofAccuracyandMultilabelAccuracyis the multilabel version ofAccuracy. Please refer to the example usage below for more details.**TheCalibrationLossmetric is only for binary and multiclass classification tasks.Object Detection:MeanAveragePrecision: Coco mean average precision (mAP) computation across different classes, under multipleIoU(s).Image Caption:BleuScore: computes the Bleu score. For more details, refer toBLEU: a Method for Automatic Evaluation of Machine Translation.METEORScore: computes the Meteor score. For more details, refer toProject page. We use the latest version (1.5) of theCode.ROUGELScore: computes the Rouge-L score. Refer toROUGE: A Package for Automatic Evaluation of Summariesfor more details.CIDErScore:  computes the CIDEr score. Refer toCIDEr: Consensus-based Image Description Evaluationfor more details.SPICEScore:  computes the SPICE score. Refer toSPICE: Semantic Propositional Image Caption Evaluationfor more details.Image Matting:MeanIOU: computes the mean intersection-over-union score.ForegroundIOU: computes the foreground intersection-over-union evaluator score.BoundaryMeanIOU: computes the boundary mean intersection-over-union score.BoundaryForegroundIOU:  computes the boundary foreground intersection-over-union score.L1Error:  computes the L1 error.Regression:MeanSquaredError: computes the mean squared error.MeanAbsoluteError: computes the mean absolute error.Retrieval:RetrievalRecall: computes Recall@k, which is the percentage of relevant items in top-k among all relevant itemsRetrievalPrecision: computes Precision@k, which is the percentage of TP among all items classified as P in top-k.RetrievalMAP: computesMean Average Precision@k, an information retrieval metric.RetrievalPrecisionRecallCurveNPoints: computes a Precision-Recall Curve, interpolated at k points and averaged over all samples.GroundingRecall: computes Recall@k, which is the percentage of correct grounding in top-k among all relevant items.Example Usageimporttorchfromvisionmetrics.classificationimportMulticlassAccuracypreds=torch.rand(10,10)target=torch.randint(0,10,(10,))# Initialize metricmetric=MulticlassAccuracy(num_classes=10,top_k=1,average='macro')# Add batch of predictions and targetsmetric.update(preds,target)# Compute metricresult=metric.compute()Implementing Custom MetricsPlease refer totorchmetricsfor more details on how to implement custom metrics.Additional RequirementsThe image caption metric calculation requires Jave Runtime Environment (JRE) (Java 1.8.0) and some extra dependencies which can be installed withpip install visionmetrics[caption]. This is not required for other evaluators. If you do not need image caption metrics, JRE is not required.