Package: mmeval
OpenMMLab websiteHOTOpenMMLab platformTRY IT OUT🤔Reporting IssuesEnglish |简体中文IntroductionMMEval is a machine learning evaluation library that supports efficient and accurate distributed evaluation on a variety of machine learning frameworks.Major features:Comprehensive metrics for various computer vision tasks (NLP will be covered soon!)Efficient and accurate distributed evaluation, backed by multiple distributed communication backendsSupport multiple machine learning frameworks via dynamic input dispatching mechanismSupported distributed communication backendsMPI4Pytorch.distributedHorovodpaddle.distributedoneflow.commMPI4PyDistTorchCPUDistTorchCUDADistTFHorovodDistPaddleDistOneFlowDistSupported metrics and ML frameworksNOTE: MMEval tested with PyTorch 1.6+, TensorFlow 2.4+, Paddle 2.2+ and OneFlow 0.8+.Metricnumpy.ndarraytorch.Tensortensorflow.Tensorpaddle.Tensoroneflow.TensorAccuracy✔✔✔✔✔SingleLabelMetric✔✔✔MultiLabelMetric✔✔✔AveragePrecision✔✔✔MeanIoU✔✔✔✔✔VOCMeanAP✔OIDMeanAP✔COCODetection✔ProposalRecall✔F1Score✔✔✔HmeanIoU✔PCKAccuracy✔MpiiPCKAccuracy✔JhmdbPCKAccuracy✔EndPointError✔✔✔AVAMeanAP✔StructuralSimilarity✔SignalNoiseRatio✔PeakSignalNoiseRatio✔MeanAbsoluteError✔MeanSquaredError✔InstallationMMEvalrequires Python 3.6+ and can be installed via pip.pipinstallmmevalTo install the dependencies required for all the metrics provided inMMEval, you can install them with the following command.pipinstall'mmeval[all]'Get StartedThere are two ways to useMMEval's metrics, usingAccuracyas an example:frommmevalimportAccuracyimportnumpyasnpaccuracy=Accuracy()The first way is to directly call the instantiatedAccuracyobject to calculate the metric.labels=np.asarray([0,1,2,3])preds=np.asarray([0,2,1,3])accuracy(preds,labels)# {'top1': 0.5}The second way is to calculate the metric after accumulating data from multiple batches.foriinrange(10):labels=np.random.randint(0,4,size=(100,))predicts=np.random.randint(0,4,size=(100,))accuracy.add(predicts,labels)accuracy.compute()# {'top1': ...}Learn MoreTutorialsImplementing a MetricUsing Distributed EvaluationExamplesMMClsTensorPackPaddleSegDesignBaseMetric DesignDistributed Communication BackendMultiple DispatchIn the worksContinue to add more metrics and expand more tasks (e.g. NLP, audio).Support more ML frameworks and explore multiple ML framework support paradigms.ContributingWe appreciate all contributions to improve MMEval. Please refer toCONTRIBUTING.mdfor the contributing guideline.LicenseThis project is released under theApache 2.0 license.Projects in OpenMMLabMMEngine: OpenMMLab foundational library for training deep learning models.MIM: MIM installs OpenMMLab packages.MMCV: OpenMMLab foundational library for computer vision.MMClassification: OpenMMLab image classification toolbox and benchmark.MMDetection: OpenMMLab detection toolbox and benchmark.MMDetection3D: OpenMMLab's next-generation platform for general 3D object detection.MMRotate: OpenMMLab rotated object detection toolbox and benchmark.MMYOLO: OpenMMLab YOLO series toolbox and benchmark.MMSegmentation: OpenMMLab semantic segmentation toolbox and benchmark.MMOCR: OpenMMLab text detection, recognition, and understanding toolbox.MMPose: OpenMMLab pose estimation toolbox and benchmark.MMHuman3D: OpenMMLab 3D human parametric model toolbox and benchmark.MMSelfSup: OpenMMLab self-supervised learning toolbox and benchmark.MMRazor: OpenMMLab model compression toolbox and benchmark.MMFewShot: OpenMMLab fewshot learning toolbox and benchmark.MMAction2: OpenMMLab's next-generation action understanding toolbox and benchmark.MMTracking: OpenMMLab video perception toolbox and benchmark.MMFlow: OpenMMLab optical flow toolbox and benchmark.MMEditing: OpenMMLab image and video editing toolbox.MMGeneration: OpenMMLab image and video generative models toolbox.MMDeploy: OpenMMLab model deployment framework.