Package: seg-metrics
Segmentaion Metrics PackageThis is a simple package to compute different metrics forMedicalimage segmentation(images with suffix.mhd,.mha,.nii,.nii.gzor.nrrd), and write them to csv file.BTW, if you need the support for more suffix, just let me know by creating new issuesSummaryTo assess the segmentation performance, there are several different methods. Two main methods are volume-based metrics and distance-based metrics.Metrics includedThis library computes the following performance metrics for segmentation:Voxel based metricsDice (F-1)JaccardPrecisionRecallFalse positive rateFalse negtive rateVolume similarityThe equations for these metrics can be seen in thewikipedia.Surface Distance based metrics (with spacing as default)Hausdorff distanceHausdorff distance 95% percentileMean (Average) surface distanceMedian surface distanceStd surface distanceNote: These metrics aresymmetric, which means the distance from A to B is the same as the distance from B to A. More detailed explanication of these surface distance based metrics could be foundhere.Installation$pipinstallseg-metricsGetting startedTutorial is at theColab.API reference is available atDocumentationExamples could be found below.UsageAt first, import the package:importseg_metrics.seg_metricsassgEvaluate two batches of images with same filenames from two different folderslabels=[0,4,5,6,7,8]gdth_path='data/gdth'# this folder saves a batch of ground truth imagespred_path='data/pred'# this folder saves the same number of prediction imagescsv_file='metrics.csv'# results will be saved to this file and prented on terminal as well. If not set, results# will only be shown on terminal.metrics=sg.write_metrics(labels=labels[1:],# exclude backgroundgdth_path=gdth_path,pred_path=pred_path,csv_file=csv_file)print(metrics)# a list of dictionaries which includes the metrics for each pair of image.After runing the above codes, you can get alist of dictionariesmetricswhich contains all the metrics.Also you can find a.csvfile containing all metrics in the same directory.If thecsv_fileis not given, the metrics results will not be saved to disk.Evaluate two imageslabels=[0,4,5,6,7,8]gdth_file='data/gdth.mhd'# ground truth image full pathpred_file='data/pred.mhd'# prediction image full pathcsv_file='metrics.csv'metrics=sg.write_metrics(labels=labels[1:],# exclude backgroundgdth_path=gdth_file,pred_path=pred_file,csv_file=csv_file)After runing the above codes, you can get adictionarymetricswhich contains all the metrics.Also you can find a.csvfile containing all metrics in the same directory.Note:When evaluating one image, the returnedmetricsis a dictionary.When evaluating a batch of images, the returnedmetricsis a list of dictionaries.Evaluate two images with specific metricslabels=[0,4,5,6,7,8]gdth_file='data/gdth.mhd'pred_file='data/pred.mhd'csv_file='metrics.csv'metrics=sg.write_metrics(labels=labels[1:],# exclude background if neededgdth_path=gdth_file,pred_path=pred_file,csv_file=csv_file,metrics=['dice','hd'])# for only one metricmetrics=sg.write_metrics(labels=labels[1:],# exclude background if neededgdth_path=gdth_file,pred_path=pred_file,csv_file=csv_file,metrics='msd')By passing the following parameters to select specific metrics.-dice:Dice(F-1)-jaccard:Jaccard-precision:Precision-recall:Recall-fpr:Falsepositiverate-fnr:Falsenegtiverate-vs:Volumesimilarity-hd:Hausdorffdistance-hd95:Hausdorffdistance95%percentile-msd:Mean(Average)surfacedistance-mdsd:Mediansurfacedistance-stdsd:StdsurfacedistanceFor example:labels=[1]gdth_file='data/gdth.mhd'pred_file='data/pred.mhd'csv_file='metrics.csv'metrics=sg.write_metrics(labels,gdth_file,pred_file,csv_file,metrics=['dice','hd95'])dice=metrics['dice']hd95=metrics['hd95']Evaluate two images in memory instead of diskNote:The two images must be both numpy.ndarray or SimpleITK.Image.Input arguments are different. Please usegdth_imgandpred_imginstead ofgdth_pathandpred_path.If evaluatingnumpy.ndarray, the defaultspacingfor all dimensions would be1.0for distance based metrics.If you want to evaluatenumpy.ndarraywith specific spacing, pass a sequence with the length of image dimension asspacing.labels=[0,1,2]gdth_img=np.array([[0,0,1],[0,1,2]])pred_img=np.array([[0,0,1],[0,2,2]])csv_file='metrics.csv'spacing=[1,2]metrics=sg.write_metrics(labels=labels[1:],# exclude background if neededgdth_img=gdth_img,pred_img=pred_img,csv_file=csv_file,spacing=spacing,metrics=['dice','hd'])# for only one metricsmetrics=sg.write_metrics(labels=labels[1:],# exclude background if neededgdth_img=gdth_img,pred_img=pred_img,csv_file=csv_file,spacing=spacing,metrics='msd')About the calculation of surface distanceThe default surface distance is calculated based onfullyConnectedborder. To change the default connected type,
you can set argumentfullyConnectedasFalseas follows.metrics=sg.write_metrics(labels=[1,2,3],gdth_img=gdth_img,pred_img=pred_img,csv_file=csv_file,fully_connected=False)In 2D image, fullyconnected means 8 neighbor points, while faceconnected means 4 neighbor points.
In 3D image, fullyconnected means 26 neighbor points, while faceconnected means 6 neighbor points.How to obtain more metrics? like "False omission rate" or "Accuracy"?A great number of different metrics, like "False omission rate" or "Accuracy", could be derived from some theconfusion matrics. To calculate more metrics or design custom metrics, useTPTNFPFN=Trueto return the number of voxels/pixels of true positive (TP), true negative (TN), false positive (FP), false negative (FN) predictions. For example,metrics=sg.write_metrics(gdth_img=gdth_img,pred_img=pred_img,TPTNFPFN=True)tp,tn,fp,fn=metrics['TP'],metrics['TN'],metrics['FP'],metrics['FN']false_omission_rate=fn/(fn+tn)accuracy=(tp+tn)/(tp+tn+fp+fn)Comparision with medpymedpyalso provide functions to calculate metrics for medical images. Butseg-metricshas several advantages.Faster.seg-metricsis10 times fastercalculating distance based metrics. Thisjupyter
notebookcould reproduce the results.More convenient.seg-metricscan calculate all different metrics in once in one function whilemedpyneeds to call different functions multiple times which cost more time and code.More Powerful.seg-metricscan calculatemulti-labelsegmentation metrics and save results to.csvfile in good manner, butmedpyonly provides binary segmentation metrics. Comparision can be found in thisjupyter
notebook.If this repository helps you in anyway, show your love ❤️ by putting a ⭐ on this project.
I would also appreciate it if you cite the package in your publication. (Note:This package isNOTapproved for clinical use and is intended for research use only. )Bibtex@misc{Jingnan,
    title  = {A package to compute segmentation metrics: seg-metrics},
    author = {Jingnan Jia},
    url    = {https://github.com/Ordgod/segmentation_metrics}, 
    year   = {2020}, 
    doi = {10.5281/zenodo.3995075}
}