Package: pycaleva
A framework for calibration evaluation of binary classification models.When performing classification tasks you sometimes want to obtain the probability of a class label instead of the class label itself. For example, it might be interesting to determine the risk of cancer for a patient. It is desireable to have a calibrated model which delivers predicted probabilities very close to the actual class membership probabilities. For this reason, this framework was developed allowing users tomeasure the calibration of binary classification models.Evaluate the calibration of binary classification models with probabilistic output (LogisticRegression, SVM, NeuronalNets ...).Apply your model to testdata and use true class labels and predicted probabilities as input for the framework.Various statistical tests, metrics and plots are available.Supports creating a calibration report in pdf-format for your model.See thedocumentationfor detailed information about classes and methods.Installation$ pip install pycalevaor build on your own$ git clone https://github.com/MartinWeigl/pycaleva.git
$ cd pycaleva
$ python setup.py installRequirementsnumpy>=1.17scipy>=1.3matplotlib>=3.1tqdm>=4.40pandas>=1.3.0statsmodels>=0.13.1fpdf2>=2.5.0ipython>=7.30.1UsageImport and initializefrompycalevaimportCalibrationEvaluatorce=CalibrationEvaluator(y_test,pred_prob,outsample=True,n_groups='auto')Apply statistical testsce.hosmerlemeshow()# Hosmer Lemeshow Testce.pigeonheyse()# Pigeon Heyse Testce.z_test()# Spiegelhalter z-Testce.calbelt(plot=False)# Calibrationi Belt (Test only)Show calibration plotce.calibration_plot()Show calibration beltce.calbelt(plot=True)Get various metricsce.metrics()Create pdf calibration reportce.calibration_report('report.pdf','my_model')See  thedocumentationof single methods for detailed usage examples.Example ResultsWell calibrated modelPoorly calibrated modelhltest_result(statistic=4.982635477424991, pvalue=0.8358193332183672, dof=9)hltest_result(statistic=26.32792475118742, pvalue=0.0018051545107069522, dof=9)ztest_result(statistic=-0.21590257919669287, pvalue=0.829063686607032)ztest_result(statistic=-3.196125145498827, pvalue=0.0013928668407116645)FeaturesStatistical tests for binary model calibrationHosmer Lemeshow TestPigeon Heyse TestSpiegelhalter z-testCalibration beltGraphical represantions showing calibration of binary modelsCalibration plotCalibration beltVarious MetricsBrier ScoreAdaptive Calibration ErrorMaximum Calibration ErrorArea within LOWESS Curve(AUROC)The above features are explained in more detail in PyCalEva'sdocumentationReferencesStatistical tests and metrics:[1] Hosmer Jr, David W., Stanley Lemeshow, and Rodney X. Sturdivant.
Applied logistic regression. Vol. 398. John Wiley & Sons, 2013.[2] Pigeon, Joseph G., and Joseph F. Heyse.
An improved goodness of fit statistic for probability prediction models.
Biometrical Journal: Journal of Mathematical Methods in Biosciences 41.1 (1999): 71-82.[3] Spiegelhalter, D. J. (1986). Probabilistic prediction in patient management and clinical trials.
Statistics in medicine, 5(5), 421-433.[4] Huang, Y., Li, W., Macheret, F., Gabriel, R. A., & Ohno-Machado, L. (2020).
A tutorial on calibration measurements and calibration models for clinical prediction models.
Journal of the American Medical Informatics Association, 27(4), 621-633.Calibration plot:[5] Jr, F. E. H. (2021). rms: Regression modeling strategies (R package version
6.2-0) [Computer software]. The Comprehensive R Archive Network.
Available fromhttps://CRAN.R-project.org/package=rmsCalibration belt:[6] Nattino, G., Finazzi, S., & Bertolini, G. (2014). A new calibration test
and a reappraisal of the calibration belt for the assessment of prediction models
based on dichotomous outcomes. Statistics in medicine, 33(14), 2390-2407.[7] Bulgarelli, L. (2021). calibrattion-belt: Assessment of calibration in binomial prediction models [Computer software].
Available fromhttps://github.com/fabiankueppers/calibration-framework[8] Nattino, G., Finazzi, S., Bertolini, G., Rossi, C., & Carrara, G. (2017).
givitiR: The giviti calibration test and belt (R package version 1.3) [Computer
software]. The Comprehensive R Archive Network.
Available fromhttps://CRAN.R-project.org/package=givitiROthers:[9] Sturges, H. A. (1926). The choice of a class interval.
Journal of the american statistical association, 21(153), 65-66.For most of the implemented methods in this software you can find references in thedocumentationas well.