Package: conformal-eval
Conformal-evalA Python library for simplifying the evaluation of conformal predictors.InstallationInstallation of Conformal-eval is made by;pipinstallconformal-evalor including the extrasreport:pipinstallconformal-eval[report]# or, depending on your shell you might need:pipinstall"conformal-eval[report]"ExamplesExamples of using the package can be seen in our examples notebooks:Conformal ClassificationConformal RegressionUsing Nonconformist and conf-evalPackage dependenciesPackage dependencies can be found in thepyproject.tomlunder[tool.poetry.dependencies], noting that theJinja2package is only required for for thereportextras (installingconformal-eval[report]).APIData formatThe code internally use numpy ndarrays for matrices and vectors, but tries to be agnostic about input being either list, arrays or Pandas equivalents. For performance reasons it is recommended that conversion to numpy format is done when using several of the methods in this library, as this is internally made in most functions.For regression we require predictions to be the same as used innonconformist, using 2D or 3D tensors in numpy ndarrays of shape(num_examples,2)or(num_examples,2,num_significance_levels), where the second dimension contains the lower and upper limits of the prediction intervals.Rendering backendPlotting is made based on thematplotliblibrary andSeabornfor the functionplot_confusion_matrix_heatmap. To easily set a nicer plotting style (in our opinion) you can use the functionconf_eval.plot.update_plot_settingswhich uses Seaborn to configure the default settings in matplotlib and e.g. easily scales font size.Theconf_eval.cpsignmoduleThis module and submodules are intended for easily loading results fromCPSignin the format required byConformal-evalExtrasreportThis extras include the functionality to generate a "report" in HTML format for a model generated byCPSign. This report is at this point only in beta testing and contains rudimentary information - this could be altered in the future. Further note that this will install a CLI script calledcpsign-reportonce installing theconformal-eval[report]package, to see the usage of this simply runcpsign-report --helpin your terminal environment.Supported plotsClassificationCalibration plotLabel ratio plot, showing ratio of single/multi/empty predictions for each significance levelp-value distribution plot: plot p-values as a scatter plot"Bubble plot" confusion matrixHeatmap confusion matrixRegressionCalibration plotEfficiency plot (mean or median prediction interval width vs significance)Prediction intervals (for a given significance level)Developer notesThe project dependencies are now configured in thepyproject.tomland handled byPoetry.We should aim at supplying proper docstrings, following thenumpy docstring guide.TestingAll python-tests are located in thetests folderand are meant to be run usingpytest. Test should be started from standing in thepythonfolder and can be run "all at once" (python -m pytest), "per file" (python -m pytest tests/conf_eval/metrics/clf_metrics_test.py), or a single test function (python -m pytest tests/conf_eval/metrics/clf_metrics_test.py::TestConfusionMatrix::test_with_custom_labels).Note1:The invocationpython -m pytest [opt args]is preferred here as the current directory is added to the python path and resolves the application code automatically. Simply runningpytestrequires manual setup of thePYTHONPATHinstead.Note2:The plotting tests generate images that are saved in thetest_outputdirectory and these should be checked manually (no good way of automating plotting-tests).Before running:For thereportmodule there are CLI tests that require the package to be installedbeforerunning them (otherwise thecpsign-reportprogram is not available, or not updated). To do this you should use the following;# Set up an venv to run inpoetryshell# Install dependencies from the pyproject.tomlpoetryinstall# Run all (or subset) of testspython-mpytestTODOs:Add/finish the following plots:calibration plot - Staffan'area plot' with label-distributions - Staffanbubbel-plot - Jonathanheatmap - Staffanp0-p1 plot - StaffanAdd regression metricsAdd plots regressionChange log:1.0.0b2:Added the png-logo files for thecpsign-reportutility (gitignore had removed them before).1.0.0b1:Forked theplot_utilslibrary to make an extensive refactor of module names.Addedconf_eval.cpsign.reportmodule for generating model-reports in HTML format. The extrasreportis needed to run this code.Fixed bugs when loading data from CPSign, e.g. missing standard-deviation columns in validation metrics.Added new flag (skip_inf) when loading regression predictions and results, where a too high confidence level leads to infinitely large prediction intervals. This simply filters out the rows that have infinitely large outputs.Added new function to load confidence-independent metrics from CPSign validation output (cpsign.load_conf_independent_metrics).Added Type hints on a few places (not finished everywhere yet).0.1.0:Addedpharmbio.cpsignpackage with loading functionality for CPSign generated files, loading calibration statistics, efficiency statistics and predictions.Updated plotting functions in order to use pre-computed metrics where applicable (e.g. when computed by CPSign).Added possibility to add a shading for +/- standard deviation where applicable, e.g. calibration curveUpdated calibration curve plotting to have a generalplotting.plot_calibrationacting on pre-computed values or for classification usingplotting.plot_calibration_clfwhere true labels and p-values can be given.Update parameter order to make it consistent across plotting functions, e.g. ordered as x, y (significance vs error rate) in the plots.Added a utility function for setting the seaborn theme and context usingplotting.update_plot_settingswhich updates the matplotlib global settings.Notethis will have effect on all other plots generated in the same python session if those rely on matplotlib.