Package: relplot
relplot: Principled Reliability Diagramsrelplotis a Python package for plotting reliability diagrams and measuring calibration error,
in a theoretically-principled way.
The package generates reliability diagrams as shown on the right
(reproduced innotebooks/figure1.ipynb):The density of predictions $f_i \in [0, 1]$ is visualized as the
thickness of the red regression line, and the gray band shows
bootstrapped confidence bands around the regression.The reliability diagram is obtained by kernel smoothing with a careful choice of parameters, and the associated calibration measure is called theSmoothECE(abbreviated smECE).
The SmoothECE is roughly equal to the standard ECE of the smoothed reliability diagram.
The reliability diagram for a toy dataset of 8 points is shown below;
more theoretical details are available in the accompanying preprintSmooth ECE: Principled Reliability Diagrams via Kernel Smoothing.InstallationInstall with Pip:>pipinstallrelplotOr, clone the repo and install with:>cdrelplot
>pipinstall.Getting StartedBasic usage (on sample data):importrelplotasrpimportnumpyasnp## generate toy data (miscalibrated)N=5000f=np.random.rand(N)y=(np.random.rand(N)>1-(f+0.2*np.sin(2*np.pi*f)))*1## compute calibration error (smECE) and plotprint('calibration error:',rp.smECE(f,y))fig=rp.rel_diagram(f,y)fig.show()This is reproduced innotebooks/demo.ipynb.For more control, one can compute the calibration data withrelplot.prepare_rel_diagram, and then plot it later withrelplot.plot_rel_diagram.
For example:...diagram=rp.prepare_rel_diagram(f,y)# compute calibration data (dictionary)print('calibration error:',diagram['ce'])plt.plot(diagram['mesh'],diagram['mu'])# plot the calibration curve manuallyfig,ax=rp.plot_rel_diagram(diagram)# plot the diagram in a new figureData FormatMethods expect inputs in the form
of a 1D array of predicted probabilities (f) and a 1D array of binary labels (y),
where $f_i \in [0, 1]$ and $y_i \in {0, 1}$.
We then consider the calibration of the
distribution $(f_i, y_i)$ of prediction-outcome pairs.
This package primarily considers the binary outcome setting, but can be used
to measure multi-class confidence calibration as shown below.Multi-class CalibrationIn the multi-class setting,confidence calibrationcan be measured by expressing it as the binary
calibration of the distribution on (confidence, accuracy) pairs.
A convenience function for this common use case is provided:# f: [N, C] array of logits over C classes# y: [N, 1] array of predicted classesconf,acc=relplot.multiclass_logits_to_confidences(f,y)# reduce to binary settingrelplot.rel_diagram(f=conf,y=acc)# plot confidence calibration diagramrelplot.smECE(f=conf,y=acc)# compute smECE of conficence calibrationCustomizationThe plot made byrelplot.rel_diagramcan be customized in various ways, as shown below.
See this notebook for examples of more options:notebooks/figure1.ipynbAdditional Notebooks and FeaturesThe header image (Figure 1 of the paper) is generated innotebooks/figure1.ipynbThe experiments in the paper are reproduced innotebooks/paper_experiments.ipynbrelplot.metricscontains implementations of various alternate calibration measures, including binnedECE and laplace kernel calibration. This is in addition to the recommended calibration measure of smoothECE (relplot.smECE).relplot.rel_diagram_binnedplots the "binned" reliability diagram. Not recommended for usage; included for comparison.relplot.config.use_tex_fontscan be set to True if you have $\LaTeX$ installed.CitationIf you use relplot in your work, please consider citing:@misc{relplot2023,title={Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing},author={Jarosław Błasiok and Preetum Nakkiran},year={2023},}