scikit-learn/scikit-learn — pull request #3574 by jmetzen (2014-08-18 18:03:10+00:00)
Title: Reliability curves for calibration of predict_proba

This PR adds the reliability_curve metric to metrics.ranking.py. Reliability diagrams allow checking if the predicted probabilities of a binary classifier are well calibrated. The PR also contains an example comparing how well the predicted probabilities of different classifiers are calibrated. A notebook giving the same example can be found under http://jmetzen.github.io/2014-08-16/reliability-diagram.html

For some backgrounds on reliability-diagrams, please refer to the paper "Predicting Good Probabilities with Supervised Learning"