Repository: scikit-learn/scikit-learn
Pull request: [MRG+2] referred reliability diagrams and added citations
<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#Contributing-Pull-Requests
-->
#### Reference Issue
<!-- Example: Fixes #1234 -->
Fixes #8423 

#### What does this implement/fix? Explain your changes.

Added reliability diagram references in the docs and modified method docstring to reflect the same.
Also added citations to `Wilks, 1995` for seminal paper on reliability diagram methods

#### Any other comments?

Easy fix to docs - sprint 03.04.2017

<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
mention "reliability diagram" in calibration docs
From the paper "Predicting Good Probabilities With Supervised Learning" it seems "reliability diagram" is a common name for the calibration curve, but it's not mentioned in the text. Googling for "reliability diagram python" looks like it doesn't exist. That's not great.

Also, we might want to cite one of the papers describing the method in the docstring.

Hints from issue discussion:

Commit message: referred reliability diagrams and added citations

Patch to review:
diff --git a/doc/modules/calibration.rst b/doc/modules/calibration.rst
index cd74f414fe3d2..44644dd169640 100644
--- a/doc/modules/calibration.rst
+++ b/doc/modules/calibration.rst
@@ -57,9 +57,9 @@ with different biases per method:
    ensemble away from 0. We observe this effect most strongly with random
    forests because the base-level trees trained with random forests have
    relatively high variance due to feature subseting." As a result, the
-   calibration curve shows a characteristic sigmoid shape, indicating that the
-   classifier could trust its "intuition" more and return probabilties closer
-   to 0 or 1 typically.
+   calibration curve also referred to as the reliability diagram (Wilks 1995[5]) shows a
+   characteristic sigmoid shape, indicating that the classifier could trust its
+   "intuition" more and return probabilties closer to 0 or 1 typically.
 
 .. currentmodule:: sklearn.svm
 
@@ -118,7 +118,7 @@ is evaluated with Brier score :func:`brier_score_loss`, reported in the legend
    :align: center
 
 One can observe here that logistic regression is well calibrated as its curve is
-nearly diagonal. Linear SVC's calibration curve has a sigmoid curve, which is
+nearly diagonal. Linear SVC's calibration curve or reliability diagram has a sigmoid curve, which is
 typical for an under-confident classifier. In the case of LinearSVC, this is
 caused by the margin property of the hinge loss, which lets the model focus on
 hard samples that are close to the decision boundary (the support vectors). Both
@@ -200,3 +200,7 @@ a similar decrease in log-loss.
 
     .. [4] Predicting Good Probabilities with Supervised Learning,
           A. Niculescu-Mizil & R. Caruana, ICML 2005
+
+    .. [5] On the combination of forecast probabilities for
+         consecutive precipitation periods. Wea. Forecasting, 5, 640–
+         650., Wilks, D. S., 1990a
diff --git a/sklearn/calibration.py b/sklearn/calibration.py
index 35cdab45a1b57..6880f26134558 100644
--- a/sklearn/calibration.py
+++ b/sklearn/calibration.py
@@ -511,7 +511,8 @@ def predict(self, T):
 
 
 def calibration_curve(y_true, y_prob, normalize=False, n_bins=5):
-    """Compute true and predicted probabilities for a calibration curve.
+    """Compute true and predicted probabilities for a calibration curve also
+    referred to as the reliability diagram.
 
     Read more in the :ref:`User Guide <calibration>`.
 


Code review comments:
diff_hunk: @@ -511,7 +511,8 @@ def predict(self, T):
 
 
 def calibration_curve(y_true, y_prob, normalize=False, n_bins=5):
-    """Compute true and predicted probabilities for a calibration curve.
+    """Compute true and predicted probabilities for a calibration curve also
path: sklearn/calibration.py
text: @user1:
The first line in the docstring should be a single line (keep forgetting the pep number for this).

Please add the sentence as a separate remark.