Repository: scikit-learn/scikit-learn
Pull request: Change documentation cross_val_predict
closes #14199

Change of the cross_val_predict documentation in order to recommend the user to be careful for computing performances on the predictions provided by this function which can lead to different results than cross_val due to a pooling instead of an averaging strategy.

Problem statement:
cross_val_predict documentation
Hi,
I don't know if this should go in issue but I think that we should update the documentation concerning the cross_val_predict function
Currently we can read :
```
    It is not appropriate to pass these predictions into an evaluation
    metric. Use :func:`cross_validate` to measure generalization error.
```
I think that in the context of a pooling computation of performance, it can be used to estimate the performances of the model (compare to the cross_validate function which uses an average computation of performances).
For instance, in the context of a one leave out, there is no other choice than pooling the outputs of the models to estimate the performances

Hints from issue discussion:
It seems to be a somewhat controversial statement. See for instance the conversation on "Why is cross_val_predict discouraged?" at https://mail.python.org/pipermail/scikit-learn/2019-April/thread.html#3069

It certainly needs to be used with caution and with the understanding that it will often not yield the same value as cross_val_score/cross_validate/*SearchCV, and that those in all cases provide a valid estimate of generalisation performance (given a reasonable choice of dataset, CV splitter and evaluation metric). `metric(y_true, cross_val_predicted)` is often not, as it depends on test set sizes and a metric that decomposes over samples.

Please feel free to submit a PR improving the wording.
I would propose something like :
```
Using these predictions to measure the generalization performance of the model can lead to different results than `cross_val` due to the aggregation of the predictions. Moreover only metrics compatible with pooling have to be considered.
```
Open a pull request and we can see what core devs think

Also requires equal sized test sets

Commit message: Change documentation cross_val_predict

Patch to review:
diff --git a/sklearn/model_selection/_validation.py b/sklearn/model_selection/_validation.py
index 7ca80c9e0927a..7a20b64b6a43c 100644
--- a/sklearn/model_selection/_validation.py
+++ b/sklearn/model_selection/_validation.py
@@ -625,8 +625,12 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None,
                       pre_dispatch='2*n_jobs', method='predict'):
     """Generate cross-validated estimates for each input data point
 
-    It is not appropriate to pass these predictions into an evaluation
-    metric. Use :func:`cross_validate` to measure generalization error.
+    Using these predictions to measure the generalization performance of the model 
+    can lead to different results than `cross_val` due to the aggregation of the 
+    predictions. 
+    
+    Moreover only metrics compatible with pooling have to be considered and test
+    sets have to present similar caracteristics.
 
     Read more in the :ref:`User Guide <cross_validation>`.
 


Code review comments:
diff_hunk: @@ -625,8 +625,12 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None,
                       pre_dispatch='2*n_jobs', method='predict'):
     """Generate cross-validated estimates for each input data point
 
-    It is not appropriate to pass these predictions into an evaluation
-    metric. Use :func:`cross_validate` to measure generalization error.
+    Using these predictions to measure the generalization performance of the model 
path: sklearn/model_selection/_validation.py
text: @user1:
I think this is not quite right: evaluating performance with these predictions is not, in general, correctly measuring generalisation performance. It only does under specific conditions.

Perhaps, then "Passing these predictions into an evaluation metric may not be a valid way to measure generalization performance. Results can differ from `cross_validate` unless all tests sets have equal size and the metric decomposes over samples."

@author:
I think that using cross_validate is also not, in general, a good way of measuring generalization performance: it requires specific characteristics of the splitting. It highly depends on the size of the testing sets and on the similarity between those sets (between them and with regard to the training set).

@user2:
What do you mean? `cross_validate` is exactly the same as `cross_val_score` apart from that you can give several metrics and you get timing information. The splitting is done through the `cv` objects similarly

@author:
I just meant that using `cross_val_predict` predictions to compute generalization performance has its own drawbacks but that the averaging estimation of performances done in` cross_val_score` and `cross_validate` also makes strong assumption on the testing sets.

@user1:
Could you clarify what you mean by strong assumptions about test sets?  In both cases, I think, the user needs to select cv carefully. 

@author:
Yes I totally agree: both cases require to be careful about the conclusions made on generalization. In the averaging, we assume equally sized test sets with similar distribution of classes


@user1:
>  In the averaging, we assume equally sized test sets with similar distribution of classes

I disagree. In averaging we assume that the test sample is similar to a random draw from the general population. Small test sets are tricky, but as a rule, if there is high variance among the per-test-set scores, it deserves investigation.

@author:
I agree that high variance deserves investigation. But don't we assume during cross validation that each fold should have the same size ?

@user1:
>  But don't we assume during cross validation that each fold should have the same size ?

Well from an implementation side, we certainly don't require it in scikit-learn, and it's far from assured in something like LeaveOneGroupOut. I don't really see how it provides much benefit to require them to have the same size, as long as all are of sufficient size for the score measurement to be reasonably accurate. But I'm not a statistician, and may not appreciate the underlying theory.

@author:
I am not either, but to me, estimation by averaging of folds of different sizes seems to be a biased estimation of performances. I am ok to just put a warning to let the user decide how to handle the problem but this warning should also be added to the `cross_validation` and `cross_val_score` functions

diff_hunk: @@ -625,8 +625,12 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None,
                       pre_dispatch='2*n_jobs', method='predict'):
     """Generate cross-validated estimates for each input data point
 
-    It is not appropriate to pass these predictions into an evaluation
-    metric. Use :func:`cross_validate` to measure generalization error.
+    Using these predictions to measure the generalization performance of the model 
+    can lead to different results than `cross_val` due to the aggregation of the 
path: sklearn/model_selection/_validation.py
text: @user1:
you mean cross_val_score?

@author:
Yes

@user2:
and `cross_validate`