Repository: scikit-learn/scikit-learn
Pull request: [MRG] Make cross_val_predict support method="predict_proba" and y=None
#### Reference Issues/PRs
Fixes #15855 

#### Description

`cross_val_predict` raises exception when passing `method="predict_proba"` and `y=None`

#### What does this implement/fix? Explain your changes.

The problem is due to the fact that when `method in ['decision_function', 'predict_proba', 'predict_log_proba']` y is cast to numpy array `y = np.asarray(y)` without checking if y is not None (when casting None to numpy array the singleton array `array(None, dtype=object)` is returned).


Problem statement:
cross_val_predict raises error when method="predict_proba" and y=None

#### Description
I found a weird behavior in the function `cross_val_predict` when passing `method="predict_proba"` and `y=None`, while using a custom estimator. I believe that the problem is due to the local variable `encode` that is defined into the function `cross_val_predict` (it should be pretty easy to fix it).

#### Steps/Code to Reproduce
This is a minimal example to reproduce this issue. 
```python
import numpy as np

from sklearn.base import BaseEstimator
from sklearn.model_selection import cross_val_predict


class TestEstimator(BaseEstimator):

    def fit(self, X, y=None):
        pass

    def predict(self, X):
        return np.zeros(len(X))

    def predict_proba(self, X):
        return np.column_stack((np.zeros(len(X)), np.ones(len(X))))


data = np.random.random((100, 10))
te = TestEstimator()
y_hat = cross_val_predict(te, data, method='predict')
print(y_hat.shape)
y_proba_hat = cross_val_predict(te, data, method='predict_proba')
print(y_proba_hat.shape)
```

#### Expected Results
When I run `y_proba_hat = cross_val_predict(te, data, method='predict_proba')` I get the following exception:
```
TypeError: Singleton array array(None, dtype=object) cannot be considered a valid collection.
```

Hints from issue discussion:
Please provide a full stack trace. I'm not sure if this is related to #15805
This is the full stack trace
```
Traceback (most recent call last):
  File "sklearn_bug_test.py", line 23, in <module>
    y_proba_hat = cross_val_predict(te, data, method='predict_proba')
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 789, in cross_val_predict
    for train, test in cv.split(X, y, groups))
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 921, in __call__
    if self.dispatch_one_batch(iterator):
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 754, in dispatch_one_batch
    self._pickle_cache)
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/joblib/parallel.py", line 210, in __init__
    self.items = list(iterator_slice)
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_validation.py", line 787, in <genexpr>
    prediction_blocks = parallel(delayed(_fit_and_predict)(
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/sklearn/model_selection/_split.py", line 327, in split
    X, y, groups = indexable(X, y, groups)
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py", line 230, in indexable
    check_consistent_length(*result)
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py", line 201, in check_consistent_length
    lengths = [_num_samples(X) for X in arrays if X is not None]
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py", line 201, in <listcomp>
    lengths = [_num_samples(X) for X in arrays if X is not None]
  File "/home/lucakubin/anaconda3/lib/python3.7/site-packages/sklearn/utils/validation.py", line 146, in _num_samples
    " a valid collection." % x)
TypeError: Singleton array array(None, dtype=object) cannot be considered a valid collection.
```
It might be related to the issue you mentioned, in the sense that both are due to casting None or True to numpy array.

In the function cross_val_predict if `method == "predict_proba"` at a certain point there is a line of code that casts y to a numpy array (`y = np.asarray(y)`) without checking if y is not None. By adding that check I think that this problem will be solved.

@lkubin @jnothman I would like to work on this issue.
@EverLookNeverSee I think it is pretty easy to fix... Just need to change line 736 of _validation.py from
```
encode = method in ['decision_function', 'predict_proba',  'predict_log_proba']
```
into

```
encode = method in ['decision_function', 'predict_proba', 'predict_log_proba'] and y is not None
```

Does it make sense to you?
@lkubin Test are failed.
What is your use case for cross_val_predict with predict_proba and y=None??

And did this error occur prior to 0.22?
@jnothman I'm working on a custom generative model that can output hard predictions or soft predictions. I thought that `predict` and `predict_proba` were suitable name to distinguish the 2 possible model outputs. For this model also makes sense cross validating, to see how the model generalizes on unseen data. 

I checked with the version 0.19.2 and a different exception is raised:

```
ValueError                                Traceback (most recent call last)
<ipython-input-3-6f01be1b677e> in <module>
     21 y_hat = cross_val_predict(te, data, method='predict')
     22 print(y_hat.shape)
---> 23 y_proba_hat = cross_val_predict(te, data, method='predict_proba')
     24 print(y_proba_hat.shape)

~/miniconda3/lib/python3.6/site-packages/sklearn/model_selection/_validation.py in cross_val_predict(estimator, X, y, groups, cv, n_jobs, verbose, fit_params, pre_dispatch, method)
    670     if method in ['decision_function', 'predict_proba', 'predict_log_proba']:
    671         le = LabelEncoder()
--> 672         y = le.fit_transform(y)
    673 
    674     # We clone the estimator to make sure that all the folds are

~/miniconda3/lib/python3.6/site-packages/sklearn/preprocessing/label.py in fit_transform(self, y)
    109         y : array-like of shape [n_samples]
    110         """
--> 111         y = column_or_1d(y, warn=True)
    112         self.classes_, y = np.unique(y, return_inverse=True)
    113         return y

~/miniconda3/lib/python3.6/site-packages/sklearn/utils/validation.py in column_or_1d(y, warn)
    612         return np.ravel(y)
    613 
--> 614     raise ValueError("bad input shape {0}".format(shape))
    615 
    616 

ValueError: bad input shape ()
```

I may work around to this problem by changing name to the prediction function from `predict_proba` to something different and everything works fine, but I thought that would be nice to support `predict_proba` even in case `y=None`.


I'd be happy to see a pull request making this work.

Commit message: Added entry in change log

Patch to review:
diff --git a/doc/whats_new/v0.23.rst b/doc/whats_new/v0.23.rst
index 30cf098eee768..c10fcadf5d425 100644
--- a/doc/whats_new/v0.23.rst
+++ b/doc/whats_new/v0.23.rst
@@ -94,6 +94,10 @@ Changelog
   type and details.
   :pr:`15622` by :user:`Gregory Morse <GregoryMorse>`.
 
+- |Fix| :func: `cross_val_predict` supports `method="predict_proba"`
+  when `y=None`.
+  :pr: `15918` by :user: `Luca Kubin <lkubin>`.
+
 :mod:`sklearn.preprocessing`
 ............................
 
diff --git a/sklearn/model_selection/_validation.py b/sklearn/model_selection/_validation.py
index 67a30c6416031..2fe4fcd7ff392 100644
--- a/sklearn/model_selection/_validation.py
+++ b/sklearn/model_selection/_validation.py
@@ -734,7 +734,7 @@ def cross_val_predict(estimator, X, y=None, groups=None, cv=None,
     # If classification methods produce multiple columns of output,
     # we need to manually encode classes to ensure consistent column ordering.
     encode = method in ['decision_function', 'predict_proba',
-                        'predict_log_proba']
+                        'predict_log_proba'] and y is not None
     if encode:
         y = np.asarray(y)
         if y.ndim == 1:
@@ -842,7 +842,11 @@ def _fit_and_predict(estimator, X, y, train, test, verbose, fit_params,
         estimator.fit(X_train, y_train, **fit_params)
     func = getattr(estimator, method)
     predictions = func(X_test)
-    if method in ['decision_function', 'predict_proba', 'predict_log_proba']:
+
+    encode = method in ['decision_function', 'predict_proba',
+                        'predict_log_proba'] and y is not None
+
+    if encode:
         if isinstance(predictions, list):
             predictions = [_enforce_prediction_order(
                 estimator.classes_[i_label], predictions[i_label],
diff --git a/sklearn/model_selection/tests/test_validation.py b/sklearn/model_selection/tests/test_validation.py
index d1c67930fac77..933433eaf48a4 100644
--- a/sklearn/model_selection/tests/test_validation.py
+++ b/sklearn/model_selection/tests/test_validation.py
@@ -214,6 +214,9 @@ def predict(self, T):
             T = T.reshape(len(T), -1)
         return T[:, 0]
 
+    def predict_proba(self, T):
+        return T
+
     def score(self, X=None, Y=None):
         return 1. / (1 + np.abs(self.a))
 
@@ -972,6 +975,18 @@ def test_cross_val_predict_unbalanced():
                               decimal=12)
 
 
+def test_cross_val_predict_y_none():
+    # ensure that cross_val_predict works when y is None
+    mock_classifier = MockClassifier(allow_nd=True)
+    X = np.random.random((100, 10))
+    y_hat = cross_val_predict(mock_classifier, X, y=None, cv=5,
+                              method='predict')
+    assert np.all(X[:, 0] == y_hat)
+    y_hat_proba = cross_val_predict(mock_classifier, X, y=None, cv=5,
+                                    method='predict_proba')
+    assert np.all(y_hat_proba == X)
+
+
 def test_cross_val_score_sparse_fit_params():
     iris = load_iris()
     X, y = iris.data, iris.target


Code review comments:
diff_hunk: @@ -972,6 +975,18 @@ def test_cross_val_predict_unbalanced():
                               decimal=12)
 
 
+def test_cross_val_predict_y_none():
+    # ensure that cross_val_predict works when y is None
+    mock_classifier = MockClassifier(allow_nd=True)
+    X = np.random.random((100, 10))
+    y_hat = cross_val_predict(mock_classifier, X, y=None, cv=5,
+                              method='predict')
+    assert np.all(X[:, 0] == y_hat)
path: sklearn/model_selection/tests/test_validation.py
text: @user1:
For comparing floats, we have been moving to:

```py
assert_allclose(X[:, 0], y_hat)
```

diff_hunk: @@ -972,6 +975,18 @@ def test_cross_val_predict_unbalanced():
                               decimal=12)
 
 
+def test_cross_val_predict_y_none():
+    # ensure that cross_val_predict works when y is None
+    mock_classifier = MockClassifier(allow_nd=True)
+    X = np.random.random((100, 10))
+    y_hat = cross_val_predict(mock_classifier, X, y=None, cv=5,
+                              method='predict')
+    assert np.all(X[:, 0] == y_hat)
+    y_hat_proba = cross_val_predict(mock_classifier, X, y=None, cv=5,
+                                    method='predict_proba')
+    assert np.all(y_hat_proba == X)
path: sklearn/model_selection/tests/test_validation.py
text: @user1:
For comparing floats, we have been moving to:

```py
assert_allclose(X, y_hat_proba)
```

diff_hunk: @@ -972,6 +975,18 @@ def test_cross_val_predict_unbalanced():
                               decimal=12)
 
 
+def test_cross_val_predict_y_none():
+    # ensure that cross_val_predict works when y is None
+    mock_classifier = MockClassifier(allow_nd=True)
path: sklearn/model_selection/tests/test_validation.py
text: @user1:
`allow_nd` is not required for this test:

```suggestion
    mock_classifier = MockClassifier()
```

(So future developers do not assume `allow_nd` is required.)

diff_hunk: @@ -972,6 +975,18 @@ def test_cross_val_predict_unbalanced():
                               decimal=12)
 
 
+def test_cross_val_predict_y_none():
+    # ensure that cross_val_predict works when y is None
+    mock_classifier = MockClassifier(allow_nd=True)
+    X = np.random.random((100, 10))
path: sklearn/model_selection/tests/test_validation.py
text: @user1:
For generating random numbers:

```py
rng = np.random.RandomState(42)
X = rng.random((100, 10))
```

so this test can be deterministic.