Repository: scikit-learn/scikit-learn
Pull request: Check predict_proba shape in ThresholdScorer
#### Reference Issues/PRs
Continues and resolves #12221, fixes #7598

#### What does this implement/fix? Explain your changes.

#### Any other comments?
this time, I ran flake8, fixed formatting

cc:  @AMDonati

@amueller Can you point me to the GMM test? I did not see on in here:
scikit-learn/sklearn/tests

>Looks good in general, but you need to add a regression test (you could use the GMM one or just a classification one with a single class maybe)


Problem statement:
BUG: Using GridSearchCV with scoring='roc_auc' and GMM as classifier gives IndexError
When performing grid search using GridSearchCV using ootb scoring method 'roc_auc' and ootb GMM classifier from sklearn.mixture.GMM I get an index error.
Code to reproduce:

```
from sklearn import datasets
from sklearn.grid_search import GridSearchCV
from sklearn.mixture import GMM
X,y = datasets.make_classification(n_samples = 10000, n_features=10,n_classes=2)
# Vanilla GMM_model
gmm_model = GMM()
# Standard param grid
param_grid = {'n_components' : [1,2,3,4],
              'covariance_type': ['tied','full','spherical']}
grid_search = GridSearchCV(gmm_model, param_grid, scoring='roc_auc')
# Fit GS with this data
grid_search.fit(X, y)
```

Sorry if the format is incorrect. First time I am posting.

ERROR:
  File "*/python2.7/site-packages/sklearn/metrics/scorer.py", line 175, in **call**
    y_pred = y_pred[:, 1]
IndexError: index 1 is out of bounds for axis 1 with size 1

Hints from issue discussion:
Updated to 0.17.1 and issue persists ( Changing GMM to GaussianMixture)

The error is strange, but GMM is not a supervised model, so AUC doesn't really make sense.
We might want to raise a better error, though it's hard to detect what's going on here in a sense.

Do you really mean updated to 0.17.1, not 0.18?

On 8 October 2016 at 03:28, Andreas Mueller notifications@github.com
wrote:

> The error is strange, but GMM is not a supervised model, so AUC doesn't
> really make sense.
> 
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> https://github.com/scikit-learn/scikit-learn/issues/7598#issuecomment-252298229,
> or mute the thread
> https://github.com/notifications/unsubscribe-auth/AAEz6_dNOBwzUCFpb4N1bNKGAjC02ZEDks5qxnMlgaJpZM4KRC_m
> .

Getting the same error with

```
    cv = GridSearchCV(
        estimator=DecisionTreeClassifier(),
        param_grid={
            'max_depth': [20],
            'class_weight': ['auto'],
            'min_samples_split': [100],
            'min_samples_leaf': [30],
            'criterion': ['gini']
        },
        scoring='roc_auc',
        n_jobs=-1,
    )
```

Log:

```
__call__(self=make_scorer(roc_auc_score, needs_threshold=True), clf=DecisionTreeClassifier(class_weight='auto', crit...resort=False, random_state=None, splitter='best'), X=memmap([[ 2.14686672e-01, 0.00000000e+00, 0...000000e+00, 1.00000000e+00, 0.00000000e+00]]), y=memmap([0, 0, 0, ..., 0, 0, 0]), sample_weight=None) 
170 
171 except (NotImplementedError, AttributeError): 
172 y_pred = clf.predict_proba(X) 
173 
174 if y_type == "binary": 
--> 175 y_pred = y_pred[:, 1] 
y_pred = array([[ 1.], 
[ 1.], 
[ 1.], 
..., 
[ 1.], 
[ 1.], 
[ 1.]]) 
176 elif isinstance(y_pred, list): 
177 y_pred = np.vstack([p[:, -1] for p in y_pred]).T 
178 
179 if sample_weight is not None: 

IndexError: index 1 is out of bounds for axis 1 with size 1 
```

It looks there like you might have been training your `DecisionTreeClassifier` on a single class... what does the `y` you pass to `GridSearchCV.fit` look like?

Yes, this error message is not very helpful.

I ran into this error and you are correct @amueller about the single class explanation. Here's what my data looks like.

```
X_test.shape: (750, 34)
y_test.shape: (750,)
y_test value_counts: True    750
```
So my data contains a single class: `True`.

Perhaps, a more descriptive error message would help. Something along the line of your comment: `It looks like you might have a single class`. Looking at line 175 long enough may give that away too.
Can someone help with this issue? I do not know how to fix it still.  Thank you!
clf = ExtraTreesClassifier()
my_cv = TimeSeriesSplit(n_splits=5)  # time series split

param_grid = {
              'n_estimators': [100, 300, 500, 700, 1000, 2000, 5000],
              'min_samples_split': [2, 5, 10],
              'min_samples_leaf': range(2,20,2),
              'bootstrap': [True, False]
             }

clf = GridSearchCV(estimator=clf, param_grid=param_grid, cv=my_cv, n_jobs=-1, scoring='roc_auc', return_train_score=False)
clf.fit(X, y)
@liuwanfei you likely have just one class in ```y```, at least looks like it was an issue for most people in this thread. It should be an error message with a clear text instead of the exception.
Yes, what the people above have mentioned is correct - if you train with one class you _will_ get this error.

However, if you have a look at my code, I generated a dataset which has 2 classes so that was not the case with me. What _was_ the causing the issue is that my param grid was set up with a subtle error.  Remember the "roc_auc" scorer is using probabilities as inputs to create the ROC curve, and in my example above, my parameter space for `n_components:` was `[1,2,3,4]`. 

If you think about it, a GMM with one component will output only one probability. Thus, the output of `model.predict_proba` will be a one dimensional array which is why an `IndexError` occurs on `pred[:, 1]`.

So, for your case, see if one of the parameter combinations might not result in the classifier being constrained to predicting a single class. 

P.S I only realised this _now_, almost 2 years after the post. lol

Commit message: add regression test, also add fix for proba scorers

Patch to review:
diff --git a/sklearn/metrics/scorer.py b/sklearn/metrics/scorer.py
index 59ecd65386823..e500fab82a86c 100644
--- a/sklearn/metrics/scorer.py
+++ b/sklearn/metrics/scorer.py
@@ -126,7 +126,12 @@ def __call__(self, clf, X, y, sample_weight=None):
         y_type = type_of_target(y)
         y_pred = clf.predict_proba(X)
         if y_type == "binary":
-            y_pred = y_pred[:, 1]
+            if y_pred.shape[1] == 2:
+                y_pred = y_pred[:, 1]
+            else:
+                raise ValueError('Must use classifier with two classes'
+                                 ' for {} scoring'.format(
+                                     self._score_func.__name__))
         if sample_weight is not None:
             return self._sign * self._score_func(y, y_pred,
                                                  sample_weight=sample_weight,
@@ -183,7 +188,12 @@ def __call__(self, clf, X, y, sample_weight=None):
                 y_pred = clf.predict_proba(X)
 
                 if y_type == "binary":
-                    y_pred = y_pred[:, 1]
+                    if y_pred.shape[1] == 2:
+                        y_pred = y_pred[:, 1]
+                    else:
+                        raise ValueError('Must use classifier with two classes'
+                                         ' for {} scoring'.format(
+                                             self._score_func.__name__))
                 elif isinstance(y_pred, list):
                     y_pred = np.vstack([p[:, -1] for p in y_pred]).T
 
diff --git a/sklearn/metrics/tests/test_score_objects.py b/sklearn/metrics/tests/test_score_objects.py
index a676541743b71..ef8d953526f85 100644
--- a/sklearn/metrics/tests/test_score_objects.py
+++ b/sklearn/metrics/tests/test_score_objects.py
@@ -370,7 +370,21 @@ def test_thresholded_scorers():
     X, y = make_blobs(random_state=0, centers=3)
     X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
     clf.fit(X_train, y_train)
-    assert_raises(ValueError, get_scorer('roc_auc'), clf, X_test, y_test)
+    with pytest.raises(ValueError, match="multiclass format is not supported"):
+        get_scorer('roc_auc')(clf, X_test, y_test)
+
+    # test error is raised with a single class present in model
+    # (predict_proba shape is not suitable for binary auc)
+    X, y = make_blobs(random_state=0, centers=2)
+    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)
+    clf = DecisionTreeClassifier()
+    clf.fit(X_train, np.zeros_like(y_train))
+    with pytest.raises(ValueError, match="use classifier with two classes"):
+        get_scorer('roc_auc')(clf, X_test, y_test)
+
+    # for proba scorers
+    with pytest.raises(ValueError, match="use classifier with two classes"):
+        get_scorer('neg_log_loss')(clf, X_test, y_test)
 
 
 def test_thresholded_scorers_multilabel_indicator_data():


Code review comments:
diff_hunk: @@ -126,7 +126,12 @@ def __call__(self, clf, X, y, sample_weight=None):
         y_type = type_of_target(y)
         y_pred = clf.predict_proba(X)
         if y_type == "binary":
-            y_pred = y_pred[:, 1]
+            if y_pred.shape[1] == 2:
+                y_pred = y_pred[:, 1]
+            else:
+                raise ValueError('Must use classifier with two classes'
path: sklearn/metrics/scorer.py
text: @user1:
should we say "got predict_proba of shape"?

@author:
That seems like a good idea, because that is what is actually happening.  

*But*, will the *user* understand that?  Is there some place in the documentation where we can write:
>When obtaining predictions from a classifier, if it is only one class, the returned ValueError will be "got predict_proba of shape".  That means it returns one column all with same probabilities (equal to 1). 
**Q:** How to fix this?
**A:** Must use a classifier with 2 classes.

Or will they be able to see it from the test below:  
`with pytest.raises(ValueError, match="use classifier with two classes")`
I would feel better if the user had explicit feedback on how to solve the error.

cc:  @user2



@user1:
Are you saying the current error message is not informative enough? The test is just to make sure that the error that is raised is actually informative.
I suggested adding the predict_proba shape to the error message, not replace the message that's there now.

@author:
That works.
I tried making the update myself but ran into multiple problems:  [errors list](https://gist.github.com/reshamas/93af459fd61aed10ff3c162ca62a8a98)

@author:
ok, making progress.  Please let me know next steps. 

cc:  @user2