Repository: DistrictDataLabs/yellowbrick

Issue:
ROCAUC.score() raises ValueError: Found input variables with inconsistent numbers of samples
**Describe the bug**

Fitting a LinearSVC model on a binary class dataset and calling the ROCAUC score() function causes an unexpected error to be raised from sklearn. If the binary=True parameter is passed into ROCAUC, the error goes away.

```ValueError: Found input variables with inconsistent numbers of samples: [4, 2]```

**To Reproduce**

```python
import numpy as np
from sklearn.svm import LinearSVC
from yellowbrick.classifier import ROCAUC

X = np.array([1, 2]).reshape(-1, 1)
y = np.array([0, 1])
model = LinearSVC()
visualizer = ROCAUC(model)
visualizer.fit(X, y)
visualizer.score(X, y)
```

**Expected behavior**

Yellowbrick should raise a more meaningful error if the binary=True parameter is required to score ROCAUC on binary class datasets.

**Traceback**

```
Traceback (most recent call last):
  File "visualizer.py", line 9, in <module>
    visualizer.score(X, y)
  File "C:\Users\DezielPa\conda\yellowbrick\yellowbrick\classifier\rocauc.py", line 330, in score
    self._score_micro_average(y, y_pred, classes, n_classes)
  File "C:\Users\DezielPa\conda\yellowbrick\yellowbrick\classifier\rocauc.py", line 486, in _score_micro_average
    self.fpr[MICRO], self.tpr[MICRO], _ = roc_curve(y.ravel(), y_pred.ravel())
  File "C:\Users\DezielPa\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 73, in inner_f
    return f(**kwargs)
  File "C:\Users\DezielPa\Anaconda3\lib\site-packages\sklearn\metrics\_ranking.py", line 776, in roc_curve
    y_true, y_score, pos_label=pos_label, sample_weight=sample_weight)
  File "C:\Users\DezielPa\Anaconda3\lib\site-packages\sklearn\metrics\_ranking.py", line 541, in _binary_clf_curve
    check_consistent_length(y_true, y_score, sample_weight)
  File "C:\Users\DezielPa\Anaconda3\lib\site-packages\sklearn\utils\validation.py", line 257, in check_consistent_length
    " samples: %r" % [int(l) for l in lengths])
ValueError: Found input variables with inconsistent numbers of samples: [4, 2]
```

**Desktop (please complete the following information):**
 - OS: Microsoft Windows 10 Enterprise
 - Python 3.7.6
 - Yellowbrick Version 1.2.1

**Additional context**

sklearn version 0.23.0



Discussion:
Thank you @pdeziel for noting this bug (and for your recent PR #1148!). 

My guess is we should take another look at the check we're doing [here](https://github.com/DistrictDataLabs/yellowbrick/blob/develop/yellowbrick/classifier/rocauc.py#L262-L268) since we seem to be missing an edge case. Maybe we can take a look together on Wednesday and open a PR?

Resolving patch:
diff --git a/yellowbrick/classifier/rocauc.py b/yellowbrick/classifier/rocauc.py
--- a/yellowbrick/classifier/rocauc.py
+++ b/yellowbrick/classifier/rocauc.py
@@ -270,6 +270,14 @@ def score(self, X, y=None):
                     "no curves will be drawn; ",
                     "set per_class=True or micro=False and macro=False.",
                 )
+
+            # For binary, if predictions are returned in shape (n,), micro and macro
+            # curves are not defined
+            if (self.micro or self.macro) and len(y_pred.shape) == 1:
+                raise ModelError(
+                    "no curves will be drawn; set binary=True.",
+                )
+
         if self.target_type_ == MULTICLASS:
             # If it's multiclass classification, at least one of micro, macro, or
             # per_class must be True
@@ -359,7 +367,7 @@ def draw(self):
         -------
         ax : the axis with the plotted figure
         """
-        colors = self.class_colors_[0: len(self.classes_)]
+        colors = self.class_colors_[0 : len(self.classes_)]
         n_classes = len(colors)
 
         # If it's a binary decision, plot the single ROC curve


Test patch:
diff --git a/tests/test_classifier/test_rocauc.py b/tests/test_classifier/test_rocauc.py
--- a/tests/test_classifier/test_rocauc.py
+++ b/tests/test_classifier/test_rocauc.py
@@ -46,6 +46,7 @@
 ## Fixtures
 ##########################################################################
 
+
 class FakeClassifier(BaseEstimator, ClassifierMixin):
     """
     A fake classifier for testing noops on the visualizer.
@@ -129,7 +130,9 @@ def test_binary_probability_decision_single_curve(self):
         Test ROCAUC binary classifier with both decision & predict_proba with per_class=False
         """
         # Create and fit the visualizer
-        visualizer = ROCAUC(AdaBoostClassifier(), micro=False, macro=False, per_class=False)
+        visualizer = ROCAUC(
+            AdaBoostClassifier(), micro=False, macro=False, per_class=False
+        )
         visualizer.fit(self.binary.X.train, self.binary.y.train)
 
         # Score the visualizer
@@ -271,9 +274,7 @@ def test_rocauc_no_curves(self):
         Test ROCAUC with no curves specified at all
         """
         # Create and fit the visualizer
-        visualizer = ROCAUC(
-            GaussianNB(), per_class=False, macro=False, micro=False
-        )
+        visualizer = ROCAUC(GaussianNB(), per_class=False, macro=False, micro=False)
         visualizer.fit(self.multiclass.X.train, self.multiclass.y.train)
 
         # Attempt to score the visualizer
@@ -432,6 +433,19 @@ def test_binary_decision_function_rocauc(self):
         # Check to see if the first 10 y_scores match the expected
         npt.assert_array_almost_equal(y_scores[:10], first_ten_expected, decimal=1)
 
+    def test_binary_false_decision_function_error(self):
+        """
+        Test binary decision_function model raises error when the binary param is False
+        """
+        # Create and fit the visualizer
+        visualizer = ROCAUC(LinearSVC(random_state=42), binary=False)
+        visualizer.fit(self.binary.X.train, self.binary.y.train)
+
+        # Ensure score raises error
+        # (only binary curve defined for binary decisions with decision_function clf)
+        with pytest.raises(ModelError):
+            visualizer.score(self.binary.X.test, self.binary.y.test)
+
     def test_multi_decision_function_rocauc(self):
         """
         Test ROCAUC with multiclass classifiers that have a decision function