Repository: scikit-learn/scikit-learn
Pull request: [MRG] Allow scoring of dummies without testsamples
As DummyClassifier and DummyRegressor operate solely on the targets,
they can now be used without passing test samples, instead passing None.
Also includes some minor renaming in the corresponding tests for more
consistency.

<!--
Thanks for contributing a pull request! Please ensure you have taken a look at
the contribution guidelines: https://github.com/scikit-learn/scikit-learn/blob/master/CONTRIBUTING.md#pull-request-checklist
-->

#### Reference Issues/PRs
Resolves #11951 
<!--
Example: Fixes #1234. See also #3456.
Please use keywords (e.g., Fixes) to create link to the issues or pull requests
you resolved, so that they will automatically be closed when your pull request
is merged. See https://github.com/blog/1506-closing-issues-via-pull-requests
-->





<!--
Please be aware that we are a loose team of volunteers so patience is
necessary; assistance handling other issues is very welcome. We value
all user contributions, no matter how minor they are. If we are slow to
review, either the pull request needs some benchmarking, tinkering,
convincing, etc. or more likely the reviewers are simply busy. In either
case, we ask for your understanding during the review process.
For more information, see our FAQ on this topic:
http://scikit-learn.org/dev/faq.html#why-is-my-pull-request-not-getting-any-attention.

Thanks for contributing!
-->


Problem statement:
Scoring for dummy classifier does not work without test samples
When using the `Dummy classifier`, it is possible to fit the classifier without providing any examples, which makes sense as the classifier only operates on the targets. However, it is not possible to score the classifier without providing examples. Using `DummyClassifier` without examples is helpful if the examples, but not the targets, are to big to fit into memory. One can still get around this by constructing 
artificial examples, say just zeros, but avoiding this would make things a bit easier.

#### Code to Reproduce
```python
from sklearn.dummy import DummyClassifier
import numpy as np

y = [1, 1, 2]
y_ = [1, 1, 2]

d = DummyClassifier()
d.fit(None, y)
print(d.score(None, y_))
```


#### Expected Results
The score is printed.

#### Actual Results
An exception is thrown
```
ValueError: Found input variables with inconsistent numbers of samples: [3, 1]
```

So one has to do

```python
from sklearn.dummy import DummyClassifier
import numpy as np

y = [1, 1, 2]
y_ = [1, 1, 2]
x = np.zeros(shape=(3, 1))

d = DummyClassifier()
d.fit(None, y)
print(d.score(x,  y_))
```

If this seems like a useful feature, I would be happy to submit a PR.

#### Versions
Linux-4.15.0-33-generic-x86_64-with-debian-buster-sid
Python 3.6.4 |Anaconda, Inc.| (default, Jan 16 2018, 18:10:19) 
[GCC 7.2.0]
NumPy 1.14.3
SciPy 1.0.0
Scikit-Learn 0.19.2

<!-- Thanks for contributing! -->

Hints from issue discussion:
I suppose this would be consistent. PR welcome

Commit message: Add newline at end of file

Patch to review:
diff --git a/sklearn/dummy.py b/sklearn/dummy.py
index f2c866413183b..0e5371ac42239 100644
--- a/sklearn/dummy.py
+++ b/sklearn/dummy.py
@@ -318,6 +318,36 @@ def predict_log_proba(self, X):
         else:
             return [np.log(p) for p in proba]
 
+    def score(self, X, y, sample_weight=None):
+        """Returns the mean accuracy on the given test data and labels.
+
+        In multi-label classification, this is the subset accuracy
+        which is a harsh metric since you require for each sample that
+        each label set be correctly predicted.
+
+        Parameters
+        ----------
+        X : {array-like, None}
+            Test samples with shape = (n_samples, n_features) or
+            None. Passing None as test samples gives the same result
+            as passing real test samples, since DummyClassifier
+            operates solely on the targets.
+
+        y : array-like, shape = (n_samples) or (n_samples, n_outputs)
+            True labels for X.
+
+        sample_weight : array-like, shape = [n_samples], optional
+            Sample weights.
+
+        Returns
+        -------
+        score : float
+            Mean accuracy of self.predict(X) wrt. y.
+
+        """
+        X = _ensure_consistent_length(X, y)
+        return super(DummyClassifier, self).score(X, y, sample_weight)
+
 
 class DummyRegressor(BaseEstimator, RegressorMixin):
     """
@@ -478,3 +508,46 @@ def predict(self, X, return_std=False):
             y_std = np.ravel(y_std)
 
         return (y, y_std) if return_std else y
+
+    def score(self, X, y, sample_weight=None):
+        """Returns the coefficient of determination R^2 of the prediction.
+
+        The coefficient R^2 is defined as (1 - u/v), where u is the residual
+        sum of squares ((y_true - y_pred) ** 2).sum() and v is the total
+        sum of squares ((y_true - y_true.mean()) ** 2).sum().
+        The best possible score is 1.0 and it can be negative (because the
+        model can be arbitrarily worse). A constant model that always
+        predicts the expected value of y, disregarding the input features,
+        would get a R^2 score of 0.0.
+
+        Parameters
+        ----------
+        X : {array-like, None}
+            Test samples with shape = (n_samples, n_features) or None.
+            For some estimators this may be a
+            precomputed kernel matrix instead, shape = (n_samples,
+            n_samples_fitted], where n_samples_fitted is the number of
+            samples used in the fitting for the estimator.
+            Passing None as test samples gives the same result
+            as passing real test samples, since DummyRegressor
+            operates solely on the targets.
+
+        y : array-like, shape = (n_samples) or (n_samples, n_outputs)
+            True values for X.
+
+        sample_weight : array-like, shape = [n_samples], optional
+            Sample weights.
+
+        Returns
+        -------
+        score : float
+            R^2 of self.predict(X) wrt. y.
+        """
+        X = _ensure_consistent_length(X, y)
+        return super(DummyRegressor, self).score(X, y, sample_weight)
+
+
+def _ensure_consistent_length(X, y):
+    if X is None:
+        X = np.zeros(shape=(len(y), 1))
+    return X
diff --git a/sklearn/tests/test_dummy.py b/sklearn/tests/test_dummy.py
index 5d955f51017a1..0ecbfd84c905b 100644
--- a/sklearn/tests/test_dummy.py
+++ b/sklearn/tests/test_dummy.py
@@ -200,6 +200,30 @@ def test_string_labels():
     assert_array_equal(clf.predict(X), ["paris"] * 5)
 
 
+def test_classifier_score_with_None():
+    y = [2, 1, 1, 1]
+    y_test = [2, 2, 1, 1]
+
+    clf = DummyClassifier(strategy="constant", constant=1)
+    clf.fit(None, y)
+    assert_equal(clf.score(None, y_test), 0.5)
+
+
+def test_classifier_score_with_None_multioutput():
+    y = np.array([[2, 2],
+                  [1, 1],
+                  [1, 1],
+                  [1, 1]])
+    y_test = np.array([[2, 2],
+                       [2, 2],
+                       [1, 1],
+                       [1, 1]])
+
+    clf = DummyClassifier(strategy="most_frequent")
+    clf.fit(None, y)
+    assert_equal(clf.score(None, y_test), 0.5)
+
+
 def test_classifier_exceptions():
     clf = DummyClassifier(strategy="unknown")
     assert_raises(ValueError, clf.fit, [], [])
@@ -633,3 +657,25 @@ def test_dummy_regressor_return_std():
     assert_equal(len(y_pred_list), 2)
     # the second element should be all zeros
     assert_array_equal(y_pred_list[1], y_std_expected)
+
+
+def test_regressor_score_with_None():
+    y = [1, 1, 1, 2]
+    y_test = [1.25] * 4
+
+    reg = DummyRegressor()
+    reg.fit(None, y)
+    assert_equal(reg.score(None, y_test), 1.0)
+
+
+def test_regressor_score_with_None_multioutput():
+    y = np.array([[2, 2],
+                  [1, 1],
+                  [1, 1],
+                  [1, 1]])
+
+    y_test = [[1.25, 1.25]] * 4
+
+    reg = DummyRegressor()
+    reg.fit(None, y)
+    assert_equal(reg.score(None, y_test), 1.0)


Code review comments:
diff_hunk: @@ -478,3 +508,46 @@ def predict(self, X, return_std=False):
             y_std = np.ravel(y_std)
 
         return (y, y_std) if return_std else y
+
+    def score(self, X, y, sample_weight=None):
+        """Returns the coefficient of determination R^2 of the prediction.
+
+        The coefficient R^2 is defined as (1 - u/v), where u is the residual
+        sum of squares ((y_true - y_pred) ** 2).sum() and v is the total
+        sum of squares ((y_true - y_true.mean()) ** 2).sum().
+        The best possible score is 1.0 and it can be negative (because the
+        model can be arbitrarily worse). A constant model that always
+        predicts the expected value of y, disregarding the input features,
+        would get a R^2 score of 0.0.
+
+        Parameters
+        ----------
+        X : {array-like, None}
+            Test samples with shape = (n_samples, n_features) or None.
+            For some estimators this may be a
+            precomputed kernel matrix instead, shape = (n_samples,
+            n_samples_fitted], where n_samples_fitted is the number of
+            samples used in the fitting for the estimator.
+            Passing None as test samples gives the same result
+            as passing real test samples, since DummyRegressor
+            operates solely on the targets.
+
+        y : array-like, shape = (n_samples) or (n_samples, n_outputs)
+            True values for X.
+
+        sample_weight : array-like, shape = [n_samples], optional
+            Sample weights.
+
+        Returns
+        -------
+        score : float
+            R^2 of self.predict(X) wrt. y.
+        """
+        X = _ensure_consistent_length(X, y)
+        return super(DummyRegressor, self).score(X, y, sample_weight)
+
+
+def _ensure_consistent_length(X, y):
path: sklearn/dummy.py
text: @user1:
For readability, I think you should just inline this code, and not use a separate function.

@author:
Well, I thought that using a function with a speaking name would actually increase readability and avoid duplication, as the functionality is needed for both the classifier und the regressor. Maybe you can elaborate on how inlining the code would improve readability. I can of course change that, if you want.

@user1:
A function name *can* improve readability, but I don't think that the meaning of `_ensure_consistent_length` is obvious enough to do that. Perhaps `_handle_no_x` would be clearer.

@author:
I renamed the function, let me know what you think.

diff_hunk: @@ -200,6 +200,30 @@ def test_string_labels():
     assert_array_equal(clf.predict(X), ["paris"] * 5)
 
 
+def test_classifier_score_with_None():
path: sklearn/tests/test_dummy.py
text: @user1:
you can use pytest.mark.parametrize to simply pass two different versions of `y` and `y_test`, avoiding the duplication between these two tests

diff_hunk: @@ -478,3 +508,46 @@ def predict(self, X, return_std=False):
             y_std = np.ravel(y_std)
 
         return (y, y_std) if return_std else y
+
+    def score(self, X, y, sample_weight=None):
+        """Returns the coefficient of determination R^2 of the prediction.
+
+        The coefficient R^2 is defined as (1 - u/v), where u is the residual
+        sum of squares ((y_true - y_pred) ** 2).sum() and v is the total
+        sum of squares ((y_true - y_true.mean()) ** 2).sum().
+        The best possible score is 1.0 and it can be negative (because the
+        model can be arbitrarily worse). A constant model that always
+        predicts the expected value of y, disregarding the input features,
+        would get a R^2 score of 0.0.
+
+        Parameters
+        ----------
+        X : {array-like, None}
+            Test samples with shape = (n_samples, n_features) or None.
+            For some estimators this may be a
+            precomputed kernel matrix instead, shape = (n_samples,
+            n_samples_fitted], where n_samples_fitted is the number of
+            samples used in the fitting for the estimator.
+            Passing None as test samples gives the same result
+            as passing real test samples, since DummyRegressor
+            operates solely on the targets.
+
+        y : array-like, shape = (n_samples) or (n_samples, n_outputs)
+            True values for X.
+
+        sample_weight : array-like, shape = [n_samples], optional
+            Sample weights.
+
+        Returns
+        -------
+        score : float
+            R^2 of self.predict(X) wrt. y.
+        """
+        X = _ensure_consistent_length(X, y)
+        return super(DummyRegressor, self).score(X, y, sample_weight)
path: sklearn/dummy.py
text: @author:
Is this code meant to be Python 2 compatible? Otherwise we could just use `super()` instead of `super(DummyRegressor, self)`

@user1:
We haven't yet removed Python 2 from continuous integration, so it can be soon...

@user1:
Feel free to help out at #11989 

@author:
Ok, I will make a another PR once #11995 is merged.