Repository: scikit-learn/scikit-learn
Pull request: [MRG] Fixes tree and forest classification for non-numeric multi-target
Fixes #11451.

This fixes the issue that trees and forests cannot classify (but they can fit) non-numeric targets, when there are multiple targets.

Problem statement:
An error is thrown when using Random forest classifier multi-target non-numeric classes
#### Description

Random forest classifier can be fit on multiple target columns where columns have string labels and are not numeric, but when trying to predict, an error is thrown.

#### Steps/Code to Reproduce

```python
import numpy as np
from sklearn.datasets import make_classification
from sklearn.ensemble.forest import RandomForestClassifier

c = RandomForestClassifier()

X, y = make_classification()
y = np.array(['foo' if v else 'bar' for v in y]).reshape((y.shape[0], 1))
ys = np.hstack([y, y])

c.fit(X, ys)
c.predict(X)
```

#### Expected Results

No error is thrown.

#### Actual Results

```
  File ".../local/lib/python2.7/site-packages/sklearn/ensemble/forest.py", line 550, in predict
    axis=0)
ValueError: could not convert string to float: foo
```

#### Versions

```
Linux-4.13.0-45-generic-x86_64-with-Ubuntu-17.10-artful
('Python', '2.7.14 (default, Sep 23 2017, 22:06:14) \n[GCC 7.2.0]')
('NumPy', '1.14.5')
('SciPy', '1.1.0')
('Scikit-Learn', '0.19.1')
```

Hints from issue discussion:
It seems the following code fixes the problem:

```python
        if self.n_outputs_ == 1:
            return self.classes_.take(np.argmax(proba, axis=1), axis=0)

        else:
            n_samples = proba[0].shape[0]
            predictions = []

            for k in range(self.n_outputs_):
                predictions.append(self.classes_[k].take(np.argmax(proba[k], axis=1), axis=0))

            return np.array(predictions).T
```
Yeah, I don't know if we formally support string labels for multi-output
classification anywhere.

Could you offer a pull request?​

I mean, the code above looks like you are trying to support. Otherwise there would be no `self.classes_[k].take` there, no?

I can make a pull request. Is the example solution OK? Going through a list and not pre-allocate?
probably. write a test and make it pass.

Pull request with tests made: #11458
While fixing this, I searched a code a bit for similar (bad) pattern and found out that trees also do not work. So I fixed them as well.

Commit message: Adding ignore warning to test_forest.py.

Patch to review:
diff --git a/sklearn/ensemble/tests/test_forest.py b/sklearn/ensemble/tests/test_forest.py
index e4eb282923a29..ef809cc3eadb4 100644
--- a/sklearn/ensemble/tests/test_forest.py
+++ b/sklearn/ensemble/tests/test_forest.py
@@ -1337,3 +1337,23 @@ def test_backend_respected():
         clf.predict_proba(X)
 
     assert ba.count == 0
+
+
+@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
+@pytest.mark.parametrize('name', FOREST_CLASSIFIERS)
+@pytest.mark.parametrize('oob_score', (True, False))
+def test_multi_target(name, oob_score):
+    ForestClassifier = FOREST_CLASSIFIERS[name]
+
+    clf = ForestClassifier(bootstrap=True, oob_score=oob_score)
+
+    X, y = datasets.make_classification()
+    # Make y have string classes.
+    y = np.array(['foo' if v else 'bar' for v in y]).reshape((y.shape[0], 1))
+
+    # Make multi-target.
+    ys = np.hstack([y, y])
+
+    # Try to fix and predict.
+    clf.fit(X, ys)
+    clf.predict(X)
diff --git a/sklearn/tree/tests/test_tree.py b/sklearn/tree/tests/test_tree.py
index 7001b7741c4e6..30288260d9183 100644
--- a/sklearn/tree/tests/test_tree.py
+++ b/sklearn/tree/tests/test_tree.py
@@ -1828,3 +1828,21 @@ def test_empty_leaf_infinite_threshold():
         infinite_threshold = np.where(~np.isfinite(tree.tree_.threshold))[0]
         assert len(infinite_threshold) == 0
         assert len(empty_leaf) == 0
+
+
+@pytest.mark.parametrize('name', CLF_TREES)
+def test_multi_target(name):
+    Tree = CLF_TREES[name]
+
+    clf = Tree()
+
+    X, y = datasets.make_classification()
+    # Make y have string classes.
+    y = np.array(['foo' if v else 'bar' for v in y]).reshape((y.shape[0], 1))
+
+    # Make multi-target.
+    ys = np.hstack([y, y])
+
+    # Try to fix and predict.
+    clf.fit(X, ys)
+    clf.predict(X)
diff --git a/sklearn/tree/tree.py b/sklearn/tree/tree.py
index 973d7f9d1d715..f377df78c6952 100644
--- a/sklearn/tree/tree.py
+++ b/sklearn/tree/tree.py
@@ -436,8 +436,9 @@ def predict(self, X, check_input=True):
                 return self.classes_.take(np.argmax(proba, axis=1), axis=0)
 
             else:
-                predictions = np.zeros((n_samples, self.n_outputs_))
-
+                class_type = self.classes_[0].dtype
+                predictions = np.zeros((n_samples, self.n_outputs_),
+                                       dtype=class_type)
                 for k in range(self.n_outputs_):
                     predictions[:, k] = self.classes_[k].take(
                         np.argmax(proba[:, k], axis=1),


Code review comments:
diff_hunk: @@ -1337,3 +1337,23 @@ def test_backend_respected():
         clf.predict_proba(X)
 
     assert ba.count == 0
+
+
+@pytest.mark.filterwarnings('ignore:The default value of n_estimators')
+@pytest.mark.parametrize('name', FOREST_CLASSIFIERS)
+@pytest.mark.parametrize('oob_score', (True, False))
+def test_multi_target(name, oob_score):
+    ForestClassifier = FOREST_CLASSIFIERS[name]
+
+    clf = ForestClassifier(bootstrap=True, oob_score=oob_score)
+
+    X, y = datasets.make_classification()
+    # Make y have string classes.
+    y = np.array(['foo' if v else 'bar' for v in y]).reshape((y.shape[0], 1))
+
+    # Make multi-target.
+    ys = np.hstack([y, y])
+
+    # Try to fix and predict.
path: sklearn/ensemble/tests/test_forest.py
text: @user1:
fix -> fit?

@user2:
Fixed :)