Repository: scikit-learn/scikit-learn
Pull request: [MRG] MAINT add base class for voting and stacking
closes #15056 

Create a base class for Voting* and Stacking*. They both are an ensemble of multiple learners type.
They could share the `get_params`, `set_params` and validation of `estimators` (as well as the fitted attributes then).

This base class could be contrasted with the ensemble of single learner type such as boosting (adaboost, GBDT), RF and Bagging.

Problem statement:
VotingClassifier and roc_auc TypeError: Cannot cast array data from dtype('float64') to dtype('int64') according to the rule 'safe' and
#### Description
VotingClassifier
TypeError: Cannot cast array data from dtype('float64') to dtype('int64') according to the rule 'safe'

#### Steps/Code to Reproduce
```python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler, Normalizer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.ensemble import VotingClassifier
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Ridge
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import roc_auc_score

pipePre = Pipeline([
    ('simpleimputer', SimpleImputer(missing_values=np.nan, strategy='constant', fill_value=0)),
    ('standardscaler', StandardScaler()),
    ('normalizer', Normalizer())
     ])

df_train_x = pipePre.fit_transform(df_train_x)

X_train, X_test, y_train, y_test = train_test_split(df_train_x, df_train_y, test_size = 0.25, random_state=42)

lrg = LinearRegression().fit(X_train, y_train)

rig = Ridge().fit(X_train, y_train)

lreg = LogisticRegression().fit(X_train, y_train)

voting = VotingClassifier(estimators=[('lrg_v', lrg), ('rig_v', rig), 
                                      ('lreg_v', lreg)], voting='hard')
voting_fit = voting.fit(X_train, y_train)

y_pred = voting_fit.predict(X_test)
roc_auc_score(y_test, y_pred)

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-50-506a80086b81> in <module>
----> 1 val_error(voting_fit, X_test, y_test)

<ipython-input-6-0fa46ec754f8> in val_error(model, tested, prediction)
     14         Data, prepaired as tested labels
     15     """
---> 16     y_pred = model.predict(tested)
     17     err = roc_auc_score(prediction, y_pred)
     18     return err

~\Anaconda3\lib\site-packages\sklearn\ensemble\voting.py in predict(self, X)
    302                 lambda x: np.argmax(
    303                     np.bincount(x, weights=self._weights_not_none)),
--> 304                 axis=1, arr=predictions)
    305 
    306         maj = self.le_.inverse_transform(maj)

~\Anaconda3\lib\site-packages\numpy\lib\shape_base.py in apply_along_axis(func1d, axis, arr, *args, **kwargs)
    378     except StopIteration:
    379         raise ValueError('Cannot apply_along_axis when any iteration dimensions are 0')
--> 380     res = asanyarray(func1d(inarr_view[ind0], *args, **kwargs))
    381 
    382     # build a buffer for storing evaluations of func1d.

~\Anaconda3\lib\site-packages\sklearn\ensemble\voting.py in <lambda>(x)
    301             maj = np.apply_along_axis(
    302                 lambda x: np.argmax(
--> 303                     np.bincount(x, weights=self._weights_not_none)),
    304                 axis=1, arr=predictions)
    305 

TypeError: Cannot cast array data from dtype('float64') to dtype('int64') according to the rule 'safe'

```

scikit-learn  0.21.2  anaconda


<!-- Thanks for contributing! -->

Hints from issue discussion:
`Ridge` and `LinearRegression` are not classifiers, which makes them incompatible with `VotingClassifier`.
> Ridge and LinearRegression are not classifiers, which makes them incompatible with VotingClassifier.

+1 though maybe we should return a better error message.
Shall we check the base estimators with `sklearn.base.is_classifier` in the `VotingClassifier.__init__` or `fit` and raise a `ValueError`? 
> Shall we check the base estimators with sklearn.base.is_classifier in the VotingClassifier.__init__ or fit and raise a ValueError?

We have something similar for the StackingClassifier and StackingRegressor.

Actually, these 2 classes shared something in common by having `estimators` parameters. In some way they are an ensemble of "heterogeneous" estimators (i.e. multiple estimators) while bagging, RF, GBDT is an ensemble of "homogeneous" estimator (i.e. single "base_estimator").

We could have a separate base class for each type. Not sure about the naming but it will reduce code redundancy and make checking easier and consistent.

Commit message: remove parameters from base class

Patch to review:
diff --git a/doc/whats_new/v0.22.rst b/doc/whats_new/v0.22.rst
index ca92bcd57ed97..b2653f8c5565f 100644
--- a/doc/whats_new/v0.22.rst
+++ b/doc/whats_new/v0.22.rst
@@ -236,6 +236,11 @@ Changelog
   :user:`Matt Hancock <notmatthancock>` and
   :pr:`5963` by :user:`Pablo Duboue <DrDub>`.
 
+- |Fix| Enforce underlying estimator to be the same type than voting estimator.
+  We introduced a new base class to raise consistent error message in
+  Stacking and Voting estimator.
+  By :pr:`15084` :user:`Guillaume Lemaitre <glemaitre>`.
+
 :mod:`sklearn.feature_extraction`
 .................................
 
diff --git a/sklearn/ensemble/_stacking.py b/sklearn/ensemble/_stacking.py
index 97f66aa077772..004ef7931cce5 100644
--- a/sklearn/ensemble/_stacking.py
+++ b/sklearn/ensemble/_stacking.py
@@ -15,6 +15,7 @@
 from ..base import MetaEstimatorMixin
 
 from .base import _parallel_fit_estimator
+from .base import _BaseHeterogeneousEnsemble
 
 from ..linear_model import LogisticRegression
 from ..linear_model import RidgeCV
@@ -32,80 +33,26 @@
 from ..utils.validation import column_or_1d
 
 
-class _BaseStacking(TransformerMixin, MetaEstimatorMixin, _BaseComposition,
+class _BaseStacking(TransformerMixin, _BaseHeterogeneousEnsemble,
                     metaclass=ABCMeta):
     """Base class for stacking method."""
-    _required_parameters = ['estimators']
 
     @abstractmethod
     def __init__(self, estimators, final_estimator=None, cv=None,
                  stack_method='auto', n_jobs=None, verbose=0):
-        self.estimators = estimators
+        super().__init__(estimators=estimators)
         self.final_estimator = final_estimator
         self.cv = cv
         self.stack_method = stack_method
         self.n_jobs = n_jobs
         self.verbose = verbose
 
-    @abstractmethod
-    def _validate_estimators(self):
-        if self.estimators is None or len(self.estimators) == 0:
-            raise ValueError(
-                "Invalid 'estimators' attribute, 'estimators' should be a list"
-                " of (string, estimator) tuples."
-            )
-        names, estimators = zip(*self.estimators)
-        self._validate_names(names)
-        return names, estimators
-
     def _clone_final_estimator(self, default):
         if self.final_estimator is not None:
             self.final_estimator_ = clone(self.final_estimator)
         else:
             self.final_estimator_ = clone(default)
 
-    def set_params(self, **params):
-        """Set the parameters for the stacking estimator.
-
-        Valid parameter keys can be listed with `get_params()`.
-
-        Parameters
-        ----------
-        params : keyword arguments
-            Specific parameters using e.g.
-            `set_params(parameter_name=new_value)`. In addition, to setting the
-            parameters of the stacking estimator, the individual estimator of
-            the stacking estimators can also be set, or can be removed by
-            setting them to 'drop'.
-
-        Examples
-        --------
-        In this example, the RandomForestClassifier is removed.
-
-        >>> from sklearn.linear_model import LogisticRegression
-        >>> from sklearn.ensemble import RandomForestClassifier
-        >>> from sklearn.ensemble import VotingClassifier
-        >>> clf1 = LogisticRegression()
-        >>> clf2 = RandomForestClassifier()
-        >>> eclf = StackingClassifier(estimators=[('lr', clf1), ('rf', clf2)])
-        >>> eclf.set_params(rf='drop')
-        StackingClassifier(estimators=[('lr', LogisticRegression()),
-                                        ('rf', 'drop')])
-        """
-        super()._set_params('estimators', **params)
-        return self
-
-    def get_params(self, deep=True):
-        """Get the parameters of the stacking estimator.
-
-        Parameters
-        ----------
-        deep : bool
-            Setting it to True gets the various classifiers and the parameters
-            of the classifiers as well.
-        """
-        return super()._get_params('estimators', deep=deep)
-
     def _concatenate_predictions(self, predictions):
         """Concatenate the predictions of each first layer learner.
 
@@ -172,13 +119,6 @@ def fit(self, X, y, sample_weight=None):
         names, all_estimators = self._validate_estimators()
         self._validate_final_estimator()
 
-        has_estimator = any(est != 'drop' for est in all_estimators)
-        if not has_estimator:
-            raise ValueError(
-                "All estimators are dropped. At least one is required "
-                "to be an estimator."
-            )
-
         stack_method = [self.stack_method] * len(all_estimators)
 
         # Fit the base estimators on the whole training data. Those
@@ -416,16 +356,6 @@ def __init__(self, estimators, final_estimator=None, cv=None,
             verbose=verbose
         )
 
-    def _validate_estimators(self):
-        names, estimators = super()._validate_estimators()
-        for est in estimators:
-            if est != 'drop' and not is_classifier(est):
-                raise ValueError(
-                    "The estimator {} should be a classifier."
-                    .format(est.__class__.__name__)
-                )
-        return names, estimators
-
     def _validate_final_estimator(self):
         self._clone_final_estimator(default=LogisticRegression())
         if not is_classifier(self.final_estimator_):
@@ -651,16 +581,6 @@ def __init__(self, estimators, final_estimator=None, cv=None, n_jobs=None,
             verbose=verbose
         )
 
-    def _validate_estimators(self):
-        names, estimators = super()._validate_estimators()
-        for est in estimators:
-            if est != 'drop' and not is_regressor(est):
-                raise ValueError(
-                    "The estimator {} should be a regressor."
-                    .format(est.__class__.__name__)
-                )
-        return names, estimators
-
     def _validate_final_estimator(self):
         self._clone_final_estimator(default=RidgeCV())
         if not is_regressor(self.final_estimator_):
diff --git a/sklearn/ensemble/base.py b/sklearn/ensemble/base.py
index b266c38bcbfaa..9fd67594fcb7c 100644
--- a/sklearn/ensemble/base.py
+++ b/sklearn/ensemble/base.py
@@ -5,16 +5,20 @@
 # Authors: Gilles Louppe
 # License: BSD 3 clause
 
-import numpy as np
+from abc import ABCMeta, abstractmethod
 import numbers
 
+import numpy as np
+
 from joblib import effective_n_jobs
 
 from ..base import clone
+from ..base import is_classifier, is_regressor
 from ..base import BaseEstimator
 from ..base import MetaEstimatorMixin
+from ..utils import Bunch
 from ..utils import check_random_state
-from abc import ABCMeta, abstractmethod
+from ..utils.metaestimators import _BaseComposition
 
 MAX_RAND_SEED = np.iinfo(np.int32).max
 
@@ -178,3 +182,76 @@ def _partition_estimators(n_estimators, n_jobs):
     starts = np.cumsum(n_estimators_per_job)
 
     return n_jobs, n_estimators_per_job.tolist(), [0] + starts.tolist()
+
+
+class _BaseHeterogeneousEnsemble(MetaEstimatorMixin, _BaseComposition,
+                                 metaclass=ABCMeta):
+    """Base class for ensemble learners based on heterogeneous estimators."""
+    _required_parameters = ['estimators']
+
+    @property
+    def named_estimators(self):
+        return Bunch(**dict(self.estimators))
+
+    @abstractmethod
+    def __init__(self, estimators):
+        self.estimators = estimators
+
+    def _validate_estimators(self):
+        if self.estimators is None or len(self.estimators) == 0:
+            raise AttributeError(
+                "Invalid 'estimators' attribute, 'estimators' should be a list"
+                " of (string, estimator) tuples."
+            )
+        names, estimators = zip(*self.estimators)
+        # defined by MetaEstimatorMixin
+        self._validate_names(names)
+
+        has_estimator = any(est not in (None, 'drop') for est in estimators)
+        if not has_estimator:
+            raise ValueError(
+                "All estimators are dropped. At least one is required "
+                "to be an estimator."
+            )
+
+        is_estimator_type = (is_classifier if is_classifier(self)
+                             else is_regressor)
+
+        for est in estimators:
+            if est not in (None, 'drop') and not is_estimator_type(est):
+                raise ValueError(
+                    "The estimator {} should be a {}."
+                    .format(
+                        est.__class__.__name__, is_estimator_type.__name__[3:]
+                    )
+                )
+
+        return names, estimators
+
+    def set_params(self, **params):
+        """Set the parameters of an estimator from the ensemble.
+
+        Valid parameter keys can be listed with `get_params()`.
+
+        Parameters
+        ----------
+        params : keyword arguments
+            Specific parameters using e.g.
+            `set_params(parameter_name=new_value)`. In addition, to setting the
+            parameters of the stacking estimator, the individual estimator of
+            the stacking estimators can also be set, or can be removed by
+            setting them to 'drop'.
+        """
+        super()._set_params('estimators', **params)
+        return self
+
+    def get_params(self, deep=True):
+        """Get the parameters of an estimator from the ensemble.
+
+        Parameters
+        ----------
+        deep : bool
+            Setting it to True gets the various classifiers and the parameters
+            of the classifiers as well.
+        """
+        return super()._get_params('estimators', deep=deep)
diff --git a/sklearn/ensemble/tests/test_stacking.py b/sklearn/ensemble/tests/test_stacking.py
index 3a61456a5665e..93b25b6efff44 100644
--- a/sklearn/ensemble/tests/test_stacking.py
+++ b/sklearn/ensemble/tests/test_stacking.py
@@ -223,10 +223,10 @@ def fit(self, X, y):
     "y, params, type_err, msg_err",
     [(y_iris,
       {'estimators': None},
-      ValueError, "Invalid 'estimators' attribute,"),
+      AttributeError, "Invalid 'estimators' attribute,"),
      (y_iris,
       {'estimators': []},
-      ValueError, "Invalid 'estimators' attribute,"),
+      AttributeError, "Invalid 'estimators' attribute,"),
      (y_iris,
       {'estimators': [('lr', LinearRegression()),
                       ('svm', LinearSVC(max_iter=5e4))]},
@@ -265,10 +265,10 @@ def test_stacking_classifier_error(y, params, type_err, msg_err):
     "y, params, type_err, msg_err",
     [(y_diabetes,
       {'estimators': None},
-      ValueError, "Invalid 'estimators' attribute,"),
+      AttributeError, "Invalid 'estimators' attribute,"),
      (y_diabetes,
       {'estimators': []},
-      ValueError, "Invalid 'estimators' attribute,"),
+      AttributeError, "Invalid 'estimators' attribute,"),
      (y_diabetes,
       {'estimators': [('lr', LogisticRegression()), ('svm', LinearSVR())]},
       ValueError, 'should be a regressor'),
diff --git a/sklearn/ensemble/tests/test_voting.py b/sklearn/ensemble/tests/test_voting.py
index 84fafc4f74eb3..ff13002ce0857 100644
--- a/sklearn/ensemble/tests/test_voting.py
+++ b/sklearn/ensemble/tests/test_voting.py
@@ -37,8 +37,8 @@
 
 def test_estimator_init():
     eclf = VotingClassifier(estimators=[])
-    msg = ('Invalid `estimators` attribute, `estimators` should be'
-           ' a list of (string, estimator) tuples')
+    msg = ("Invalid 'estimators' attribute, 'estimators' should be"
+           " a list of (string, estimator) tuples.")
     assert_raise_message(AttributeError, msg, eclf.fit, X, y)
 
     clf = LogisticRegression(random_state=1)
@@ -417,7 +417,7 @@ def test_set_estimator_none(drop):
     eclf2.set_params(voting='soft').fit(X, y)
     assert_array_equal(eclf1.predict(X), eclf2.predict(X))
     assert_array_almost_equal(eclf1.predict_proba(X), eclf2.predict_proba(X))
-    msg = 'All estimators are None or "drop". At least one is required!'
+    msg = 'All estimators are dropped. At least one is required'
     assert_raise_message(
         ValueError, msg, eclf2.set_params(lr=drop, rf=drop, nb=drop).fit, X, y)
 
diff --git a/sklearn/ensemble/voting.py b/sklearn/ensemble/voting.py
index 9189600cf074f..a1eb65c5ce0bd 100644
--- a/sklearn/ensemble/voting.py
+++ b/sklearn/ensemble/voting.py
@@ -24,25 +24,20 @@
 from ..base import TransformerMixin
 from ..base import clone
 from .base import _parallel_fit_estimator
+from .base import _BaseHeterogeneousEnsemble
 from ..preprocessing import LabelEncoder
 from ..utils import Bunch
 from ..utils.validation import check_is_fitted
-from ..utils.metaestimators import _BaseComposition
 from ..utils.multiclass import check_classification_targets
 from ..utils.validation import column_or_1d
 
 
-class _BaseVoting(TransformerMixin, _BaseComposition):
+class _BaseVoting(TransformerMixin, _BaseHeterogeneousEnsemble):
     """Base class for voting.
 
     Warning: This class should not be used directly. Use derived classes
     instead.
     """
-    _required_parameters = ['estimators']
-
-    @property
-    def named_estimators(self):
-        return Bunch(**dict(self.estimators))
 
     @property
     def _weights_not_none(self):
@@ -61,10 +56,7 @@ def fit(self, X, y, sample_weight=None):
         """
         common fit operations.
         """
-        if self.estimators is None or len(self.estimators) == 0:
-            raise AttributeError('Invalid `estimators` attribute, `estimators`'
-                                 ' should be a list of (string, estimator)'
-                                 ' tuples')
+        names, clfs = self._validate_estimators()
 
         if (self.weights is not None and
                 len(self.weights) != len(self.estimators)):
@@ -72,17 +64,6 @@ def fit(self, X, y, sample_weight=None):
                              '; got %d weights, %d estimators'
                              % (len(self.weights), len(self.estimators)))
 
-        names, clfs = zip(*self.estimators)
-        self._validate_names(names)
-
-        n_isnone = np.sum(
-            [clf in (None, 'drop') for _, clf in self.estimators]
-        )
-        if n_isnone == len(self.estimators):
-            raise ValueError(
-                'All estimators are None or "drop". At least one is required!'
-            )
-
         self.estimators_ = Parallel(n_jobs=self.n_jobs)(
                 delayed(_parallel_fit_estimator)(clone(clf), X, y,
                                                  sample_weight=sample_weight)
@@ -94,46 +75,6 @@ def fit(self, X, y, sample_weight=None):
             self.named_estimators_[k[0]] = e
         return self
 
-    def set_params(self, **params):
-        """ Setting the parameters for the ensemble estimator
-
-        Valid parameter keys can be listed with get_params().
-
-        Parameters
-        ----------
-        **params : keyword arguments
-            Specific parameters using e.g. set_params(parameter_name=new_value)
-            In addition, to setting the parameters of the ensemble estimator,
-            the individual estimators of the ensemble estimator can also be
-            set or replaced by setting them to None.
-
-        Examples
-        --------
-        In this example, the RandomForestClassifier is removed.
-
-        >>> from sklearn.linear_model import LogisticRegression
-        >>> from sklearn.ensemble import RandomForestClassifier
-        >>> from sklearn.ensemble import VotingClassifier
-        >>> clf1 = LogisticRegression()
-        >>> clf2 = RandomForestClassifier()
-        >>> eclf = VotingClassifier(estimators=[('lr', clf1), ('rf', clf2)])
-        >>> eclf.set_params(rf=None)
-        VotingClassifier(estimators=[('lr', LogisticRegression()),
-                                     ('rf', None)])
-        """
-        return self._set_params('estimators', **params)
-
-    def get_params(self, deep=True):
-        """ Get the parameters of the ensemble estimator
-
-        Parameters
-        ----------
-        deep : bool
-            Setting it to True gets the various estimators and the parameters
-            of the estimators as well
-        """
-        return self._get_params('estimators', deep=deep)
-
 
 class VotingClassifier(ClassifierMixin, _BaseVoting):
     """Soft Voting/Majority Rule classifier for unfitted estimators.
@@ -230,7 +171,7 @@ class VotingClassifier(ClassifierMixin, _BaseVoting):
 
     def __init__(self, estimators, voting='hard', weights=None, n_jobs=None,
                  flatten_transform=True):
-        self.estimators = estimators
+        super().__init__(estimators=estimators)
         self.voting = voting
         self.weights = weights
         self.n_jobs = n_jobs
@@ -423,7 +364,7 @@ class VotingRegressor(RegressorMixin, _BaseVoting):
     """
 
     def __init__(self, estimators, weights=None, n_jobs=None):
-        self.estimators = estimators
+        super().__init__(estimators=estimators)
         self.weights = weights
         self.n_jobs = n_jobs
 


Code review comments:
diff_hunk: @@ -236,6 +236,11 @@ Changelog
   :user:`Matt Hancock <notmatthancock>` and
   :pr:`5963` by :user:`Pablo Duboue <DrDub>`.
 
+- |Fix| Enforce underlying estimator to be the same type than voting estimator.
path: doc/whats_new/v0.22.rst
text: @user1:
it's not clear what type is. Maybe

> Stacking and Voting estimators now ensure that their underlying estimators are either all classifiers or all regressors.

diff_hunk: @@ -178,3 +182,76 @@ def _partition_estimators(n_estimators, n_jobs):
     starts = np.cumsum(n_estimators_per_job)
 
     return n_jobs, n_estimators_per_job.tolist(), [0] + starts.tolist()
+
+
+class _BaseHeterogeneousEnsemble(MetaEstimatorMixin, _BaseComposition,
+                                 metaclass=ABCMeta):
+    """Base class for ensemble learners based on heterogeneous estimators."""
+    _required_parameters = ['estimators']
+
+    @property
+    def named_estimators(self):
+        return Bunch(**dict(self.estimators))
+
+    @abstractmethod
+    def __init__(self, estimators):
+        self.estimators = estimators
path: sklearn/ensemble/base.py
text: @user1:
Should we leave the setting of `self.estimators` for the child classes and make `_BaseHeterogeneousEnsemble` a mixin?

It mainly does input validation anyway?

No strong opinion, but I feel like setting a single attribute in the parent class makes reaading the code harder.

@author:
Since we inherit from `_BaseComposition` and `BaseEstimator`, it will be weird to create a mixin which defines the `__init__`. Basically, by inheritance, it is already a base class and not a mixin.

@user1:
i'm suggesting to not have the init method