Repository: scikit-learn/scikit-learn
Pull request: ENH Add verbose option to VotingClassifier and VotingRegre…
#### Reference Issues/PRs
Fixes #10360
If merged closes #14772, #15991, #10365, #10974 

#### What does this implement/fix? Explain your changes.
Add a boolean verbose option to `VotingClassifier` and `VotingRegressor`.
Follows the example of #11364.

Problem statement:
Verbose VotingClassifier
Sometimes the fitting procedure lasts a long time when using many voters. Therefore it would be nice to get some feedback. 

I think it could be done by just passing the "verbose" keyword to the Parallel() call in line 186 of the current stable voting_classifier.py and adding it to `VotingClassifier.__init__`.

Hints from issue discussion:
Sounds fair enough
@mohdsanadzakirizvi, @shivani-1521, there are now two PRs open for this, by each of you: #10367, #10365. Neither of you have included a test which ensures that no verbosity is used by default / with verbose=0. Both of your implementations would fail such a test. The test should appear in `sklearn/ensemble/tests/test_voting_classifier.py` and you can see `git grep test_.*verbose` for examples of similar tests.
Also, if one of you would like to close in deference to the other, go ahead.
A few other friendly points:
* it would have been clearer if you had stated you were working on the issue before opening PRs
* it would be clearer if you gave your PRs better titles like "Add verbosity option to VotingClassifier"
* @mohdsanadzakirizvi, you should have mentioned `Fixes #10360` or similar in your PR description.
* Please do read the contributors' guidelines for a project when you begin to contribute.
@jnothman thanks for the detailed feedback! I will be closing my PR. Also, could you please suggest other beginner-friendly bugs?
@jnothman thanks for your help and patience, I'm a new contributor. I'm attempting to fix this.
@jnothman , if no one is working, I'd like to take up this issue?
Feel free to take it up and just use Parallel's verbose option, but I also
have a PR open (#10425) which creates verbose infrastructure without
Parallel, so I'm not sure if this wants to wait, or you want to submit a PR
to my branch...

@jnothman meant #10435 I think.
If everyone is ok with it, I would like to take the issue. 

I have created the PR #10974. It includes: 
- a new `verbose` option for `VotingClassifier` that is passed to `Parallel`
- an update of the `VotingClassifier` docstring to add the `verbose` option
- a unit test that checks stdout and stderr with 3 different values of the verbose option: the default one (ie. 0), a value between 1 and 50, and a value above 50. Hopefully using the `capfd` fixture is correct.

Commit message: nitpicks

Patch to review:
diff --git a/doc/whats_new/v0.23.rst b/doc/whats_new/v0.23.rst
index 53c416c506614..b23a1fab704bf 100644
--- a/doc/whats_new/v0.23.rst
+++ b/doc/whats_new/v0.23.rst
@@ -67,6 +67,16 @@ Changelog
   `ValueError` for arguments `n_classes < 1` OR `length < 1`.
   :pr:`16006` by :user:`Rushabh Vasani <rushabh-v>`.
 
+:mod:`sklearn.ensemble`
+.................................
+
+- |API| Added boolean `verbose` flag to classes:
+  :class:`ensemble.VotingClassifier` and :class:`ensemble.VotingRegressor`.
+  :pr:`15991` by :user:`Sam Bail <spbail>`,
+  :user:`Hanna Bruce MacDonald <hannahbrucemacdonald>`,
+  :user:`Reshama Shaikh <reshamas>`, and
+  :user:`Chiara Marmo <cmarmo>`.
+
 :mod:`sklearn.feature_extraction`
 .................................
 
diff --git a/sklearn/ensemble/_base.py b/sklearn/ensemble/_base.py
index a7f018b94c54d..30a690c974d3a 100644
--- a/sklearn/ensemble/_base.py
+++ b/sklearn/ensemble/_base.py
@@ -15,12 +15,13 @@
 from ..base import is_classifier, is_regressor
 from ..base import BaseEstimator
 from ..base import MetaEstimatorMixin
-from ..utils import Bunch
+from ..utils import Bunch, _print_elapsed_time
 from ..utils import check_random_state
 from ..utils.metaestimators import _BaseComposition
 
 
-def _parallel_fit_estimator(estimator, X, y, sample_weight=None):
+def _parallel_fit_estimator(estimator, X, y, sample_weight=None,
+                            message_clsname=None, message=None):
     """Private function used to fit an estimator within a job."""
     if sample_weight is not None:
         try:
@@ -33,7 +34,8 @@ def _parallel_fit_estimator(estimator, X, y, sample_weight=None):
                 ) from exc
             raise
     else:
-        estimator.fit(X, y)
+        with _print_elapsed_time(message_clsname, message):
+            estimator.fit(X, y)
     return estimator
 
 
diff --git a/sklearn/ensemble/_voting.py b/sklearn/ensemble/_voting.py
index 838d2440a9e4d..dd41de5dddf14 100644
--- a/sklearn/ensemble/_voting.py
+++ b/sklearn/ensemble/_voting.py
@@ -39,6 +39,11 @@ class _BaseVoting(TransformerMixin, _BaseHeterogeneousEnsemble):
     instead.
     """
 
+    def _log_message(self, name, idx, total):
+        if not self.verbose:
+            return None
+        return '(%d of %d) Processing %s' % (idx, total, name)
+
     @property
     def _weights_not_none(self):
         """Get the weights of not `None` estimators."""
@@ -63,9 +68,13 @@ def fit(self, X, y, sample_weight=None):
                              % (len(self.weights), len(self.estimators)))
 
         self.estimators_ = Parallel(n_jobs=self.n_jobs)(
-                delayed(_parallel_fit_estimator)(clone(clf), X, y,
-                                                 sample_weight=sample_weight)
-                for clf in clfs if clf not in (None, 'drop')
+                delayed(_parallel_fit_estimator)(
+                        clone(clf), X, y,
+                        sample_weight=sample_weight,
+                        message_clsname='Voting',
+                        message=self._log_message(names[idx], idx, len(clfs))
+                )
+                for idx, clf in enumerate(clfs) if clf not in (None, 'drop')
             )
 
         self.named_estimators_ = Bunch()
@@ -122,6 +131,10 @@ class VotingClassifier(ClassifierMixin, _BaseVoting):
         flatten_transform=False, it returns
         (n_classifiers, n_samples, n_classes).
 
+    verbose : bool, default=False
+        If True, the time elapsed while fitting will be printed as it
+        is completed.
+
     Attributes
     ----------
     estimators_ : list of classifiers
@@ -176,13 +189,14 @@ class VotingClassifier(ClassifierMixin, _BaseVoting):
     (6, 6)
     """
 
-    def __init__(self, estimators, voting='hard', weights=None, n_jobs=None,
-                 flatten_transform=True):
+    def __init__(self, estimators, voting='hard', weights=None,
+                 n_jobs=None, flatten_transform=True, verbose=False):
         super().__init__(estimators=estimators)
         self.voting = voting
         self.weights = weights
         self.n_jobs = n_jobs
         self.flatten_transform = flatten_transform
+        self.verbose = verbose
 
     def fit(self, X, y, sample_weight=None):
         """Fit the estimators.
@@ -346,6 +360,10 @@ class VotingRegressor(RegressorMixin, _BaseVoting):
         ``-1`` means using all processors. See :term:`Glossary <n_jobs>`
         for more details.
 
+    verbose : bool, default=False
+        If True, the time elapsed while fitting will be printed as it
+        is completed.
+
     Attributes
     ----------
     estimators_ : list of regressors
@@ -376,10 +394,11 @@ class VotingRegressor(RegressorMixin, _BaseVoting):
     [ 3.3  5.7 11.8 19.7 28.  40.3]
     """
 
-    def __init__(self, estimators, weights=None, n_jobs=None):
+    def __init__(self, estimators, weights=None, n_jobs=None, verbose=False):
         super().__init__(estimators=estimators)
         self.weights = weights
         self.n_jobs = n_jobs
+        self.verbose = verbose
 
     def fit(self, X, y, sample_weight=None):
         """Fit the estimators.
diff --git a/sklearn/ensemble/tests/test_voting.py b/sklearn/ensemble/tests/test_voting.py
index 8619296536964..1fdbd0bf42ad7 100644
--- a/sklearn/ensemble/tests/test_voting.py
+++ b/sklearn/ensemble/tests/test_voting.py
@@ -1,6 +1,7 @@
 """Testing for the VotingClassifier and VotingRegressor"""
 
 import pytest
+import re
 import numpy as np
 
 from sklearn.utils._testing import assert_almost_equal, assert_array_equal
@@ -513,6 +514,29 @@ def test_check_estimators_voting_estimator(estimator):
     check_no_attributes_set_in_init(estimator.__class__.__name__, estimator)
 
 
+@pytest.mark.parametrize(
+    "estimator",
+    [VotingRegressor(
+        estimators=[('lr', LinearRegression()),
+                    ('rf', RandomForestRegressor(random_state=123))],
+        verbose=True),
+     VotingClassifier(
+         estimators=[('lr', LogisticRegression(random_state=123)),
+                     ('rf', RandomForestClassifier(random_state=123))],
+        verbose=True)]
+)
+def test_voting_verbose(estimator, capsys):
+
+    X = np.array([[-1.1, -1.5], [-1.2, -1.4], [-3.4, -2.2], [1.1, 1.2]])
+    y = np.array([1, 1, 2, 2])
+
+    pattern = (r'\[Voting\].*\(0 of 2\) Processing lr, total=.*\n'
+               r'\[Voting\].*\(1 of 2\) Processing rf, total=.*\n$')
+
+    estimator.fit(X, y)
+    assert re.match(pattern, capsys.readouterr()[0])
+
+
 # TODO: Remove in 0.24 when None is removed in Voting*
 @pytest.mark.parametrize(
     "Voter, BaseEstimator",


Code review comments:
diff_hunk: @@ -513,6 +514,29 @@ def test_check_estimators_voting_estimator(estimator):
     check_no_attributes_set_in_init(estimator.__class__.__name__, estimator)
 
 
+@pytest.mark.parametrize(
+    "estimator",
+    [VotingRegressor(
+        estimators=[('lr', LinearRegression()),
+                    ('rf', RandomForestRegressor(random_state=123))],
+        verbose=True),
+     VotingClassifier(
+         estimators=[('lr', LogisticRegression(random_state=123)),
+                     ('rf', RandomForestClassifier(random_state=123))],
+        verbose=True)]
+)
+def test_voting_verbose(estimator, capsys):
+
+    X = np.array([[-1.1, -1.5], [-1.2, -1.4], [-3.4, -2.2], [1.1, 1.2]])
+    y = np.array([1, 1, 2, 2])
+
+    pattern = (r'\[Voting\].*\(0 of 2\) Processing lr, total=.*\n'
+               r'\[Voting\].*\(1 of 2\) Processing rf, total=.*\n$')
path: sklearn/ensemble/tests/test_voting.py
text: @user1:
```suggestion
               r'\[Voting\].*\(2 of 2\) Processing rf, total=.*\n$')
```

diff_hunk: @@ -63,9 +68,13 @@ def fit(self, X, y, sample_weight=None):
                              % (len(self.weights), len(self.estimators)))
 
         self.estimators_ = Parallel(n_jobs=self.n_jobs)(
-                delayed(_parallel_fit_estimator)(clone(clf), X, y,
-                                                 sample_weight=sample_weight)
-                for clf in clfs if clf not in (None, 'drop')
+                delayed(_parallel_fit_estimator)(
+                        clone(clf), X, y,
+                        sample_weight=sample_weight,
+                        message_clsname='Voting',
+                        message=self._log_message(names[idx], idx, len(clfs))
path: sklearn/ensemble/_voting.py
text: @user1:
So the log messages start from 1:

```suggestion
                        message=self._log_message(names[idx], idx + 1, len(clfs))
```

@user2:
Thanks, good catch :)

diff_hunk: @@ -513,6 +514,29 @@ def test_check_estimators_voting_estimator(estimator):
     check_no_attributes_set_in_init(estimator.__class__.__name__, estimator)
 
 
+@pytest.mark.parametrize(
+    "estimator",
+    [VotingRegressor(
+        estimators=[('lr', LinearRegression()),
+                    ('rf', RandomForestRegressor(random_state=123))],
+        verbose=True),
+     VotingClassifier(
+         estimators=[('lr', LogisticRegression(random_state=123)),
+                     ('rf', RandomForestClassifier(random_state=123))],
+        verbose=True)]
+)
+def test_voting_verbose(estimator, capsys):
+
+    X = np.array([[-1.1, -1.5], [-1.2, -1.4], [-3.4, -2.2], [1.1, 1.2]])
+    y = np.array([1, 1, 2, 2])
+
+    pattern = (r'\[Voting\].*\(0 of 2\) Processing lr, total=.*\n'
path: sklearn/ensemble/tests/test_voting.py
text: @user1:
```suggestion
    pattern = (r'\[Voting\].*\(1 of 2\) Processing lr, total=.*\n'
```