Repository: scikit-learn/scikit-learn
Pull request: FIX accept meta-estimator in SelfTrainingClassifier
closes #19119 

`SelfTrainingClassifier` did not accept nested estimators that did not expose `predict_proba`.
One fix is to validate a fitted estimator where we know if `predict_proba` will be then available.



Problem statement:
StackClassifier is SelfTrainingClassifier throws error 
<!--
Before submitting a bug, please make sure the issue hasn't been already
addressed by searching through the past issues.
-->

#### Describe the bug
When using StackClassifier as the "estimator" for SelfTrainingClassifier, an error is thrown related to "predict_proba" even though it is there within StackClassifier
Error:
`ValueError: base_estimator (StackingClassifier) should implement predict_proba!`

In occurs here:
`/usr/local/lib/python3.6/dist-packages/sklearn/semi_supervised/_self_training.py in _validate_estimator(estimator)`


#### Steps/Code to Reproduce
Use StackClassifier as an estimator within for SelfTrainingClassifier and see the error when you attempt to fit the SelfTrainingClassifier.

Example:


```python
from sklearn.datasets import load_iris
from sklearn.ensemble import StackingClassifier
from sklearn.semi_supervised import SelfTrainingClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.naive_bayes import GaussianNB

X, y = load_iris(return_X_y=True)
y[:10] = -1

estimators = [
    ('sgd', LogisticRegression()),
    ('gnb', GaussianNB()),
]

final = LogisticRegression()

stacked_classifiers = StackingClassifier(
    estimators=estimators, final_estimator=final, verbose=1
)

clf_self_training = SelfTrainingClassifier(stacked_classifiers, max_iter=1000)
clf_self_training.fit(X, y)
```

#### Expected Results
No error is thrown

#### Actual Results
The following error is thrown:
Error:
`ValueError: base_estimator (StackingClassifier) should implement predict_proba!`

#### Versions
System:
    python: 3.6.9 (default, Oct  8 2020, 12:12:24)  [GCC 8.4.0]
executable: /usr/bin/python3
   machine: Linux-4.19.112+-x86_64-with-Ubuntu-18.04-bionic

Python dependencies:
          pip: 19.3.1
   setuptools: 51.0.0
      sklearn: 0.24.0
        numpy: 1.19.4
        scipy: 1.4.1
       Cython: 0.29.21
       pandas: 1.1.5
   matplotlib: 3.2.2
       joblib: 1.0.0
threadpoolctl: 2.1.0

Built with OpenMP: True


<!-- Thanks for contributing! -->

Hints from issue discussion:
~~ @riyadhctg Could you modify your minimal example such that it can run.~~

I did update the original example with a case that should not indeed fail.
I think that there might be an underlying bug but the error that you obtain is normal. `SGDClassifier()` that is your final estimator does not have `predict_proba` defined when the loss is `'hinge'` (the default):

```python
stacked_classifiers.fit(X, y)
stacked_classifiers.predict_proba(X)
```

```pytb

stacked_classifiers.predict_proba(X)
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/tmp/.font-unix/tmp.py in <module>
----> 1 stacked_classifiers.predict_proba(X)

~/Documents/packages/scikit-learn/sklearn/utils/metaestimators.py in __get__(self, obj, type)
    112                     continue
    113                 else:
--> 114                     getattr(delegate, self.attribute_name)
    115                     break
    116             else:

~/Documents/packages/scikit-learn/sklearn/linear_model/_stochastic_gradient.py in predict_proba(self)
   1027         http://jmlr.csail.mit.edu/papers/volume2/zhang02c/zhang02c.pdf
   1028         """
-> 1029         self._check_proba()
   1030         return self._predict_proba
   1031 

~/Documents/packages/scikit-learn/sklearn/linear_model/_stochastic_gradient.py in _check_proba(self)
    987     def _check_proba(self):
    988         if self.loss not in ("log", "modified_huber"):
--> 989             raise AttributeError("probability estimates are not available for"
    990                                  " loss=%r" % self.loss)
    991 

AttributeError: probability estimates are not available for loss='hinge'
```

However, the `SelfTrainingClassifier` does not raise an error for me but does not have a final estimator fitted. So I will check why this is the case,
Arff, I forgot to set some labels to `-1`
OK so the issue here is that we don't know if `predict_proba` exist for the `StackingClassifier` until it has been fitted since `final_estimator_` does not exist yet.

I should check more deeply, but the fix could be to validate `base_estimator_` instead of `base_estimator` because it would be a fitted estimator:

https://github.com/scikit-learn/scikit-learn/blob/6902b1014745cbf3b1a7a85eac60b7e8e882d715/sklearn/semi_supervised/_self_training.py#L204-L211
take

Commit message: update whats new

Patch to review:
diff --git a/doc/whats_new/v0.24.rst b/doc/whats_new/v0.24.rst
index 7197b74b94faa..3e28934eee61b 100644
--- a/doc/whats_new/v0.24.rst
+++ b/doc/whats_new/v0.24.rst
@@ -2,6 +2,23 @@
 
 .. currentmodule:: sklearn
 
+.. _changes_0_24_1:
+
+Version 0.24.1
+==============
+
+Changelog
+---------
+
+:mod:`sklearn.semi_supervised`
+..............................
+
+- |Fix| :class:`semi_supervised.SelfTrainingClassifier` is now accepting
+  meta-estimator (e.g. :class:`ensemble.StackingClassifier`). The validation
+  of this estimator is done a fitted estimator, once we know the existence of
+  the method `predict_proba`.
+  :pr:`19126` by :user:`Guillaume Lemaitre <glemaitre>`.
+
 .. _changes_0_24:
 
 Version 0.24.0
diff --git a/sklearn/semi_supervised/_self_training.py b/sklearn/semi_supervised/_self_training.py
index c3ed0baeaae68..efc259ae34d8d 100644
--- a/sklearn/semi_supervised/_self_training.py
+++ b/sklearn/semi_supervised/_self_training.py
@@ -207,8 +207,11 @@ def fit(self, X, y):
 
             if self.n_iter_ == 1:
                 # Only validate in the first iteration so that n_iter=0 is
-                # equivalent to the base_estimator itself.
-                _validate_estimator(self.base_estimator)
+                # equivalent to the base_estimator_ itself.
+                # Validate the fitted estimator since `predict_proba` can be
+                # delegated to an underlying "final" fitted estimator as
+                # generally done in meta-estimator or pipeline.
+                _validate_estimator(self.base_estimator_)
 
             # Predict on the unlabeled samples
             prob = self.base_estimator_.predict_proba(
diff --git a/sklearn/semi_supervised/tests/test_self_training.py b/sklearn/semi_supervised/tests/test_self_training.py
index b5c44996d5e52..b6a617157c962 100644
--- a/sklearn/semi_supervised/tests/test_self_training.py
+++ b/sklearn/semi_supervised/tests/test_self_training.py
@@ -4,14 +4,16 @@
 from numpy.testing import assert_array_equal
 import pytest
 
+from sklearn.ensemble import StackingClassifier
 from sklearn.exceptions import NotFittedError
-from sklearn.semi_supervised import SelfTrainingClassifier
 from sklearn.neighbors import KNeighborsClassifier
 from sklearn.svm import SVC
 from sklearn.model_selection import train_test_split
 from sklearn.datasets import load_iris, make_blobs
 from sklearn.metrics import accuracy_score
 
+from sklearn.semi_supervised import SelfTrainingClassifier
+
 # Author: Oliver Rausch <rauscho@ethz.ch>
 # License: BSD 3 clause
 
@@ -318,3 +320,26 @@ def test_k_best_selects_best():
 
     for row in most_confident_svc.tolist():
         assert row in added_by_st
+
+
+def test_base_estimator_meta_estimator():
+    # Check that a meta-estimator relying on an estimator implementing
+    # `predict_proba` will work even if it does expose this method before being
+    # fitted.
+    # Non-regression test for:
+    # https://github.com/scikit-learn/scikit-learn/issues/19119
+
+    base_estimator = StackingClassifier(
+        estimators=[
+            ("svc_1", SVC(probability=True)), ("svc_2", SVC(probability=True)),
+        ],
+        final_estimator=SVC(probability=True)
+    )
+
+    # make sure that the `base_estimator` does not expose `predict_proba`
+    # without being fitted
+    assert not hasattr(base_estimator, "predict_proba")
+
+    clf = SelfTrainingClassifier(base_estimator=base_estimator)
+    clf.fit(X_train, y_train_missing_labels)
+    clf.predict_proba(X_test)


Code review comments:
diff_hunk: @@ -207,8 +207,11 @@ def fit(self, X, y):
 
             if self.n_iter_ == 1:
                 # Only validate in the first iteration so that n_iter=0 is
-                # equivalent to the base_estimator itself.
-                _validate_estimator(self.base_estimator)
+                # equivalent to the base_estimator_ itself.
path: sklearn/semi_supervised/_self_training.py
text: @author:
@user1 Do you recall what is the meaning of this line and the previous one?
It was not obvious to me.

@user2:
May I suggest:

```
# Only validate the fitted estimator of the first iteration.
```

Actually since the validation is really cheap, I would not mind simplifying this code and removing the `if self.n_iter_ == 1:` condition.

diff_hunk: @@ -318,3 +320,26 @@ def test_k_best_selects_best():
 
     for row in most_confident_svc.tolist():
         assert row in added_by_st
+
+
+def test_base_estimator_meta_estimator():
+    # Check that a meta-estimator relying on an estimator implementing
+    # `predict_proba` will work even if it does expose this method before being
+    # fitted.
+    # Non-regression test for:
+    # https://github.com/scikit-learn/scikit-learn/issues/19119
+
+    base_estimator = StackingClassifier(
+        estimators=[
+            ("svc_1", SVC(probability=True)), ("svc_2", SVC(probability=True)),
+        ],
+        final_estimator=SVC(probability=True)
path: sklearn/semi_supervised/tests/test_self_training.py
text: @user1:
To make the rest run faster:

```suggestion
        final_estimator=SVC(probability=True), cv=2
```