Repository: scikit-learn-contrib/imbalanced-learn

Issue:
[BUG] The estimator_ in CondensedNearestNeighbour() is incorrect for multiple classes
#### Describe the bug
The estimator_ object fit by `CondensedNearestNeighbour()` (and probably other sampling strategies) is incorrect when y has multiple classes (and possibly also for binary classes). In particular, the estimator is only fit to a subset of 2 of the classes.  

#### Steps/Code to Reproduce

```python
from sklearn.datasets import make_blobs
from sklearn.neighbors import KNeighborsClassifier
from imblearn.under_sampling import CondensedNearestNeighbour

n_clusters = 10
X, y = make_blobs(n_samples=2000, centers=n_clusters, n_features=2, cluster_std=.5, random_state=0)

n_neighbors = 1
condenser = CondensedNearestNeighbour(sampling_strategy='all', n_neighbors=n_neighbors)
X_cond, y_cond = condenser.fit_resample(X, y)
print('condenser.estimator_.classes_', condenser.estimator_.classes_) # this should have 10 classes, which it does!
print("condenser.estomator_ accuracy", condenser.estimator_.score(X, y))
```

```
condenser.estimator_.classes_ [5 9]
condenser.estomator_ accuracy 0.2
```

```python
# I think the estimator we want should look like this
knn_cond_manual = KNeighborsClassifier(n_neighbors=n_neighbors).fit(X_cond, y_cond)
print('knn_cond_manual.classes_', knn_cond_manual.classes_)  # yes 10 classes!
print("Manual KNN on condensted data accuracy", knn_cond_manual.score(X, y)) # good accuracy!
```

```
knn_cond_manual.classes_ [0 1 2 3 4 5 6 7 8 9]
Manual KNN on condensted data accuracy 0.996
```

### The issue

The issue that we set `estimator_` in each run of the loop in `_fit_resample` e.g. [this line](https://github.com/scikit-learn-contrib/imbalanced-learn/blob/6176807c9c5d68126a79771b6c0fce329f632d2f/imblearn/under_sampling/_prototype_selection/_condensed_nearest_neighbour.py#L178). We should really set `estimator_` after the loop ends on the condensed datasets. 

This looks like it's also an issue with [OneSidedSelection](https://github.com/scikit-learn-contrib/imbalanced-learn/blob/6176807c9c5d68126a79771b6c0fce329f632d2f/imblearn/under_sampling/_prototype_selection/_one_sided_selection.py#L28) and possibly other samplers.

### Fix
I think we should just add the following to directly [before the return](https://github.com/scikit-learn-contrib/imbalanced-learn/blob/6176807c9c5d68126a79771b6c0fce329f632d2f/imblearn/under_sampling/_prototype_selection/_condensed_nearest_neighbour.py#L223) statement in `fit_resample`

```python
X_condensed, y_condensed = _safe_indexing(X, idx_under), _safe_indexing(y, idx_under)
self.estimator_.fit(X_condensed, y_condensed)
return X_condensed, y_condensed
```


#### Versions
```

System:
    python: 3.8.12 (default, Oct 12 2021, 06:23:56)  [Clang 10.0.0 ]
executable: /Users/iaincarmichael/anaconda3/envs/comp_onc/bin/python
   machine: macOS-10.16-x86_64-i386-64bit

Python dependencies:
      sklearn: 1.1.1
          pip: 21.2.4
   setuptools: 58.0.4
        numpy: 1.21.4
        scipy: 1.7.3
       Cython: 0.29.25
       pandas: 1.3.5
   matplotlib: 3.5.0
       joblib: 1.1.0
threadpoolctl: 2.2.0

Built with OpenMP: True

threadpoolctl info:
       filepath: /Users/iaincarmichael/anaconda3/envs/comp_onc/lib/python3.8/site-packages/sklearn/.dylibs/libomp.dylib
         prefix: libomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8

       filepath: /Users/iaincarmichael/anaconda3/envs/comp_onc/lib/python3.8/site-packages/numpy/.dylibs/libopenblas.0.dylib
         prefix: libopenblas
       user_api: blas
   internal_api: openblas
        version: 0.3.17
    num_threads: 4
threading_layer: pthreads
   architecture: Haswell

       filepath: /Users/iaincarmichael/anaconda3/envs/comp_onc/lib/libmkl_rt.1.dylib
         prefix: libmkl_rt
       user_api: blas
   internal_api: mkl
        version: 2021.4-Product
    num_threads: 4
threading_layer: intel

       filepath: /Users/iaincarmichael/anaconda3/envs/comp_onc/lib/libomp.dylib
         prefix: libomp
       user_api: openmp
   internal_api: openmp
        version: None
    num_threads: 8
```


Discussion:

Patch:
diff --git a/doc/whats_new/v0.12.rst b/doc/whats_new/v0.12.rst
index 8b6864e..1063ffb 100644
--- a/doc/whats_new/v0.12.rst
+++ b/doc/whats_new/v0.12.rst
@@ -3,6 +3,14 @@
 Version 0.12.0 (Under development)
 ==================================
 
-
 Changelog
 ---------
+
+Deprecations
+............
+
+- Deprecate `estimator_` argument in favor of `estimators_` for the classes
+  :class:`~imblearn.under_sampling.CondensedNearestNeighbour` and
+  :class:`~imblearn.under_sampling.OneSidedSelection`. `estimator_` will be removed
+  in 0.14.
+  :pr:`xxx` by :user:`Guillaume Lemaitre <glemaitre>`.
diff --git a/imblearn/under_sampling/_prototype_selection/_condensed_nearest_neighbour.py b/imblearn/under_sampling/_prototype_selection/_condensed_nearest_neighbour.py
index c80e40f..3b812e6 100644
--- a/imblearn/under_sampling/_prototype_selection/_condensed_nearest_neighbour.py
+++ b/imblearn/under_sampling/_prototype_selection/_condensed_nearest_neighbour.py
@@ -6,6 +6,7 @@ method."""
 # License: MIT
 
 import numbers
+import warnings
 from collections import Counter
 
 import numpy as np
@@ -59,6 +60,16 @@ class CondensedNearestNeighbour(BaseCleaningSampler):
     estimator_ : estimator object
         The validated K-nearest neighbor estimator created from `n_neighbors` parameter.
 
+        .. deprecated:: 0.12
+           `estimator_` is deprecated in 0.12 and will be removed in 0.14. Use
+           `estimators_` instead that contains the list of all K-nearest
+           neighbors estimator used for each pair of class.
+
+    estimators_ : list of estimator objects of shape (n_resampled_classes - 1,)
+        Contains the K-nearest neighbor estimator used for per of classes.
+
+        .. versionadded:: 0.12
+
     sample_indices_ : ndarray of shape (n_new_samples,)
         Indices of the samples selected.
 
@@ -87,8 +98,8 @@ class CondensedNearestNeighbour(BaseCleaningSampler):
     -----
     The method is based on [1]_.
 
-    Supports multi-class resampling. A one-vs.-rest scheme is used when
-    sampling a class as proposed in [1]_.
+    Supports multi-class resampling: a strategy one (minority) vs. each other
+    classes is applied.
 
     References
     ----------
@@ -142,22 +153,25 @@ CondensedNearestNeighbour  # doctest: +SKIP
     def _validate_estimator(self):
         """Private function to create the NN estimator"""
         if self.n_neighbors is None:
-            self.estimator_ = KNeighborsClassifier(n_neighbors=1, n_jobs=self.n_jobs)
+            estimator = KNeighborsClassifier(n_neighbors=1, n_jobs=self.n_jobs)
         elif isinstance(self.n_neighbors, numbers.Integral):
-            self.estimator_ = KNeighborsClassifier(
+            estimator = KNeighborsClassifier(
                 n_neighbors=self.n_neighbors, n_jobs=self.n_jobs
             )
         elif isinstance(self.n_neighbors, KNeighborsClassifier):
-            self.estimator_ = clone(self.n_neighbors)
+            estimator = clone(self.n_neighbors)
+
+        return estimator
 
     def _fit_resample(self, X, y):
-        self._validate_estimator()
+        estimator = self._validate_estimator()
 
         random_state = check_random_state(self.random_state)
         target_stats = Counter(y)
         class_minority = min(target_stats, key=target_stats.get)
         idx_under = np.empty((0,), dtype=int)
 
+        self.estimators_ = []
         for target_class in np.unique(y):
             if target_class in self.sampling_strategy_.keys():
                 # Randomly get one sample from the majority class
@@ -184,7 +198,7 @@ CondensedNearestNeighbour  # doctest: +SKIP
                 S_y = _safe_indexing(y, S_indices)
 
                 # fit knn on C
-                self.estimator_.fit(C_x, C_y)
+                self.estimators_.append(clone(estimator).fit(C_x, C_y))
 
                 good_classif_label = idx_maj_sample.copy()
                 # Check each sample in S if we keep it or drop it
@@ -196,7 +210,7 @@ CondensedNearestNeighbour  # doctest: +SKIP
                     # Classify on S
                     if not issparse(x_sam):
                         x_sam = x_sam.reshape(1, -1)
-                    pred_y = self.estimator_.predict(x_sam)
+                    pred_y = self.estimators_[-1].predict(x_sam)
 
                     # If the prediction do not agree with the true label
                     # append it in C_x
@@ -210,12 +224,12 @@ CondensedNearestNeighbour  # doctest: +SKIP
                         C_y = _safe_indexing(y, C_indices)
 
                         # fit a knn on C
-                        self.estimator_.fit(C_x, C_y)
+                        self.estimators_[-1].fit(C_x, C_y)
 
                         # This experimental to speed up the search
                         # Classify all the element in S and avoid to test the
                         # well classified elements
-                        pred_S_y = self.estimator_.predict(S_x)
+                        pred_S_y = self.estimators_[-1].predict(S_x)
                         good_classif_label = np.unique(
                             np.append(idx_maj_sample, np.flatnonzero(pred_S_y == S_y))
                         )
@@ -230,5 +244,15 @@ CondensedNearestNeighbour  # doctest: +SKIP
 
         return _safe_indexing(X, idx_under), _safe_indexing(y, idx_under)
 
+    @property
+    def estimator_(self):
+        """Last fitted k-NN estimator."""
+        warnings.warn(
+            "`estimator_` attribute has been deprecated in 0.12 and will be "
+            "removed in 0.14. Use `estimators_` instead.",
+            FutureWarning,
+        )
+        return self.estimators_[-1]
+
     def _more_tags(self):
         return {"sample_indices": True}
diff --git a/imblearn/under_sampling/_prototype_selection/_one_sided_selection.py b/imblearn/under_sampling/_prototype_selection/_one_sided_selection.py
index 42e9a6e..e0e5b41 100644
--- a/imblearn/under_sampling/_prototype_selection/_one_sided_selection.py
+++ b/imblearn/under_sampling/_prototype_selection/_one_sided_selection.py
@@ -5,6 +5,7 @@
 # License: MIT
 
 import numbers
+import warnings
 from collections import Counter
 
 import numpy as np
@@ -58,6 +59,16 @@ class OneSidedSelection(BaseCleaningSampler):
     estimator_ : estimator object
         Validated K-nearest neighbors estimator created from parameter `n_neighbors`.
 
+        .. deprecated:: 0.12
+           `estimator_` is deprecated in 0.12 and will be removed in 0.14. Use
+           `estimators_` instead that contains the list of all K-nearest
+           neighbors estimator used for each pair of class.
+
+    estimators_ : list of estimator objects of shape (n_resampled_classes - 1,)
+        Contains the K-nearest neighbor estimator used for per of classes.
+
+        .. versionadded:: 0.12
+
     sample_indices_ : ndarray of shape (n_new_samples,)
         Indices of the samples selected.
 
@@ -138,16 +149,18 @@ class OneSidedSelection(BaseCleaningSampler):
     def _validate_estimator(self):
         """Private function to create the NN estimator"""
         if self.n_neighbors is None:
-            self.estimator_ = KNeighborsClassifier(n_neighbors=1, n_jobs=self.n_jobs)
+            estimator = KNeighborsClassifier(n_neighbors=1, n_jobs=self.n_jobs)
         elif isinstance(self.n_neighbors, int):
-            self.estimator_ = KNeighborsClassifier(
+            estimator = KNeighborsClassifier(
                 n_neighbors=self.n_neighbors, n_jobs=self.n_jobs
             )
         elif isinstance(self.n_neighbors, KNeighborsClassifier):
-            self.estimator_ = clone(self.n_neighbors)
+            estimator = clone(self.n_neighbors)
+
+        return estimator
 
     def _fit_resample(self, X, y):
-        self._validate_estimator()
+        estimator = self._validate_estimator()
 
         random_state = check_random_state(self.random_state)
         target_stats = Counter(y)
@@ -155,6 +168,7 @@ class OneSidedSelection(BaseCleaningSampler):
 
         idx_under = np.empty((0,), dtype=int)
 
+        self.estimators_ = []
         for target_class in np.unique(y):
             if target_class in self.sampling_strategy_.keys():
                 # select a sample from the current class
@@ -177,8 +191,8 @@ class OneSidedSelection(BaseCleaningSampler):
                 idx_maj_extracted = np.delete(idx_maj, sel_idx_maj, axis=0)
                 S_x = _safe_indexing(X, idx_maj_extracted)
                 S_y = _safe_indexing(y, idx_maj_extracted)
-                self.estimator_.fit(C_x, C_y)
-                pred_S_y = self.estimator_.predict(S_x)
+                self.estimators_.append(clone(estimator).fit(C_x, C_y))
+                pred_S_y = self.estimators_[-1].predict(S_x)
 
                 S_misclassified_indices = np.flatnonzero(pred_S_y != S_y)
                 idx_tmp = idx_maj_extracted[S_misclassified_indices]
@@ -199,5 +213,15 @@ class OneSidedSelection(BaseCleaningSampler):
 
         return X_cleaned, y_cleaned
 
+    @property
+    def estimator_(self):
+        """Last fitted k-NN estimator."""
+        warnings.warn(
+            "`estimator_` attribute has been deprecated in 0.12 and will be "
+            "removed in 0.14. Use `estimators_` instead.",
+            FutureWarning,
+        )
+        return self.estimators_[-1]
+
     def _more_tags(self):
         return {"sample_indices": True}


Test patch:
diff --git a/imblearn/under_sampling/_prototype_selection/tests/test_condensed_nearest_neighbour.py b/imblearn/under_sampling/_prototype_selection/tests/test_condensed_nearest_neighbour.py
index 5b41b87..5cc8f41 100644
--- a/imblearn/under_sampling/_prototype_selection/tests/test_condensed_nearest_neighbour.py
+++ b/imblearn/under_sampling/_prototype_selection/tests/test_condensed_nearest_neighbour.py
@@ -5,6 +5,7 @@
 
 import numpy as np
 import pytest
+from sklearn.datasets import make_classification
 from sklearn.neighbors import KNeighborsClassifier
 from sklearn.utils._testing import assert_array_equal
 
@@ -95,3 +96,34 @@ def test_cnn_fit_resample_with_object(n_neighbors):
     X_resampled, y_resampled = cnn.fit_resample(X, Y)
     assert_array_equal(X_resampled, X_gt)
     assert_array_equal(y_resampled, y_gt)
+
+
+def test_condensed_nearest_neighbour_multiclass():
+    """Check the validity of the fitted attributes `estimators_`."""
+    X, y = make_classification(
+        n_samples=1_000,
+        n_classes=4,
+        weights=[0.1, 0.2, 0.2, 0.5],
+        n_clusters_per_class=1,
+        random_state=0,
+    )
+    cnn = CondensedNearestNeighbour(random_state=RND_SEED)
+    cnn.fit_resample(X, y)
+
+    assert len(cnn.estimators_) == len(cnn.sampling_strategy_)
+    other_classes = []
+    for est in cnn.estimators_:
+        assert est.classes_[0] == 0  # minority class
+        assert est.classes_[1] in {1, 2, 3}  # other classes
+        other_classes.append(est.classes_[1])
+    assert len(set(other_classes)) == len(other_classes)
+
+
+# TODO: remove in 0.14
+def test_condensed_nearest_neighbors_deprecation():
+    """Check that we raise a FutureWarning when accessing the parameter `estimator_`."""
+    cnn = CondensedNearestNeighbour(random_state=RND_SEED)
+    cnn.fit_resample(X, Y)
+    warn_msg = "`estimator_` attribute has been deprecated"
+    with pytest.warns(FutureWarning, match=warn_msg):
+        cnn.estimator_
diff --git a/imblearn/under_sampling/_prototype_selection/tests/test_one_sided_selection.py b/imblearn/under_sampling/_prototype_selection/tests/test_one_sided_selection.py
index 7d3adde..3fb5458 100644
--- a/imblearn/under_sampling/_prototype_selection/tests/test_one_sided_selection.py
+++ b/imblearn/under_sampling/_prototype_selection/tests/test_one_sided_selection.py
@@ -5,6 +5,7 @@
 
 import numpy as np
 import pytest
+from sklearn.datasets import make_classification
 from sklearn.neighbors import KNeighborsClassifier
 from sklearn.utils._testing import assert_array_equal
 
@@ -95,3 +96,34 @@ def test_oss_with_object(n_neighbors):
     X_resampled, y_resampled = oss.fit_resample(X, Y)
     assert_array_equal(X_resampled, X_gt)
     assert_array_equal(y_resampled, y_gt)
+
+
+def test_one_sided_selection_multiclass():
+    """Check the validity of the fitted attributes `estimators_`."""
+    X, y = make_classification(
+        n_samples=1_000,
+        n_classes=4,
+        weights=[0.1, 0.2, 0.2, 0.5],
+        n_clusters_per_class=1,
+        random_state=0,
+    )
+    oss = OneSidedSelection(random_state=RND_SEED)
+    oss.fit_resample(X, y)
+
+    assert len(oss.estimators_) == len(oss.sampling_strategy_)
+    other_classes = []
+    for est in oss.estimators_:
+        assert est.classes_[0] == 0  # minority class
+        assert est.classes_[1] in {1, 2, 3}  # other classes
+        other_classes.append(est.classes_[1])
+    assert len(set(other_classes)) == len(other_classes)
+
+
+# TODO: remove in 0.14
+def test_one_sided_selection_deprecation():
+    """Check that we raise a FutureWarning when accessing the parameter `estimator_`."""
+    oss = OneSidedSelection(random_state=RND_SEED)
+    oss.fit_resample(X, Y)
+    warn_msg = "`estimator_` attribute has been deprecated"
+    with pytest.warns(FutureWarning, match=warn_msg):
+        oss.estimator_