Repository: scikit-learn/scikit-learn
Pull request: ENH support unsupervised feature selection for percentile and k-best
closes #27609

Allow `SelectKBest`, `SelectPercentile`, and `GenericUnivariateSelection` (with k-best and percentile modes) to work with unsupervised scoring function.


Problem statement:
Add a version of `GenericUnivariateSelect`/`SelectPercentile`/`SelectKBest` that allows input of X with missing values and `y=None`.
### Describe the workflow you want to enable

- Select features by the percentage of missing values of X
- Select features only by statistical properties of X before y is available

### Describe your proposed solution

Create a version with weaker X, y checks.

### Describe alternatives you've considered, if relevant

1.
```python
class LooseGenericUnivariateSelect(GenericUnivariateSelect):
    def fit(self, X: Any, y: Any = None) -> Self:
        y = np.ones(X.shape[0])
        return super().fit(X, y)
    
    def _validate_data(self, X, y=None, *args, **kwargs):
        kwargs["force_all_finite"] = False
        kwargs.pop("multi_output", None)
        return super()._validate_data(X, y, *args, **kwargs)
    
    def _more_tags(self):
        return {"requires_y": False, "allow_nan": True}
```
Not sure if this code works properly.
2.
Bother to create an estimator that outputs the desired function in `feature_importance_` and use `SelectFromModel`

### Additional context

_No response_

Hints from issue discussion:
Thanks for creating the issue. Could you add a bit more context on use-cases where it would be useful to have this and maybe examples in the wild where this is being used. This will help to decide if we want to include this in scikit-learn.
> Select features by the percentage of missing values of X

For example, if one wants to use both an estimator that allows NaNs and one that does not allow NaNs in a single complex `Pipeline`,  NaNs need to be removed at some point; in certain fields where feature imputation is not prefered, something like [feature_engine.DropMissingData]( https://feature-engine.trainindata.com/en/1.3.x/api_doc/imputation/DropMissingData.html#feature_engine.imputation.DropMissingData) can be used to remove rows that contain NaNs, but if for some reason a column is included where all values are NaN in X, all rows will be removed, so one may want to preliminarily remove some columns with such a transformer within a `Pipeline`.

> Select features only by statistical properties of X before y is available

This could be used when you want to use a `Pipeline` for feature transformation, and then experiment with multiple estimators. 
When reducing features by their monotonicity or periodicity, y is unnecessary.
> Select features by the percentage of missing values of X

I am not sure that we will even advocate for dropping missing values. I am not convinced about this one.
> Select features only by statistical properties of X before y is available

Nothing prevent you to provide a function that does not use internally the `y`, ins't it?
Isn't it a matter for the function, not these estimators, to determine if y is necessary in the first place? I don't think it is a good idea to reduce the availability of estimators by making such unnecessary checks. This shouldn't be such a big change to refuse.

> > Select features only by statistical properties of X before y is available
> 
> Nothing prevent you to provide a function that does not use internally the `y`, ins't it?

This does not address the problem. Please read my statement again.

> > Select features by the percentage of missing values of X
> 
> I am not sure that we will even advocate for dropping missing values. I am not convinced about this one.

When dealing with data that has a complex arrangement of missing values, the removal of missing values can be more difficult than one might imagine.
> This does not address the problem. Please read my statement again.

A code snippet would be better. Specifically, we overlooked that we validate `y` and `None` cannot be pass there. I assume that we could make `y` to be optional at `fit` and only validate if provided.

<details>

```python
import numpy as np
from sklearn.feature_selection import GenericUnivariateSelect
X = np.random.randn(100, 10)

def func(X, y=None):
    return np.ones(X.shape[1])

transformer = GenericUnivariateSelect(func, mode='k_best')
X_new = transformer.fit(X, y=None)
```

```pytb
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[2], line 9
      6     return np.ones(X.shape[1])
      8 transformer = GenericUnivariateSelect(func, mode='k_best')
----> 9 X_new = transformer.fit(X, y=None)

File ~/Documents/packages/scikit-learn/sklearn/base.py:1215, in _fit_context.<locals>.decorator.<locals>.wrapper(estimator, *args, **kwargs)
   1208     estimator._validate_params()
   1210 with config_context(
   1211     skip_parameter_validation=(
   1212         prefer_skip_nested_validation or global_skip_validation
   1213     )
   1214 ):
-> 1215     return fit_method(estimator, *args, **kwargs)

File ~/Documents/packages/scikit-learn/sklearn/feature_selection/_univariate_selection.py:501, in _BaseFilter.fit(self, X, y)
    483 @_fit_context(prefer_skip_nested_validation=True)
    484 def fit(self, X, y):
    485     """Run score function on (X, y) and get the appropriate features.
    486 
    487     Parameters
   (...)
    499         Returns the instance itself.
    500     """
--> 501     X, y = self._validate_data(
    502         X, y, accept_sparse=["csr", "csc"], multi_output=True
    503     )
    505     self._check_params(X, y)
    506     score_func_ret = self.score_func(X, y)

File ~/Documents/packages/scikit-learn/sklearn/base.py:583, in BaseEstimator._validate_data(self, X, y, reset, validate_separately, cast_to_ndarray, **check_params)
    580 self._check_feature_names(X, reset=reset)
    582 if y is None and self._get_tags()["requires_y"]:
--> 583     raise ValueError(
    584         f"This {self.__class__.__name__} estimator "
    585         "requires y to be passed, but the target y is None."
    586     )
    588 no_val_X = isinstance(X, str) and X == "no_validation"
    589 no_val_y = y is None or isinstance(y, str) and y == "no_validation"

ValueError: This GenericUnivariateSelect estimator requires y to be passed, but the target y is None.
```

</details>
> When dealing with data that has a complex arrangement of missing values, the removal of missing values can be more difficult than one might imagine.

To be more explicit, I think that we should leverage a bit more the following work in the documentation on the topic:

- https://proceedings.neurips.cc/paper/2021/file/5fe8fdc79ce292c39c5f209d734b7206-Paper.pdf
- https://academic.oup.com/gigascience/article/doi/10.1093/gigascience/giac013/6568998

In short, if you are using a strong learner, then the imputation techniques does not matter and leveraging the missing handling of random-forest or gradient boosting would be best. More advanced imputation techniques could be useful with simpler learner with a much higher cost.

Regarding dropping the missing values does not seem wise because in real use-case, the missingness is usually MNAR (missing not at random) and potentially have a predictive power for your final task.
Something related to this feature request is the desired to drop constant feature as requested in `OneHotEncoder`: https://github.com/scikit-learn/scikit-learn/issues/27435

Making these selectors unsupervised allow to not to have to create a new estimator. I still think that it could be useful to get a `SelectThreshold` filter in this case.

I am going to implement those.

/take
I open https://github.com/scikit-learn/scikit-learn/pull/27721

Commit message: Update doc/whats_new/v1.4.rst

Patch to review:
diff --git a/doc/modules/feature_selection.rst b/doc/modules/feature_selection.rst
index 1190d17019183..7fcec524e7168 100644
--- a/doc/modules/feature_selection.rst
+++ b/doc/modules/feature_selection.rst
@@ -108,6 +108,12 @@ applied to non-negative features, such as frequencies.
     Beware not to use a regression scoring function with a classification
     problem, you will get useless results.
 
+.. note::
+
+    The :class:`SelectPercentile` and :class:`SelectKBest` support unsupervised
+    feature selection as well. One needs to provide a `score_func` where `y=None`.
+    The `score_func` should use internally `X` to compute the scores.
+
 .. topic:: Examples:
 
     * :ref:`sphx_glr_auto_examples_feature_selection_plot_feature_selection.py`
diff --git a/doc/whats_new/v1.4.rst b/doc/whats_new/v1.4.rst
index 69549e6527b69..37addc1c7e72e 100644
--- a/doc/whats_new/v1.4.rst
+++ b/doc/whats_new/v1.4.rst
@@ -304,6 +304,12 @@ Changelog
 :mod:`sklearn.feature_selection`
 ................................
 
+- |Enhancement| :class:`feature_selection.SelectKBest`,
+  :class:`feature_selection.SelectPercentile`, and
+  :class:`feature_selection.GenericUnivariateSelect` now support unsupervised
+  feature selection by providing a `score_func` taking `X` and `y=None`.
+  :pr:`27721` by :user:`Guillaume Lemaitre <glemaitre>.`
+
 - |Fix| :class:`feature_selection.RFE` and :class:`feature_selection.RFECV` do
   not check for nans during input validation.
   :pr:`21807` by `Thomas Fan`_.
diff --git a/sklearn/feature_selection/_univariate_selection.py b/sklearn/feature_selection/_univariate_selection.py
index f06889fb3b140..7a64d4c3d5943 100644
--- a/sklearn/feature_selection/_univariate_selection.py
+++ b/sklearn/feature_selection/_univariate_selection.py
@@ -481,7 +481,7 @@ def __init__(self, score_func):
         self.score_func = score_func
 
     @_fit_context(prefer_skip_nested_validation=True)
-    def fit(self, X, y):
+    def fit(self, X, y=None):
         """Run score function on (X, y) and get the appropriate features.
 
         Parameters
@@ -489,18 +489,21 @@ def fit(self, X, y):
         X : array-like of shape (n_samples, n_features)
             The training input samples.
 
-        y : array-like of shape (n_samples,)
+        y : array-like of shape (n_samples,) or None
             The target values (class labels in classification, real numbers in
-            regression).
+            regression). If the selector is unsupervised then `y` can be set to `None`.
 
         Returns
         -------
         self : object
             Returns the instance itself.
         """
-        X, y = self._validate_data(
-            X, y, accept_sparse=["csr", "csc"], multi_output=True
-        )
+        if y is None:
+            X = self._validate_data(X, accept_sparse=["csr", "csc"])
+        else:
+            X, y = self._validate_data(
+                X, y, accept_sparse=["csr", "csc"], multi_output=True
+            )
 
         self._check_params(X, y)
         score_func_ret = self.score_func(X, y)
@@ -581,6 +584,9 @@ class SelectPercentile(_BaseFilter):
     Ties between features with equal scores will be broken in an unspecified
     way.
 
+    This filter supports unsupervised feature selection that only request `X` for
+    computing the scores.
+
     Examples
     --------
     >>> from sklearn.datasets import load_digits
@@ -621,6 +627,9 @@ def _get_support_mask(self):
             mask[kept_ties] = True
         return mask
 
+    def _more_tags(self):
+        return {"requires_y": False}
+
 
 class SelectKBest(_BaseFilter):
     """Select features according to the k highest scores.
@@ -680,6 +689,9 @@ class SelectKBest(_BaseFilter):
     Ties between features with equal scores will be broken in an unspecified
     way.
 
+    This filter supports unsupervised feature selection that only request `X` for
+    computing the scores.
+
     Examples
     --------
     >>> from sklearn.datasets import load_digits
@@ -724,6 +736,9 @@ def _get_support_mask(self):
             mask[np.argsort(scores, kind="mergesort")[-self.k :]] = 1
             return mask
 
+    def _more_tags(self):
+        return {"requires_y": False}
+
 
 class SelectFpr(_BaseFilter):
     """Filter: Select the pvalues below alpha based on a FPR test.
@@ -991,7 +1006,8 @@ class GenericUnivariateSelect(_BaseFilter):
         a single array scores.
 
     mode : {'percentile', 'k_best', 'fpr', 'fdr', 'fwe'}, default='percentile'
-        Feature selection mode.
+        Feature selection mode. Note that the `'percentile'` and `'kbest'`
+        modes are supporting unsupervised feature selection (when `y` is `None`).
 
     param : "all", float or int, default=1e-5
         Parameter of the corresponding mode.
diff --git a/sklearn/feature_selection/tests/test_feature_select.py b/sklearn/feature_selection/tests/test_feature_select.py
index 8ad70d3c6f554..1c5e5bb4f0874 100644
--- a/sklearn/feature_selection/tests/test_feature_select.py
+++ b/sklearn/feature_selection/tests/test_feature_select.py
@@ -987,3 +987,30 @@ def selector(X, y):
     )
     for name, dtype in output.dtypes.items():
         assert dtype == X.dtypes[name]
+
+
+@pytest.mark.parametrize(
+    "selector",
+    [
+        SelectKBest(k=4),
+        SelectPercentile(percentile=80),
+        GenericUnivariateSelect(mode="k_best", param=4),
+        GenericUnivariateSelect(mode="percentile", param=80),
+    ],
+)
+def test_unsupervised_filter(selector):
+    """Check support for unsupervised feature selection for the filter that could
+    require only `X`.
+    """
+    rng = np.random.RandomState(0)
+    X = rng.randn(10, 5)
+
+    def score_func(X, y=None):
+        return np.array([1, 1, 1, 1, 0])
+
+    selector.set_params(score_func=score_func)
+    selector.fit(X)
+    X_trans = selector.transform(X)
+    assert_allclose(X_trans, X[:, :4])
+    X_trans = selector.fit_transform(X)
+    assert_allclose(X_trans, X[:, :4])


Code review comments:
diff_hunk: @@ -680,6 +689,9 @@ class SelectKBest(_BaseFilter):
     Ties between features with equal scores will be broken in an unspecified
     way.
 
+    This filter supports unsupervised feature selection that only request `X` for
path: sklearn/feature_selection/_univariate_selection.py
text: @user1:
```suggestion
    This filter supports unsupervised feature selection that only requests `X` for
```

diff_hunk: @@ -581,6 +584,9 @@ class SelectPercentile(_BaseFilter):
     Ties between features with equal scores will be broken in an unspecified
     way.
 
+    This filter supports unsupervised feature selection that only request `X` for
path: sklearn/feature_selection/_univariate_selection.py
text: @user1:
```suggestion
    This filter supports unsupervised feature selection that only requests `X` for
```