Repository: feature-engine/feature_engine

Issue:
Feature Request: Add Correlation with target as selection method for `SmartCorrelatedSelection`
**Is your feature request related to a problem? Please describe.**
No.

**Describe the solution you'd like**
I would like to add as a selection method of `SmartCorrelatedSelection` a new selection method based on the correlation of the features with the target variable.

**Describe alternatives you've considered**
I have not considered other alternatives.

**Additional context**
For linear models it can make a lot of sense to select features using correlation with target and it can be faster than computing single model performances.

I would like to contribute to this project by implementing this feature by myself.

Hints from issue comments:
@solegalli: Hi @ClaudioSalvatoreArcidiacono 

This functionality is already supported by sklearn through the `f_regression`. 

I admit that `f_regression` is not great because it selects based on p_values and f_value instead of correlation coefficient, although the correlation coefficient could be inferred from the pvalue with the number of samples, so in essence, it is more or less the same. Highlight on more or less.@ClaudioSalvatoreArcidiacono: Hey @solegalli, thanks for taking a look and thanks for mentioning `f_regression` from sklearn, it is one of those functionalities that I never fully explored to be completely honest.

I believe that adding correlation with target as a selection method to `SmartCorrelatedSelection` could still bring value to the library for the following reasons:
- `f_regression` selects features to remove by looking at the interactions with the feature and the target, without considering correlations between features. In the proposed change the candidates features to drop are selected by looking at the correlation with other features and the correlation with target is used to make the final decision. So they achieve different purposes:
  - `f_regression`: Is used to remove features with a low correlation with the target.
  - `SmartCorrelatedSelection` with `selection_method=corr_with_target` as a selection method: Removes features with lower correlation with target **among a correlated features group**.

I would say that it is more like sklearn `VarianceThreshold` vs `SmartCorrelatedSelection` with variance as a selector method, they are quite different.

- Unlike `f_regression`, which is not designed as an estimator, `SmartCorrelatedSelection` is. This makes it easier for users to incorporate it directly into their feature selection pipelines.
- A direct correlation-based selection method can be more intuitive for many users, allowing them to easily understand and interpret the relationship between features and the target variable. This could enhance the usability and accessibility of the library.

Again, thanks for your time and I am happy to discuss this further!

Solution patch:
diff --git a/docs/user_guide/outliers/Winsorizer.rst b/docs/user_guide/outliers/Winsorizer.rst
index 5fb07c9d5..741690238 100644
--- a/docs/user_guide/outliers/Winsorizer.rst
+++ b/docs/user_guide/outliers/Winsorizer.rst
@@ -127,7 +127,7 @@ The default values for fold are as follows:
 - 'gaussian': `fold` is set to 3.0;
 - 'iqr': `fold` is set to 1.5;
 - 'mad': `fold` is set to 3.29;
-- 'percentiles': `fold` is set to 0.05.
+- 'quantiles': `fold` is set to 0.05.
 
 You can manually adjust the `fold` value to make the outlier detection process more or less
 conservative, thus customizing the extent of outlier capping.
@@ -187,4 +187,4 @@ Or read our book:
 |
 
 Both our book and course are suitable for beginners and more advanced data scientists
-alike. By purchasing them you are supporting Sole, the main developer of Feature-engine.
\ No newline at end of file
+alike. By purchasing them you are supporting Sole, the main developer of Feature-engine.
diff --git a/docs/user_guide/selection/SmartCorrelatedSelection.rst b/docs/user_guide/selection/SmartCorrelatedSelection.rst
index e29c30813..682d1993b 100644
--- a/docs/user_guide/selection/SmartCorrelatedSelection.rst
+++ b/docs/user_guide/selection/SmartCorrelatedSelection.rst
@@ -19,6 +19,8 @@ such us:
 
 3. **Missing Data**: Features with less missing data are generally more reliable and informative.
 
+4. **Correlation with Target**: Features that show a stronger correlation with the target variable are often more predictive and should be prioritized.
+
 We can apply this selection strategies out of the box with the :class:`SmartCorrelatedSelection`.
 
 From a group of correlated variables, the :class:`SmartCorrelatedSelection` will retain
@@ -28,6 +30,7 @@ the variable with:
 - the highest cardinality
 - the least missing data
 - the best performing model (based on a single feature)
+- the strongest correlation with the target variable
 
 The remaining features within each correlated group will be dropped.
 
@@ -62,6 +65,10 @@ larger scales will dominate the selection procedure**, unless you have a scaled
 If the criteria is based on missing data, :class:`SmartCorrelatedSelection` will determine the
 number of NA in each feature from the correlated group and keep the one with less NA.
 
+If the criteria is based on correlation with the target, :class:`SmartCorrelatedSelection`
+will calculate the correlation coefficient between each feature and the target, and retain
+the one with the highest absolute correlation.
+
 Variance
 ~~~~~~~~
 
@@ -344,6 +351,181 @@ the dataframe:
 
     [True, True, True, True, False, True, False, True, False, False, True, True]
 
+
+Correlation with Target
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Finally, let's select the feature that shows the strongest correlation with the target
+variable from each group.
+
+Let's create another toy dataframe with 4 features and a target variable:
+
+.. code:: python
+
+    import pandas as pd
+    from sklearn.datasets import make_classification
+    from feature_engine.selection import SmartCorrelatedSelection
+
+    # make dataframe with some correlated variables
+    def make_data():
+        X, y = make_classification(n_samples=1000,
+                                   n_features=12,
+                                   n_redundant=4,
+                                   n_clusters_per_class=1,
+                                   weights=[0.50],
+                                   class_sep=2,
+                                   random_state=1)
+
+        # transform arrays into pandas df and series
+        colnames = ['var_'+str(i) for i in range(12)]
+        X = pd.DataFrame(X, columns=colnames)
+        y = pd.Series(y)
+        return X, y
+
+    X, y = make_data()
+
+Now, we set up :class:`SmartCorrelatedSelection` to find feature groups with absolute
+correlation coefficient >0.8 and retain the feature with the strongest correlation with
+the target variable:
+
+.. code:: python
+
+    # set up the selector
+    tr = SmartCorrelatedSelection(
+        variables=None,
+        method="pearson",
+        threshold=0.8,
+        missing_values="raise",
+        selection_method="corr_with_target",
+        estimator=None,
+    )
+
+With `fit()`, the transformer finds the correlated variables and selects the ones to
+keep. With `transform()`, it drops the remaining features in the correlated group from
+the dataset:
+
+.. code:: python
+
+    Xt = tr.fit_transform(X, y)
+
+Let's explore the correlated feature groups:
+
+.. code:: python
+
+    tr.correlated_feature_sets_
+
+Similarly to the previous examples, we see that the groups of correlated features are
+slightly different. Here, what happened is that the features were ordered first based 
+on their absolute correlation with the target, before carrying on the search of correlation
+with the other features in the dataset. Like this, the first feature of the group is retained, 
+which is the one with highest correlation with the target.
+
+.. code:: python
+
+    [{'var_4', 'var_6', 'var_7', 'var_9'}, {'var_0', 'var_8'}]
+
+We can find the features that will be retained as keys in the following dictionary:
+
+.. code:: python
+
+    tr.correlated_feature_dict_
+
+The variables `var_7` and `var_0` will be retained, and the remaining ones will be dropped.
+
+.. code:: python
+
+    {'var_7': {'var_4', 'var_6', 'var_9'}, 'var_0': {'var_8'}}
+
+We can check the correlation of the features with the target variable as follows:
+
+.. code:: python
+
+    print(X.corrwith(y).abs())
+
+.. code:: python
+
+    var_0     0.270913
+    var_1     0.088358
+    var_2     0.038257
+    var_3     0.027320
+    var_4     0.838361
+    var_5     0.028020
+    var_6     0.834925
+    var_7     0.916045
+    var_8     0.007724
+    var_9     0.797149
+    var_10    0.006742
+    var_11    0.023710
+    dtype: float64
+
+
+We notice that indeed the the `var_0` has a much higher correlation with the target than
+the `var_8`, which is why `var_0` was retained:
+
+.. code:: python
+
+    print(X.corrwith(y).abs().loc[['var_0', 'var_8']])
+
+.. code:: python
+
+    var_0    0.270913
+    var_8    0.007724
+    dtype: float64
+
+The variables that will be dropped are available in the following attribute:
+
+.. code:: python
+
+    tr.features_to_drop_
+
+.. code:: python
+
+    ['var_4', 'var_6', 'var_9', 'var_8']
+
+And now we can print the resulting dataframe after the transformation:
+
+.. code:: python
+
+    print(Xt.head())
+
+.. code:: python
+
+        var_0     var_1     var_2     var_3     var_5     var_7    var_10  \
+    0  1.471061 -2.376400 -0.247208  1.210290  0.091527 -2.230170  2.070526
+    1  1.819196  1.969326 -0.126894  0.034598 -0.186802 -1.447490  1.184820
+    2  1.625024  1.499174  0.334123 -2.233844 -0.313881 -2.240741 -0.066448
+    3  1.939212  0.075341  1.627132  0.943132 -0.468041 -3.534861  0.713558
+    4  1.579307  0.372213  0.338141  0.951526  0.729005 -2.053965  0.398790
+
+        var_11
+    0 -1.989335
+    1 -1.309524
+    2 -0.852703
+    3  0.484649
+    4 -0.186530
+
+We can also obtain the names of the features in the resulting dataframe as follows:
+
+.. code:: python
+
+    tr.get_feature_names_out()
+
+.. code:: python
+
+    ['var_0', 'var_1', 'var_2', 'var_3', 'var_5', 'var_7', 'var_10', 'var_11']
+
+We also find the `get_support` method that flags the features that will be retained from
+the dataframe:
+
+.. code:: python
+
+    tr.get_support()
+
+.. code:: python
+
+    [True, True, True, True, False, True, False, True, False, False, True, True]
+
+
 And that's it!
 
 Additional resources
diff --git a/feature_engine/selection/smart_correlation_selection.py b/feature_engine/selection/smart_correlation_selection.py
index 8fd9afa60..bf8629344 100644
--- a/feature_engine/selection/smart_correlation_selection.py
+++ b/feature_engine/selection/smart_correlation_selection.py
@@ -29,6 +29,7 @@
     _check_contains_inf,
     _check_contains_na,
     check_X,
+    check_y,
 )
 from feature_engine.selection.base_selector import BaseSelector
 
@@ -64,6 +65,7 @@ class SmartCorrelatedSelection(BaseSelector):
     - Feature with the highest cardinality (greatest number of unique values).
     - Feature with the highest variance.
     - Feature with the highest importance according to an estimator.
+    - Feature with the highest correlation with the target variable.
 
     SmartCorrelatedSelection() returns a dataframe containing from each group of
     correlated features, the selected variable, plus all the features that were
@@ -99,8 +101,8 @@ class SmartCorrelatedSelection(BaseSelector):
     {missing_values}
 
     selection_method: str, default= "missing_values"
-        Takes the values "missing_values", "cardinality", "variance" and
-        "model_performance".
+        Takes the values "missing_values", "cardinality", "variance",
+        "model_performance", and "corr_with_target".
 
         **"missing_values"**: keeps the feature from the correlated group with the least
         missing observations.
@@ -115,6 +117,11 @@ class SmartCorrelatedSelection(BaseSelector):
         features in a correlated group and retains the feature with the highest
         importance.
 
+        **"corr_with_target"**: keeps the feature from the correlated group that has the
+        highest (absolute) correlation with the target variable. The same correlation
+        method defined in the `method` parameter is used to calculate the correlation
+        between the features and the target.
+
     {estimator}
 
     {scoring}
@@ -229,10 +236,12 @@ def __init__(
             "cardinality",
             "variance",
             "model_performance",
+            "corr_with_target",
         ]:
             raise ValueError(
                 "selection_method takes only values 'missing_values', 'cardinality', "
-                f"'variance' or 'model_performance'. Got {selection_method} instead."
+                "'variance', 'model_performance' or 'corr_with_target'. "
+                f"Got {selection_method} instead."
             )
 
         if selection_method == "model_performance" and estimator is None:
@@ -271,7 +280,8 @@ def fit(self, X: pd.DataFrame, y: pd.Series = None):
             The training dataset.
 
         y: pandas series. Default = None
-            y is needed if selection_method == 'model_performance'.
+            y is needed if selection_method == 'model_performance' or
+                'corr_with_target'.
         """
 
         # check input dataframe
@@ -289,9 +299,11 @@ def fit(self, X: pd.DataFrame, y: pd.Series = None):
             _check_contains_na(X, self.variables_)
             _check_contains_inf(X, self.variables_)
 
-        if self.selection_method == "model_performance" and y is None:
+        if (
+            self.selection_method in ["model_performance", "corr_with_target"]
+        ) and y is None:
             raise ValueError(
-                "When `selection_method = 'model_performance'` y is needed to "
+                f"When `selection_method = '{self.selection_method}'` y is needed to "
                 "fit the transformer."
             )
 
@@ -314,6 +326,15 @@ def fit(self, X: pd.DataFrame, y: pd.Series = None):
                 .sort_values(ascending=False)
                 .index.to_list()
             )
+        elif self.selection_method == "corr_with_target":
+            y = check_y(y)
+            features = (
+                X[self.variables_]
+                .corrwith(y, method=self.method)
+                .abs()
+                .sort_values(ascending=False)
+                .index.to_list()
+            )
         else:
             features = sorted(self.variables_)
 


Test patch:
diff --git a/tests/test_selection/test_smart_correlation_selection.py b/tests/test_selection/test_smart_correlation_selection.py
index c7a3254e7..d40f7970b 100644
--- a/tests/test_selection/test_smart_correlation_selection.py
+++ b/tests/test_selection/test_smart_correlation_selection.py
@@ -117,7 +117,7 @@ def test_raises_error_when_threshold_not_permitted(_threshold):
 def test_raises_error_when_selection_method_not_permitted(_method):
     msg = (
         "selection_method takes only values 'missing_values', 'cardinality', "
-        f"'variance' or 'model_performance'. Got {_method} instead."
+        f"'variance', 'model_performance' or 'corr_with_target'. Got {_method} instead."
     )
     with pytest.raises(ValueError) as record:
         SmartCorrelatedSelection(selection_method=_method)
@@ -252,6 +252,21 @@ def test_error_if_select_model_performance_and_y_is_none(df_single):
     assert record.value.args[0] == msg
 
 
+def test_error_if_select_corr_with_target_and_y_is_none(df_single):
+    X, _ = df_single
+
+    transformer = SmartCorrelatedSelection(
+        selection_method="corr_with_target",
+    )
+    msg = (
+        "When `selection_method = 'corr_with_target'` y is needed to fit "
+        "the transformer."
+    )
+    with pytest.raises(ValueError) as record:
+        transformer.fit(X)
+    assert record.value.args[0] == msg
+
+
 def test_selection_method_variance(df_var_car):
     X = df_var_car
 
@@ -434,3 +449,27 @@ def test_smart_correlation_selection_with_groups(df_test_with_groups):
     X_tr = transformer.fit_transform(X, y)
 
     pd.testing.assert_frame_equal(X_tr_expected, X_tr)
+
+
+def test_corr_with_target_single_corr_group(df_single):
+    X, y = df_single
+
+    transformer = SmartCorrelatedSelection(
+        variables=None,
+        method="pearson",
+        threshold=0.8,
+        missing_values="raise",
+        selection_method="corr_with_target",
+    )
+
+    Xt = transformer.fit_transform(X, y)
+
+    # expected result
+    df = X[["var_0", "var_2", "var_3", "var_5"]].copy()
+
+    # test fit attrs
+    assert transformer.correlated_feature_sets_ == [{"var_1", "var_2", "var_4"}]
+    assert transformer.features_to_drop_ == ['var_4', 'var_1']
+    assert transformer.correlated_feature_dict_ == {"var_2": {"var_1", "var_4"}}
+    # test transform output
+    pd.testing.assert_frame_equal(Xt, df)