Repository: alteryx/woodwork
Pull request #1531 (2022-09-21 14:45:49)

Allow calculation of correlation metrics with a target column
Add `target_col` argument to `dependence/dependence_dict` methods to calculate correlation metrics between features and the specified `target_col`

codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1531?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) Report
> Merging [#1531](https://codecov.io/gh/alteryx/woodwork/pull/1531?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (301fc53) into [main](https://codecov.io/gh/alteryx/woodwork/commit/0844dee3604adfffc8e3d5bab49b52df9b720323?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (0844dee) will **decrease** coverage by `80.66%`.
> The diff coverage is `14.41%`.

```diff
@@             Coverage Diff             @@
##             main    #1531       +/-   ##
===========================================
- Coverage   98.71%   18.05%   -80.67%     
===========================================
  Files          95       96        +1     
  Lines       10969    11061       +92     
===========================================
- Hits        10828     1997     -8831     
- Misses        141     9064     +8923     
```


| [Impacted Files](https://codecov.io/gh/alteryx/woodwork/pull/1531?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | Coverage Δ | |
|---|---|---|
| [...istics\_utils/\_calculate\_max\_dependence\_for\_pair.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc3RhdGlzdGljc191dGlscy9fY2FsY3VsYXRlX21heF9kZXBlbmRlbmNlX2Zvcl9wYWlyLnB5) | `25.00% <0.00%> (-75.00%)` | :arrow_down: |
| [woodwork/tests/accessor/test\_statistics.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF9zdGF0aXN0aWNzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/schema/test\_column\_schema.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvc2NoZW1hL3Rlc3RfY29sdW1uX3NjaGVtYS5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF91dGlscy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/statistics\_utils/\_parse\_measures.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc3RhdGlzdGljc191dGlscy9fcGFyc2VfbWVhc3VyZXMucHk=) | `9.09% <10.00%> (-90.91%)` | :arrow_down: |
| [woodwork/statistics\_utils/\_get\_dependence\_dict.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc3RhdGlzdGljc191dGlscy9fZ2V0X2RlcGVuZGVuY2VfZGljdC5weQ==) | `12.19% <11.76%> (-87.81%)` | :arrow_down: |
| [.../statistics\_utils/\_calculate\_dependence\_measure.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc3RhdGlzdGljc191dGlscy9fY2FsY3VsYXRlX2RlcGVuZGVuY2VfbWVhc3VyZS5weQ==) | `19.44% <16.66%> (-80.56%)` | :arrow_down: |
| [...rk/statistics\_utils/\_convert\_ordinal\_to\_numeric.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc3RhdGlzdGljc191dGlscy9fY29udmVydF9vcmRpbmFsX3RvX251bWVyaWMucHk=) | `16.66% <16.66%> (ø)` | |
| [woodwork/utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdXRpbHMucHk=) | `32.01% <30.00%> (-67.99%)` | :arrow_down: |
| [woodwork/table\_accessor.py](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGFibGVfYWNjZXNzb3IucHk=) | `50.63% <62.50%> (-48.93%)` | :arrow_down: |
| ... and [67 more](https://codecov.io/gh/alteryx/woodwork/pull/1531/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)

bchen1116: > This looks great nice work! Do we have any future plans on adding another parameter that takes `column_names` to only calculate the dependence dict between specified columns?

@ParthivNaresh currently no plan to do so. If there is a request for a feature like this, it can easily be done, but functionality-wise, it would be the same as taking a subset of the dataset and running `dependence/dependence_dict` on that set, ie
```python
df_subset = df[[column_names]]
df_subset.ww.dependence_dict()
```

Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index f6fd37f..79dd885 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -8,6 +8,7 @@ Future Release
     * Enhancements
         * Added ``Spearman Correlation`` to options for dependence calculations (:pr:`1523`)
         * Added ``ignore_zeros`` as an argument for ``box_plot_dict`` to allow for calculations of outliers without 0 values (:pr:`1524`)
+        * Added ``target_col`` argument to ``dependence`` and ``dependence_dict`` to calculate correlations between features and target_col (:pr:`1531`)
     * Fixes
         * Fix datetime pivot point to be set at current year + 10 rather than the default for two-digit years when ``datetime_format`` provided (:pr:`1512`)
     * Changes
diff --git a/woodwork/config.py b/woodwork/config.py
index 182ab2c..842893d 100644
--- a/woodwork/config.py
+++ b/woodwork/config.py
@@ -50,6 +50,7 @@ CONFIG_DEFAULTS = {
     ],
     "frequence_inference_window_length": 15,
     "frequence_inference_threshold": 0.9,
+    "correlation_metrics": ["mutual_info", "pearson", "spearman", "max", "all"],
 }
 
 
diff --git a/woodwork/statistics_utils/_calculate_dependence_measure.py b/woodwork/statistics_utils/_calculate_dependence_measure.py
index 3b2aa1c..2e2d2c7 100644
--- a/woodwork/statistics_utils/_calculate_dependence_measure.py
+++ b/woodwork/statistics_utils/_calculate_dependence_measure.py
@@ -16,6 +16,7 @@ def _calculate_dependence_measure(
     notna_mask,
     min_shared,
     col_names,
+    target_col,
 ):
     """
     Calculates the specified dependence measure for each pair of columns in the
@@ -32,11 +33,15 @@ def _calculate_dependence_measure(
             a cell is null or not.
         min_shared (int): Mininum rows of shared data to calculate dependence.
         col_names (list): List of columns to use for this calculation.
+        target_col (str): The string name of the target column.
 
     Returns:
         None
     """
-    column_pairs = itertools.combinations(col_names, 2)
+    if target_col is None:
+        column_pairs = itertools.combinations(col_names, 2)
+    else:
+        column_pairs = [(x, target_col) for x in col_names if x != target_col]
     for a_col, b_col in column_pairs:
         result = results[(a_col, b_col)]
         # check if result already has keys, meaning function has been called
diff --git a/woodwork/statistics_utils/_get_dependence_dict.py b/woodwork/statistics_utils/_get_dependence_dict.py
index 2037d9a..34c9220 100644
--- a/woodwork/statistics_utils/_get_dependence_dict.py
+++ b/woodwork/statistics_utils/_get_dependence_dict.py
@@ -40,6 +40,7 @@ def _get_dependence_dict(
     min_shared=25,
     random_seed=0,
     max_nunique=6000,
+    target_col=None,
 ):
     """Calculates dependence measures between all pairs of columns in the DataFrame that
     support measuring dependence. Supports boolean, categorical, datetime, and numeric data.
@@ -88,6 +89,8 @@ def _get_dependence_dict(
         max_nunique (int): The total maximum number of unique values for all large categorical columns (> 800 unique values).
             Categorical columns will be dropped until this number is met or until there is only one large categorical column.
             Defaults to 6000.
+        target_col (str): The column name of the target. If provided, will only calculate the dependence dictionary between other columns and this target column.
+            The target column will be `column_2` in the returned result. Defaults to None.
 
     Returns:
         list(dict): A list containing dictionaries that have keys `column_1`,
@@ -98,6 +101,8 @@ def _get_dependence_dict(
         still no dependence.
     """
     start_time = timer()
+    if target_col is not None and target_col not in list(dataframe.columns):
+        raise ValueError("target_col '{}' not in the dataframe".format(target_col))
 
     returned_measures, calc_order, calc_max = _parse_measures(measures)
 
@@ -203,6 +208,7 @@ def _get_dependence_dict(
             notna_mask=notna_mask,
             min_shared=min_shared,
             col_names=col_names,
+            target_col=target_col,
         )
 
     for result in results.values():
diff --git a/woodwork/statistics_utils/_parse_measures.py b/woodwork/statistics_utils/_parse_measures.py
index 1eeb1fc..5a7e20e 100644
--- a/woodwork/statistics_utils/_parse_measures.py
+++ b/woodwork/statistics_utils/_parse_measures.py
@@ -1,5 +1,6 @@
 import warnings
 
+from woodwork.config import CONFIG_DEFAULTS
 from woodwork.exceptions import ParametersIgnoredWarning
 
 
@@ -46,7 +47,12 @@ def _parse_measures(measures):
             calc_spearman = True
             calc_max = True
         else:
-            raise ValueError("Unrecognized dependence measure %s" % measure)
+            raise ValueError(
+                "Unrecognized dependence measure {}. Valid measures are {}".format(
+                    measure,
+                    CONFIG_DEFAULTS["correlation_metrics"],
+                ),
+            )
 
     if calc_pearson:
         calc_order.append("pearson")
diff --git a/woodwork/table_accessor.py b/woodwork/table_accessor.py
index a09f86f..0db349a 100644
--- a/woodwork/table_accessor.py
+++ b/woodwork/table_accessor.py
@@ -1243,6 +1243,7 @@ class WoodworkTableAccessor:
         min_shared=25,
         random_seed=0,
         max_nunique=6000,
+        target_col=None,
     ):
         """Calculates dependence measures between all pairs of columns in the DataFrame that
         support measuring dependence. Supports boolean, categorical, datetime, and numeric data.
@@ -1291,6 +1292,8 @@ class WoodworkTableAccessor:
             max_nunique (int): The total maximum number of unique values for all large categorical columns (> 800 unique values).
                 Categorical columns will be dropped until this number is met or until there is only one large categorical column.
                 Defaults to 6000.
+            target_col (str): The column name of the target. If provided, will only calculate the dependence dictionary between other columns and this target column.
+                The target column will be `column_2` in the returned result. Defaults to None.
 
         Returns:
             list(dict): A list containing dictionaries that have keys `column_1`,
@@ -1311,6 +1314,7 @@ class WoodworkTableAccessor:
             min_shared=min_shared,
             random_seed=random_seed,
             max_nunique=max_nunique,
+            target_col=target_col,
         )
 
     def dependence(
@@ -1324,6 +1328,7 @@ class WoodworkTableAccessor:
         min_shared=25,
         random_seed=0,
         max_nunique=6000,
+        target_col=None,
     ):
         """Calculates dependence measures between all pairs of columns in the DataFrame that
         support measuring dependence. Supports boolean, categorical, datetime, and numeric data.
@@ -1372,6 +1377,8 @@ class WoodworkTableAccessor:
             max_nunique (int): The maximum number of unique values for large categorical columns (> 800 unique values).
                 Categorical columns will be dropped until this number is met or until there is only one large categorical column.
                 Defaults to 6000.
+            target_col (str): The column name of the target. If provided, will only calculate the dependence dictionary between other columns and this target column.
+                The target column will be `column_2` in the returned result. Defaults to None.
 
         Returns:
             pd.DataFrame: A DataFrame with the columns `column_1`,
@@ -1393,6 +1400,7 @@ class WoodworkTableAccessor:
             min_shared=min_shared,
             random_seed=random_seed,
             max_nunique=max_nunique,
+            target_col=target_col,
         )
         return pd.DataFrame(dep_dict)
 


Test patch:
diff --git a/woodwork/tests/accessor/test_statistics.py b/woodwork/tests/accessor/test_statistics.py
index 5ce95ef..f0f139d 100644
--- a/woodwork/tests/accessor/test_statistics.py
+++ b/woodwork/tests/accessor/test_statistics.py
@@ -9,7 +9,7 @@ import pandas as pd
 import pytest
 
 from woodwork.accessor_utils import _is_spark_dataframe, init_series
-from woodwork.config import config
+from woodwork.config import CONFIG_DEFAULTS, config
 from woodwork.exceptions import ParametersIgnoredWarning, SparseDataWarning
 from woodwork.logical_types import (
     URL,
@@ -2076,6 +2076,32 @@ def test_spearman_ordinal(df_mi, use_ordinal):
     assert "strs2" not in valid_sp_columns
 
 
+def test_dependence_target_col_not_exist(df_mi):
+    df_mi.ww.init()
+    with pytest.raises(ValueError, match="target_col 'value' not in the"):
+        df_mi.ww.dependence_dict(target_col="value")
+
+    with pytest.raises(ValueError, match="target_col 'value' not in the"):
+        df_mi.ww.dependence(target_col="value")
+
+
+def test_dependence_target_col_in_output(df_mi):
+    df_mi.ww.init()
+    dep = df_mi.ww.dependence_dict(min_shared=12, target_col="ints")
+    assert all([x["column_2"] == "ints" for x in dep])
+    assert len(dep) < len(df_mi.columns)
+    assert all([isinstance(x["max"], float) for x in dep])
+
+
+def test_dependence_dict_target_col_in_output(df_mi):
+    df_mi.ww.init()
+    dep = df_mi.ww.dependence(min_shared=12, target_col="ints")
+    assert set(list(dep["column_2"])) == {"ints"}
+    assert set(list(dep.columns) + ["all"]) == set(
+        ["column_1", "column_2"] + CONFIG_DEFAULTS["correlation_metrics"],
+    )
+
+
 def test_convert_ordinal_to_numeric():
     df = pd.DataFrame(
         {