Repository: alteryx/woodwork
Pull request #1504 (2022-08-24 13:09:01)

Allow selective transformation
`ignore_columns` has been added as an argument when initializing a dataframe with woodwork. Any column name(s) that are passed in to `df.ww.init(ignore_columns=[col_1, col_2, etc])` will be ignored during initialization, as long as they are included as part of an existing schema and aren't being forced to a logical type via the `logical_types` argument. This will cause all columns included in `ignore_columns` to not be transformed.

codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1504?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) Report
> Merging [#1504](https://codecov.io/gh/alteryx/woodwork/pull/1504?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (1539eb8) into [main](https://codecov.io/gh/alteryx/woodwork/commit/66352cc8bb370610d6defed28f176e7a138d4560?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (66352cc) will **decrease** coverage by `80.35%`.
> The diff coverage is `14.58%`.

```diff
@@             Coverage Diff             @@
##             main    #1504       +/-   ##
===========================================
- Coverage   98.70%   18.34%   -80.36%     
===========================================
  Files          95       95               
  Lines       10708    10755       +47     
===========================================
- Hits        10569     1973     -8596     
- Misses        139     8782     +8643     
```


| [Impacted Files](https://codecov.io/gh/alteryx/woodwork/pull/1504?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | Coverage Δ | |
|---|---|---|
| [woodwork/tests/accessor/test\_table\_accessor.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF90YWJsZV9hY2Nlc3Nvci5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/table\_accessor.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGFibGVfYWNjZXNzb3IucHk=) | `50.53% <26.31%> (-49.47%)` | :arrow_down: |
| [woodwork/exceptions.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvZXhjZXB0aW9ucy5weQ==) | `68.00% <100.00%> (-32.00%)` | :arrow_down: |
| [woodwork/tests/test\_config.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdGVzdF9jb25maWcucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/test\_version.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdGVzdF92ZXJzaW9uLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF91dGlscy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_concat.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF9jb25jYXQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/type\_system/conftest.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdHlwZV9zeXN0ZW0vY29uZnRlc3QucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_read\_file.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF9yZWFkX2ZpbGUucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/accessor/test\_indexers.py](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF9pbmRleGVycy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [59 more](https://codecov.io/gh/alteryx/woodwork/pull/1504/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)


Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index f3e176e..621f7ed 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -8,12 +8,13 @@ Future Release
     * Enhancements
     * Fixes
     * Changes
+        * Added ``ignore_columns`` as an argument when initializing a dataframe (:pr:`1504`)
     * Documentation Changes
     * Testing Changes
         * Add kickoff for create conda forge pull request from release (:pr:`1515`)
 
     Thanks to the following people for contributing to this release:
-    :user:`gsheni`
+    :user:`gsheni`, :user:`ParthivNaresh`
 
 v0.18.0 August 31, 2022
 =======================
diff --git a/woodwork/exceptions.py b/woodwork/exceptions.py
index b9dd283..a84f105 100644
--- a/woodwork/exceptions.py
+++ b/woodwork/exceptions.py
@@ -75,6 +75,10 @@ class ColumnNotPresentInSchemaError(KeyError):
         )
 
 
+class ColumnBothIgnoredAndSetError(KeyError):
+    pass
+
+
 class WoodworkNotInitError(AttributeError):
     pass
 
diff --git a/woodwork/table_accessor.py b/woodwork/table_accessor.py
index d727215..77c79ca 100644
--- a/woodwork/table_accessor.py
+++ b/woodwork/table_accessor.py
@@ -1,7 +1,7 @@
 import copy
 import warnings
 import weakref
-from typing import Any, Callable, Dict, Iterable, List, Sequence, Set, Union
+from typing import Any, Callable, Dict, Iterable, List, Optional, Sequence, Set, Union
 
 import pandas as pd
 
@@ -14,10 +14,12 @@ from woodwork.accessor_utils import (
     init_series,
 )
 from woodwork.exceptions import (
+    ColumnBothIgnoredAndSetError,
     ColumnNotPresentError,
     IndexTagRemovedWarning,
     ParametersIgnoredWarning,
     TypingInfoMismatchWarning,
+    WoodworkNotInitError,
 )
 from woodwork.indexers import _iLocIndexer, _locIndexer
 from woodwork.logical_types import Datetime, LogicalType
@@ -64,6 +66,10 @@ class WoodworkTableAccessor:
             logical_types (Dict[str -> LogicalType], optional): Dictionary mapping column names in
                 the DataFrame to the LogicalType for the column. Setting a column's logical type to None in this dict will
                 force a logical to be inferred.
+            ignore_columns (list[str] or set[str], optional): List of columns to ignore for inferring logical types. If a column name
+                is included in this list, then it cannot be part of the logical_types dictionary argument, and it must be part
+                of an existing schema for the dataframe. This argument can be used when a column has a logical type that has
+                already been inferred and its physical dtype is not expected to have changed since its last inference.
             already_sorted (bool, optional): Indicates whether the input DataFrame is already sorted on the time
                 index. If False, will sort the dataframe first on the time_index and then on the index (pandas DataFrame
                 only). Defaults to False.
@@ -121,20 +127,23 @@ class WoodworkTableAccessor:
 
     def init_with_partial_schema(
         self,
-        schema: TableSchema = None,
-        index: str = None,
-        time_index: str = None,
-        logical_types: Dict[ColumnName, Union[str, LogicalType, None]] = None,
-        already_sorted: bool = False,
-        name: str = None,
-        semantic_tags: Dict[ColumnName, Union[str, List[str], Set[str]]] = None,
-        table_metadata: dict = None,
-        column_metadata: Dict[ColumnName, dict] = None,
-        use_standard_tags: Union[bool, UseStandardTagsDict] = None,
-        column_descriptions: Dict[ColumnName, str] = None,
-        column_origins: Union[str, Dict[ColumnName, str]] = None,
-        null_invalid_values: bool = False,
-        validate: bool = True,
+        schema: Optional[TableSchema] = None,
+        index: Optional[str] = None,
+        time_index: Optional[str] = None,
+        logical_types: Optional[Dict[ColumnName, Union[str, LogicalType, None]]] = None,
+        ignore_columns: Optional[List[str]] = None,
+        already_sorted: Optional[bool] = False,
+        name: Optional[str] = None,
+        semantic_tags: Optional[
+            Dict[ColumnName, Union[str, List[str], Set[str]]]
+        ] = None,
+        table_metadata: Optional[dict] = None,
+        column_metadata: Optional[Dict[ColumnName, dict]] = None,
+        use_standard_tags: Optional[Union[bool, UseStandardTagsDict]] = None,
+        column_descriptions: Optional[Dict[ColumnName, str]] = None,
+        column_origins: Optional[Union[str, Dict[ColumnName, str]]] = None,
+        null_invalid_values: Optional[bool] = False,
+        validate: Optional[bool] = True,
         **kwargs,
     ) -> None:
         """Initializes Woodwork typing information for a DataFrame with a partial schema.
@@ -156,6 +165,10 @@ class WoodworkTableAccessor:
             logical_types (Dict[str -> LogicalType], optional): Dictionary mapping column names in
                 the DataFrame to the LogicalType for the column. Setting a column's logical type to None in this dict will
                 force a logical to be inferred.
+            ignore_columns (list[str] or set[str], optional): List of columns to ignore for inferring logical types. If a column name
+                is included in this list, then it cannot be part of the logical_types dictionary argument, and it must be part
+                of an existing schema for the dataframe. This argument can be used when a column has a logical type that has
+                already been inferred and its physical dtype is not expected to have changed since its last inference.
             already_sorted (bool, optional): Indicates whether the input DataFrame is already sorted on the time
                 index. If False, will sort the dataframe first on the time_index and then on the index (pandas DataFrame
                 only). Defaults to False.
@@ -190,6 +203,7 @@ class WoodworkTableAccessor:
                 index,
                 time_index,
                 logical_types,
+                ignore_columns,
                 schema,
                 use_standard_tags,
             )
@@ -224,6 +238,7 @@ class WoodworkTableAccessor:
             self._dataframe,
             logical_types,
             existing_logical_types,
+            ignore_columns,
             null_invalid_values=null_invalid_values,
         )
         column_descriptions = {
@@ -1505,6 +1520,7 @@ def _validate_accessor_params(
     index,
     time_index,
     logical_types,
+    ignore_columns,
     schema,
     use_standard_tags,
 ) -> None:
@@ -1520,6 +1536,8 @@ def _validate_accessor_params(
         _check_index(dataframe, index)
     if logical_types:
         _check_logical_types(dataframe.columns, logical_types)
+    if ignore_columns:
+        _check_ignore_columns(dataframe.columns, logical_types, schema, ignore_columns)
     if time_index is not None:
         datetime_format = None
         logical_type = None
@@ -1580,6 +1598,30 @@ def _check_logical_types(dataframe_columns, logical_types):
         )
 
 
+def _check_ignore_columns(dataframe_columns, logical_types, schema, ignore_columns):
+    if not isinstance(ignore_columns, (list, set)):
+        raise TypeError("ignore_columns must be a list or set")
+    cols_not_found = set(ignore_columns).difference(set(dataframe_columns))
+    if cols_not_found:
+        raise ColumnNotPresentError(
+            "ignore_columns contains columns that are not present in "
+            f"dataframe: {sorted(list(cols_not_found))}",
+        )
+    if logical_types:
+        col_ignored_and_set = set(logical_types.keys()).intersection(
+            set(ignore_columns)
+        )
+        if col_ignored_and_set:
+            raise ColumnBothIgnoredAndSetError(
+                "ignore_columns contains columns that are being set "
+                f"in logical_types: {list(col_ignored_and_set)}"
+            )
+    if schema is None:
+        raise WoodworkNotInitError(
+            "ignore_columns cannot be set when the dataframe has no existing " "schema."
+        )
+
+
 def _check_schema(dataframe, schema):
     if not isinstance(schema, TableSchema):
         raise TypeError("Provided schema must be a Woodwork.TableSchema object.")
@@ -1610,15 +1652,20 @@ def _check_use_standard_tags(use_standard_tags):
 
 def _infer_missing_logical_types(
     dataframe: AnyDataFrame,
-    force_logical_types: Dict[ColumnName, Union[str, LogicalType]] = None,
-    existing_logical_types: Dict[ColumnName, Union[str, LogicalType]] = None,
+    force_logical_types: Optional[Dict[ColumnName, Union[str, LogicalType]]] = None,
+    existing_logical_types: Optional[Dict[ColumnName, Union[str, LogicalType]]] = None,
+    ignore_columns: Optional[List[str]] = None,
     null_invalid_values: bool = False,
 ):
     """Performs type inference and updates underlying data"""
     force_logical_types = force_logical_types or {}
     existing_logical_types = existing_logical_types or {}
+    ignore_columns = ignore_columns or []
     parsed_logical_types = {}
     for name in dataframe.columns:
+        if name in ignore_columns and name in existing_logical_types:
+            parsed_logical_types[name] = existing_logical_types.get(name)
+            continue
         logical_type = (
             force_logical_types.get(name)
             if name in force_logical_types


Test patch:
diff --git a/woodwork/tests/accessor/test_table_accessor.py b/woodwork/tests/accessor/test_table_accessor.py
index 4162696..17a7071 100644
--- a/woodwork/tests/accessor/test_table_accessor.py
+++ b/woodwork/tests/accessor/test_table_accessor.py
@@ -15,6 +15,7 @@ from woodwork.accessor_utils import (
     init_series,
 )
 from woodwork.exceptions import (
+    ColumnBothIgnoredAndSetError,
     ColumnNotPresentError,
     ColumnNotPresentInSchemaError,
     IndexTagRemovedWarning,
@@ -733,6 +734,96 @@ def test_float_dtype_inference_on_init():
     assert df["floats_nan_specified"].dtype == "float64"
 
 
+def test_ignore_columns_error_columns_not_in_dataframe():
+    df = pd.DataFrame()
+    df["ints"] = [i for i in range(100)]
+
+    err_msg = re.escape(
+        "ignore_columns contains columns that are not present in dataframe: ['floats']",
+    )
+    with pytest.raises(ColumnNotPresentError, match=err_msg):
+        df.ww.init(ignore_columns=["floats"])
+
+
+@pytest.mark.parametrize("type_", [list, set, tuple])
+def test_ignore_columns_errors_various(type_):
+    df = pd.DataFrame()
+    df["ints"] = [i for i in range(100)]
+    df["floats"] = [i * 1.1 for i in range(100)]
+    df["dates"] = pd.date_range("1/1/21", periods=100)
+
+    if type_ in [list, set]:
+        # Checks that columns in ignore_columns must be part of existing schema
+        with pytest.raises(
+            WoodworkNotInitError,
+            match="ignore_columns cannot be set when the dataframe has no existing schema.",
+        ):
+            df.ww.init(ignore_columns=type_(["ints", "floats", "dates"]))
+
+        df.ww.init()
+        logical_types = {
+            "ints": Integer,
+        }
+        err_msg = re.escape(
+            "ignore_columns contains columns that are being set in logical_types: ['ints']",
+        )
+        # Checks that ignore_columns and logical_types don't overlap
+        with pytest.raises(ColumnBothIgnoredAndSetError, match=err_msg):
+            schema = df.ww.schema
+            df.ww.init(
+                ignore_columns=["ints", "floats", "dates"],
+                logical_types=logical_types,
+                schema=schema,
+            )
+    else:
+        # Checks for acceptable type
+        with pytest.raises(
+            TypeError,
+            match="ignore_columns must be a list or set",
+        ):
+            df.ww.init(ignore_columns=type_(["ints", "floats", "dates"]))
+
+
+def test_ignore_columns_can_force_logical_type_not_in_ignore_columns():
+    df = pd.DataFrame()
+    df["ints"] = [i for i in range(100)]
+    df["floats"] = [i * 1.1 for i in range(100)]
+    df["dates"] = pd.date_range("1/1/21", periods=100)
+    df.ww.init()
+
+    assert isinstance(df.ww.logical_types["ints"], Integer)
+    assert isinstance(df.ww.logical_types["floats"], Double)
+    assert isinstance(df.ww.logical_types["dates"], Datetime)
+
+    schema = df.ww.schema
+    logical_types = {
+        "ints": IntegerNullable,
+    }
+    df.ww.init(
+        ignore_columns={"floats", "dates"}, logical_types=logical_types, schema=schema
+    )
+
+    assert isinstance(df.ww.logical_types["ints"], IntegerNullable)
+    assert isinstance(df.ww.logical_types["floats"], Double)
+    assert isinstance(df.ww.logical_types["dates"], Datetime)
+
+
+@patch("woodwork.logical_types._replace_nans")
+def test_ignore_columns(mock_replace_nans):
+    df = pd.DataFrame()
+    df["ints"] = [i for i in range(100)] + [np.nan]
+    df["floats"] = [True, False] * 50 + [pd.NA]
+
+    mock_replace_nans.side_effect = [df["ints"], df["floats"]]
+    df.ww.init()
+
+    assert mock_replace_nans.call_count == 2
+    schema = df.ww.schema
+    df.ww.init(ignore_columns=["ints", "floats"], schema=schema)
+
+    assert mock_replace_nans.call_count == 2
+
+
 def test_datetime_timezones(timezones_df):
     timezones_df.ww.init()
     actual_timezones = [lt.timezone for lt in timezones_df.ww.logical_types.values()]