Repository: alteryx/woodwork
Pull request #1458 (2022-06-29 14:25:21)

Better inference for Integer/Boolean Nullables
Fixes: #1172, #1038 

Performance tests are [here](https://alteryx.atlassian.net/wiki/spaces/PS/pages/1410727994/Performance+Tests+-+Expanding+nan+Inference+for+IntegerNullable+BooleanNullable)

codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1458?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) Report
> Merging [#1458](https://codecov.io/gh/alteryx/woodwork/pull/1458?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (fe0816d) into [main](https://codecov.io/gh/alteryx/woodwork/commit/5614dfe5e0abc30819f2338828b4e1dec1f7197f?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (5614dfe) will **decrease** coverage by `80.24%`.
> The diff coverage is `13.11%`.

> :exclamation: Current head fe0816d differs from pull request most recent head 7e14bec. Consider uploading reports for the commit 7e14bec to get more accurate results

```diff
@@             Coverage Diff             @@
##             main    #1458       +/-   ##
===========================================
- Coverage   98.68%   18.44%   -80.25%     
===========================================
  Files          95       95               
  Lines       10587    10625       +38     
===========================================
- Hits        10448     1960     -8488     
- Misses        139     8665     +8526     
```


| [Impacted Files](https://codecov.io/gh/alteryx/woodwork/pull/1458?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | Coverage Δ | |
|---|---|---|
| [woodwork/tests/accessor/test\_statistics.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF9zdGF0aXN0aWNzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/accessor/test\_table\_accessor.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF90YWJsZV9hY2Nlc3Nvci5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/logical\_types/test\_logical\_types.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvbG9naWNhbF90eXBlcy90ZXN0X2xvZ2ljYWxfdHlwZXMucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/type\_system/test\_custom\_types.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdHlwZV9zeXN0ZW0vdGVzdF9jdXN0b21fdHlwZXMucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/type\_sys/inference\_functions.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdHlwZV9zeXMvaW5mZXJlbmNlX2Z1bmN0aW9ucy5weQ==) | `30.00% <0.00%> (-70.00%)` | :arrow_down: |
| [woodwork/type\_sys/type\_system.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdHlwZV9zeXMvdHlwZV9zeXN0ZW0ucHk=) | `31.97% <0.00%> (-68.03%)` | :arrow_down: |
| [woodwork/logical\_types.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvbG9naWNhbF90eXBlcy5weQ==) | `50.32% <20.00%> (-49.68%)` | :arrow_down: |
| [woodwork/column\_accessor.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvY29sdW1uX2FjY2Vzc29yLnB5) | `50.84% <50.00%> (-49.16%)` | :arrow_down: |
| [woodwork/serializers/parquet\_serializer.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc2VyaWFsaXplcnMvcGFycXVldF9zZXJpYWxpemVyLnB5) | `90.78% <100.00%> (-9.22%)` | :arrow_down: |
| [woodwork/table\_accessor.py](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGFibGVfYWNjZXNzb3IucHk=) | `52.13% <100.00%> (-47.87%)` | :arrow_down: |
| ... and [67 more](https://codecov.io/gh/alteryx/woodwork/pull/1458/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1458?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1458?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Last update [5614dfe...7e14bec](https://codecov.io/gh/alteryx/woodwork/pull/1458?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx).

gsheni: @ParthivNaresh what was the change in inference after removing the apply?
ParthivNaresh: > @ParthivNaresh what was the change in inference after removing the apply?

It looks like `pandas` is able to handle `pd.NA` without a problem, all tests are still passing and expected logical types are still showing up
ParthivNaresh: > @ParthivNaresh what was the change in inference speed/time after removing the apply?

The change should be seen [here](https://alteryx.atlassian.net/wiki/spaces/PS/pages/1410727994/Performance+Tests+-+Expanding+nan+Inference+for+IntegerNullable+BooleanNullable)

Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 8c701ec..253fdf1 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -15,7 +15,9 @@ Future Release
         * Added ability to null invalid values for ``AgeFractional`` and ``AgeNullable`` logical types (:pr:`1462`)
         * Added ability to null invalid values for ``LatLong`` logical type (:pr:`1465`)
         * Added ability to null invalid values for ``PostalCode`` logical type (US only) (:pr:`1467`)
+        * Added smarter inference for ``IntegerNullable`` and ``BooleanNullable`` types (:pr:`1458`)
     * Fixes
+        * Fixed inference of all null string values as ``Unknown`` instead of ``Datetime`` (:pr:`1458`)
     * Changes
         * Set the minimum acceptable version of pandas to ``1.4.0`` for woodwork and ``1.4.3`` for spark add-on (:pr:`1461`)
         * Dropped support for Python 3.7 (:pr:`1461`)
diff --git a/woodwork/column_accessor.py b/woodwork/column_accessor.py
index 3f95145..a627477 100644
--- a/woodwork/column_accessor.py
+++ b/woodwork/column_accessor.py
@@ -3,6 +3,7 @@ import warnings
 import weakref
 
 import pandas as pd
+from pandas.api import types as pdtypes
 
 from woodwork.accessor_utils import (
     _check_column_schema,
@@ -105,7 +106,10 @@ class WoodworkColumnAccessor:
                     logical_type.validate(self._series)
                 else:
                     valid_dtype = logical_type._get_valid_dtype(type(self._series))
-                    if valid_dtype != str(self._series.dtype):
+                    if valid_dtype != str(self._series.dtype) and not (
+                        pdtypes.is_integer_dtype(valid_dtype)
+                        and pdtypes.is_float_dtype(self._series.dtype)
+                    ):
                         raise TypeValidationError(
                             f"Cannot initialize Woodwork. Series dtype '{self._series.dtype}' is "
                             f"incompatible with {logical_type} dtype. Try converting series "
diff --git a/woodwork/config.py b/woodwork/config.py
index 3a599c9..182ab2c 100644
--- a/woodwork/config.py
+++ b/woodwork/config.py
@@ -1,5 +1,8 @@
 import contextlib
 
+import numpy as np
+import pandas as pd
+
 CONFIG_DEFAULTS = {
     "categorical_threshold": 0.2,
     "numeric_categorical_threshold": None,
@@ -19,7 +22,32 @@ CONFIG_DEFAULTS = {
     ),
     "phone_inference_regex": r"(?:\+?(0{2})?1[-.\s●]?)?\(?([2-9][0-9]{2})\)?[-\.\s●]?([2-9][0-9]{2})[-\.\s●]?([0-9]{4})$",
     "postal_code_inference_regex": r"^[0-9]{5}(?:-[0-9]{4})?$",
-    "nan_values": ["", "None", "nan", "NaN", "<NA>", "null"],
+    "nan_values": [
+        "",
+        None,
+        np.nan,
+        pd.NaT,
+        "None",
+        "NONE",
+        "none",
+        "NULL",
+        "Null",
+        "null",
+        "NAN",
+        "NaN",
+        "Nan",
+        "nan",
+        "NA",
+        "na",
+        "N/A",
+        "n/a",
+        "n/A",
+        "N/a",
+        "<NA>",
+        "<N/A>",
+        "<n/a>",
+        "<na>",
+    ],
     "frequence_inference_window_length": 15,
     "frequence_inference_threshold": 0.9,
 }
diff --git a/woodwork/logical_types.py b/woodwork/logical_types.py
index 7976d75..4a7915a 100644
--- a/woodwork/logical_types.py
+++ b/woodwork/logical_types.py
@@ -1,9 +1,11 @@
 import re
 import warnings
 
+import numpy as np
 import pandas as pd
 from pandas.api import types as pdtypes
 
+import woodwork as ww
 from woodwork.accessor_utils import _is_dask_series, _is_spark_series
 from woodwork.config import config
 from woodwork.exceptions import (
@@ -214,6 +216,7 @@ class BooleanNullable(LogicalType):
     primary_dtype = "boolean"
 
     def transform(self, series, null_invalid_values=False):
+        series = _replace_nans(series)
         if null_invalid_values:
             series = _coerce_boolean(series)
         return super().transform(series)
@@ -371,6 +374,7 @@ class Double(LogicalType):
     standard_tags = {"numeric"}
 
     def transform(self, series, null_invalid_values=False):
+        series = _replace_nans(series)
         if null_invalid_values:
             series = _coerce_numeric(series)
         return super().transform(series)
@@ -418,6 +422,7 @@ class IntegerNullable(LogicalType):
         Returns:
             Series: A series of integers.
         """
+        series = _replace_nans(series)
         if null_invalid_values:
             series = _coerce_integer(series)
         return super().transform(series)
@@ -830,6 +835,23 @@ def _regex_validate(regex_key, series, return_invalid_values):
             raise TypeValidationError(info)
 
 
+def _replace_nans(series: pd.Series) -> pd.Series:
+    """
+    Replaces empty string values, string representations of NaN values ("nan", "<NA>"), and NaN equivalents
+    with np.nan or pd.NA depending on column dtype.
+    """
+    original_dtype = series.dtype
+    if str(original_dtype) == "string":
+        series = series.replace(ww.config.get_option("nan_values"), pd.NA)
+        return series
+    if not _is_spark_series(series):
+        series = series.replace(ww.config.get_option("nan_values"), np.nan)
+    if str(original_dtype) == "boolean":
+        series = series.astype(original_dtype)
+
+    return series
+
+
 def _validate_age(series, return_invalid_values):
     """Validates data values are non-negative."""
     invalid = _get_index_invalid_age(series)
diff --git a/woodwork/serializers/parquet_serializer.py b/woodwork/serializers/parquet_serializer.py
index c256ea2..32ec517 100644
--- a/woodwork/serializers/parquet_serializer.py
+++ b/woodwork/serializers/parquet_serializer.py
@@ -90,7 +90,7 @@ class ParquetSerializer(Serializer):
 
     def _save_parquet_table_to_disk(self):
         """Writes data to disk with the updated metadata including WW typing info."""
-        import pyarrow.parquet as pq
+        from pyarrow import parquet as pq
 
         if _is_dask_dataframe(self.dataframe):
             path, dataframe = self._setup_for_dask_and_spark()
diff --git a/woodwork/table_accessor.py b/woodwork/table_accessor.py
index 1785d6b..d7cc093 100644
--- a/woodwork/table_accessor.py
+++ b/woodwork/table_accessor.py
@@ -1599,12 +1599,12 @@ def _infer_missing_logical_types(
     existing_logical_types = existing_logical_types or {}
     parsed_logical_types = {}
     for name in dataframe.columns:
-        series = dataframe[name]
         logical_type = (
             force_logical_types.get(name)
             if name in force_logical_types
             else existing_logical_types.get(name)
         )
+        series = dataframe[name]
         parsed_logical_types[name] = _get_column_logical_type(
             series,
             logical_type,
@@ -1615,7 +1615,12 @@ def _infer_missing_logical_types(
             null_invalid_values=null_invalid_values,
         )
         if updated_series is not series:
-            dataframe[name] = updated_series
+            # NotImplementedError thrown by dask when attempting to re-initialize
+            # data after being assigned a numeric column name
+            try:
+                dataframe[name] = updated_series
+            except NotImplementedError:
+                pass
     return parsed_logical_types
 
 
diff --git a/woodwork/type_sys/inference_functions.py b/woodwork/type_sys/inference_functions.py
index 8c5f251..6904d44 100644
--- a/woodwork/type_sys/inference_functions.py
+++ b/woodwork/type_sys/inference_functions.py
@@ -41,7 +41,7 @@ def categorical_func(series):
 
 def integer_func(series):
     if integer_nullable_func(series) and not series.isnull().any():
-        return True
+        return all(series.mod(1).eq(0))
     return False
 
 
@@ -52,6 +52,11 @@ def integer_nullable_func(series):
             return not _is_categorical_series(series, threshold)
         else:
             return True
+    elif pdtypes.is_float_dtype(series.dtype):
+        if not series.isnull().any():
+            return False
+        series_no_null = series.dropna()
+        return all(series_no_null.mod(1).eq(0))
 
     return False
 
@@ -78,6 +83,18 @@ def boolean_nullable_func(series):
         series.dtype,
     ):
         return True
+    elif pdtypes.is_object_dtype(series.dtype):
+        series_no_null = series.dropna()
+        try:
+            series_no_null_unq = set(series_no_null)
+            if series_no_null_unq in [
+                {False, True},
+                {True},
+                {False},
+            ]:
+                return True
+        except TypeError:  # Necessary to check for non-hashable values because of object dtype consideration
+            return False
     return False
 
 
diff --git a/woodwork/type_sys/type_system.py b/woodwork/type_sys/type_system.py
index 9013f3b..acfe56c 100644
--- a/woodwork/type_sys/type_system.py
+++ b/woodwork/type_sys/type_system.py
@@ -29,6 +29,7 @@ from woodwork.logical_types import (
     SubRegionCode,
     Timedelta,
     Unknown,
+    _replace_nans,
 )
 from woodwork.type_sys.inference_functions import (
     boolean_func,
@@ -331,8 +332,11 @@ class TypeSystem(object):
         types_to_check = [
             ltype for ltype in self.root_types if ltype != NaturalLanguage
         ]
-        type_matches = get_inference_matches(types_to_check, series)
+        series_nan_cast = _replace_nans(series)  # Will change dtype
+        if series_nan_cast.count() == 0:
+            return Unknown()
 
+        type_matches = get_inference_matches(types_to_check, series_nan_cast)
         if len(type_matches) == 0:
             # Check if this is NaturalLanguage, otherwise set
             # type to default type (Unknown). Assume that a column
@@ -353,7 +357,12 @@ class TypeSystem(object):
             # If multiple matches, get the most specific one. If multiple
             # matches have the same level of specificity, the first
             # match found at that level will be returned
-            best_match = type_matches[0]
+            if (
+                Categorical in type_matches or Double in type_matches
+            ) and IntegerNullable in type_matches:
+                best_match = IntegerNullable
+            else:
+                best_match = type_matches[0]
             best_depth = self._get_depth(best_match)
             for logical_type in type_matches[1:]:
                 ltype_depth = self._get_depth(logical_type)
diff --git a/woodwork/utils.py b/woodwork/utils.py
index a9e12b6..72ee299 100644
--- a/woodwork/utils.py
+++ b/woodwork/utils.py
@@ -12,6 +12,7 @@ import woodwork as ww
 from woodwork.exceptions import TypeValidationError
 
 # Dictionary mapping formats/content types to the appropriate pandas read function
+
 type_to_read_func_map = {
     "csv": pd.read_csv,
     "text/csv": pd.read_csv,
@@ -143,6 +144,8 @@ def read_file(
     Returns:
         pd.DataFrame: DataFrame created from the specified file with Woodwork typing information initialized.
     """
+    from woodwork.logical_types import _replace_nans
+
     if content_type is None:
         inferred_type, _ = guess_type(filepath)
         if inferred_type is None:
@@ -176,7 +179,7 @@ def read_file(
     dataframe = type_to_read_func_map[content_type](filepath, **kwargs)
 
     if replace_nan:
-        dataframe = _replace_nan_strings(dataframe)
+        dataframe = dataframe.apply(_replace_nans)
 
     dataframe.ww.init(
         name=name,
@@ -603,35 +606,11 @@ def _infer_datetime_format(dates, n=100):
     return mode_fmt
 
 
-def _replace_nan_strings(df: pd.DataFrame) -> pd.DataFrame:
-    """Replaces empty string values and string representations of
-    NaN values ("nan", "<NA>") with np.nan or pd.NA depending on
-    column dtype."""
-    df = df.fillna(value=np.nan)
-
-    for col, dtype in df.dtypes.items():
-        replace_val = np.nan
-        if str(dtype) == "boolean":
-            # All replace calls below fail with boolean dtype
-            # but boolean cols cannot contain strings to begin with.
-            continue
-        elif str(dtype) == "string":
-            # Must use pd.NA as replacement value for string dtype
-            replace_val = pd.NA
-
-        replaced_series = df[col].replace(r"^\s*$", replace_val, regex=True)
-        replaced_series = replaced_series.replace(
-            ww.config.get_option("nan_values"),
-            replace_val,
-        )
-        df[col] = replaced_series
-
-    return df
-
-
 def _parse_latlong(latlong):
-    nan_values = [x for x in ww.config.get_option("nan_values") if len(x)]
-    nan_values = "|".join(nan_values)
+    nan_values_strs = [
+        x for x in ww.config.get_option("nan_values") if isinstance(x, str) and len(x)
+    ]
+    nan_values = "|".join(nan_values_strs)
 
     latlong = re.sub(nan_values, "None", latlong)
 


Test patch:
diff --git a/woodwork/tests/accessor/test_statistics.py b/woodwork/tests/accessor/test_statistics.py
index f2ea4f0..0fb9d59 100644
--- a/woodwork/tests/accessor/test_statistics.py
+++ b/woodwork/tests/accessor/test_statistics.py
@@ -1253,6 +1253,7 @@ def test_describe_dict_extra_stats_overflow_range(
 
 
 def test_value_counts(categorical_df):
+    nan = np.nan
     logical_types = {
         "ints": IntegerNullable,
         "categories1": Categorical,
@@ -1274,56 +1275,27 @@ def test_value_counts(categorical_df):
         {"value": 1, "count": 2},
         {"value": 3, "count": 1},
     ]
-    expected_cat2 = [
-        {"value": np.nan, "count": 6},
-        {"value": "test", "count": 3},
-        {"value": "test2", "count": 1},
-    ]
-    expected_cat3 = [
-        {"value": np.nan, "count": 7},
-        {"value": "test", "count": 3},
-    ]
     # Spark converts numeric categories to strings, so we need to update the expected values for this
     # Spark will result in `pd.NA` instead of `np.nan` in categorical columns
     if _is_spark_dataframe(categorical_df):
         updated_results = []
         for items in expected_cat1:
             updated_results.append(
-                {k: (str(v) if k == "value" else v) for k, v in items.items()},
+                {k: (str(v) if k == "value" else v) for k, v in items.items()}
             )
         expected_cat1 = updated_results
 
-        updated_results = []
-        for items in expected_cat2:
-            updated_results.append(
-                {
-                    k: (
-                        str(v)
-                        if (k == "value" and v is not np.nan)
-                        else pd.NA
-                        if v is np.nan
-                        else v
-                    )
-                    for k, v in items.items()
-                },
-            )
-        expected_cat2 = updated_results
+        nan = pd.NA
 
-        updated_results = []
-        for items in expected_cat3:
-            updated_results.append(
-                {
-                    k: (
-                        str(v)
-                        if (k == "value" and v is not np.nan)
-                        else pd.NA
-                        if v is np.nan
-                        else v
-                    )
-                    for k, v in items.items()
-                },
-            )
-        expected_cat3 = updated_results
+    expected_cat2 = [
+        {"value": nan, "count": 6},
+        {"value": "test", "count": 3},
+        {"value": "test2", "count": 1},
+    ]
+    expected_cat3 = [
+        {"value": nan, "count": 7},
+        {"value": "test", "count": 3},
+    ]
 
     assert val_cts["categories1"] == expected_cat1
     assert val_cts["categories2"] == expected_cat2
diff --git a/woodwork/tests/accessor/test_table_accessor.py b/woodwork/tests/accessor/test_table_accessor.py
index 44cf0cb..4162696 100644
--- a/woodwork/tests/accessor/test_table_accessor.py
+++ b/woodwork/tests/accessor/test_table_accessor.py
@@ -674,8 +674,8 @@ def test_int_dtype_inference_on_init():
     df.ww.init()
 
     assert df["ints_no_nans"].dtype == "int64"
-    assert df["ints_nan"].dtype == "float64"
-    assert df["ints_NA"].dtype == "category"
+    assert df["ints_nan"].dtype == "Int64"
+    assert df["ints_NA"].dtype == "Int64"
     assert df["ints_NA_specified"].dtype == "Int64"
 
 
@@ -692,8 +692,8 @@ def test_bool_dtype_inference_on_init():
     df.ww.init()
 
     assert df["bools_no_nans"].dtype == "bool"
-    assert df["bool_nan"].dtype == "category"
-    assert df["bool_NA"].dtype == "category"
+    assert df["bool_nan"].dtype == "boolean"
+    assert df["bool_NA"].dtype == "boolean"
     assert df["bool_NA_specified"].dtype == "boolean"
 
 
@@ -729,7 +729,7 @@ def test_float_dtype_inference_on_init():
 
     assert df["floats_no_nans"].dtype == "float64"
     assert df["floats_nan"].dtype == "float64"
-    assert df["floats_NA"].dtype == "category"
+    assert df["floats_NA"].dtype == "float64"
     assert df["floats_nan_specified"].dtype == "float64"
 
 
@@ -1056,20 +1056,6 @@ def test_sets_datetime64_dtype_on_init():
 def test_invalid_dtype_casting():
     column_name = "test_series"
 
-    # Cannot cast a column with pd.NA to Double
-    series = pd.Series([1.1, pd.NA, 3], name=column_name)
-    ltypes = {
-        column_name: Double,
-    }
-
-    err_msg = (
-        "Error converting datatype for test_series from type object to type "
-        "float64. Please confirm the underlying data is consistent with logical type Double."
-    )
-    df = pd.DataFrame(series)
-    with pytest.raises(TypeConversionError, match=err_msg):
-        df.ww.init(logical_types=ltypes)
-
     # Cannot cast Datetime to Double
     df = pd.DataFrame({column_name: ["2020-01-01", "2020-01-02", "2020-01-03"]})
     df.ww.init(logical_types={column_name: Datetime})
@@ -3022,7 +3008,7 @@ def test_infer_missing_logical_types_force_infer(sample_df):
         force_logical_types,
         existing_logical_types,
     )
-    assert parsed_logical_types["age"] == Double()
+    assert isinstance(parsed_logical_types["age"], IntegerNullable)
 
 
 def test_validate_unique_index_with_partial_schema():
diff --git a/woodwork/tests/logical_types/test_logical_types.py b/woodwork/tests/logical_types/test_logical_types.py
index 803a2fb..e95f0d4 100644
--- a/woodwork/tests/logical_types/test_logical_types.py
+++ b/woodwork/tests/logical_types/test_logical_types.py
@@ -1,5 +1,6 @@
 import re
 
+import numpy as np
 import pandas as pd
 import pytest
 
@@ -15,9 +16,13 @@ from woodwork.logical_types import (
     AgeFractional,
     AgeNullable,
     Boolean,
+    BooleanNullable,
     Categorical,
     Datetime,
+    Double,
     EmailAddress,
+    Integer,
+    IntegerNullable,
     LatLong,
     Ordinal,
     PhoneNumber,
@@ -660,3 +665,48 @@ def test_null_invalid_postal_code_numeric():
     expected = pd.DataFrame({"data": data})
     df.ww.init(logical_types=types, null_invalid_values=True)
     pd.testing.assert_frame_equal(df, expected)
+
+
+@pytest.mark.parametrize(
+    "null_type", [None, pd.NA, pd.NaT, np.nan, "null", "N/A", "mix", 5]
+)
+@pytest.mark.parametrize("data_type", [int, float])
+def test_integer_nullable(data_type, null_type):
+    nullable_nums = pd.DataFrame(
+        map(data_type, [1, 2, 3, 4, 5] * 20), columns=["num_nulls"]
+    )
+    nullable_nums["num_nulls"].iloc[-5:] = (
+        [None, pd.NA, np.nan, "NA", "none"]
+        if not isinstance(null_type, pd._libs.missing.NAType) and null_type == "mix"
+        else [null_type] * 5
+    )
+    nullable_nums.ww.init()
+
+    if not isinstance(null_type, int):
+        assert isinstance(nullable_nums.ww.logical_types["num_nulls"], IntegerNullable)
+        assert all(nullable_nums["num_nulls"][-5:].isna())
+    elif data_type is int:
+        assert isinstance(nullable_nums.ww.logical_types["num_nulls"], Integer)
+    else:
+        assert isinstance(nullable_nums.ww.logical_types["num_nulls"], Double)
+
+
+@pytest.mark.parametrize(
+    "null_type", [None, pd.NA, pd.NaT, np.nan, "null", "N/A", "mix", True]
+)
+def test_boolean_nullable(null_type):
+    nullable_bools = pd.DataFrame([True, False] * 50, columns=["bool_nulls"])
+    nullable_bools["bool_nulls"].iloc[-5:] = (
+        [None, pd.NA, np.nan, "NA", "none"]
+        if not isinstance(null_type, pd._libs.missing.NAType) and null_type == "mix"
+        else [null_type] * 5
+    )
+    nullable_bools.ww.init()
+
+    if not isinstance(null_type, bool):
+        assert isinstance(
+            nullable_bools.ww.logical_types["bool_nulls"], BooleanNullable
+        )
+        assert all(nullable_bools["bool_nulls"][-5:].isna())
+    else:
+        assert isinstance(nullable_bools.ww.logical_types["bool_nulls"], Boolean)
diff --git a/woodwork/tests/type_system/conftest.py b/woodwork/tests/type_system/conftest.py
index 014c156..4c003f4 100644
--- a/woodwork/tests/type_system/conftest.py
+++ b/woodwork/tests/type_system/conftest.py
@@ -326,6 +326,7 @@ def pandas_nulls():
         pd.Series([np.nan, np.nan, np.nan, np.nan]),
         pd.Series([None, None, None, None]),
         pd.Series([None, np.nan, pd.NA, None]),
+        pd.Series(["None", "null", "n/a", "NAN"]),
     ]
 
 
diff --git a/woodwork/tests/type_system/test_custom_types.py b/woodwork/tests/type_system/test_custom_types.py
index 62e022d..27a7ce1 100644
--- a/woodwork/tests/type_system/test_custom_types.py
+++ b/woodwork/tests/type_system/test_custom_types.py
@@ -51,18 +51,18 @@ def test_custom_type_with_accessor(sample_df):
 
 def test_accessor_override_default_function(sample_df):
     def new_double_func(series):
-        if pdtypes.is_integer_dtype(series.dtype):
+        if pdtypes.is_bool_dtype(series.dtype):
             return True
         return False
 
-    # Update functions to cause 'age' to be recognized as Double instead fo Integer
+    # Update functions to cause 'is_registered' to be recognized as Double instead fo BooleanNullable
     ww.type_system.update_inference_function(
         "Double",
         inference_function=new_double_func,
     )
-    ww.type_system.update_inference_function("Integer", inference_function=None)
+    ww.type_system.update_inference_function("BooleanNullable", inference_function=None)
     sample_df.ww.init()
-    assert isinstance(sample_df.ww["age"].ww.logical_type, Double)
-    assert sample_df["age"].dtype == "float64"
+    assert isinstance(sample_df.ww["is_registered"].ww.logical_type, Double)
+    assert sample_df["is_registered"].dtype == "float64"
     # Reset global type system to original settings
     ww.type_system.reset_defaults()
diff --git a/woodwork/tests/type_system/test_ltype_inference.py b/woodwork/tests/type_system/test_ltype_inference.py
index 9f7f8a2..778d41f 100644
--- a/woodwork/tests/type_system/test_ltype_inference.py
+++ b/woodwork/tests/type_system/test_ltype_inference.py
@@ -120,7 +120,9 @@ def test_categorical_inference(categories):
     dtypes = ["object", "string", "category"]
     if _is_spark_series(categories[0]):
         dtypes = get_spark_dtypes(dtypes)
-    for series in categories:
+    for ind, series in enumerate(categories):
+        if ind == len(categories) - 1:
+            dtypes = ["string", "category"]
         for dtype in dtypes:
             inferred_type = ww.type_system.infer_logical_type(series.astype(dtype))
             assert isinstance(inferred_type, Categorical)
@@ -234,10 +236,12 @@ def test_unknown_inference(strings):
 
 def test_unknown_inference_all_null(nulls):
     dtypes = ["object", "string", "category", "datetime64[ns]"]
-    if _is_spark_series(nulls[0]):
-        dtypes = get_spark_dtypes(dtypes)
 
-    for series in nulls:
+    for ind, series in enumerate(nulls):
+        if ind == len(nulls) - 1:
+            dtypes = ["object", "string", "category"]
+        if _is_spark_series(nulls[0]):
+            dtypes = get_spark_dtypes(dtypes)
         for dtype in dtypes:
             inferred_type = ww.type_system.infer_logical_type(series.astype(dtype))
             inferred_type.transform(series)
diff --git a/woodwork/tests/utils/test_read_file.py b/woodwork/tests/utils/test_read_file.py
index 18c9e43..0523fec 100644
--- a/woodwork/tests/utils/test_read_file.py
+++ b/woodwork/tests/utils/test_read_file.py
@@ -5,8 +5,8 @@ import pandas as pd
 import pytest
 
 import woodwork as ww
+from woodwork.logical_types import _replace_nans
 from woodwork.serializers.orc_serializer import save_orc_file
-from woodwork.utils import _replace_nan_strings
 
 
 def test_read_file_errors_no_content_type(sample_df_pandas, tmpdir):
@@ -239,7 +239,7 @@ def test_replace_nan_strings():
     }
 
     df = pd.DataFrame(data=data)
-    replaced_df = _replace_nan_strings(df)
+    replaced_df = df.apply(_replace_nans)
     for col in replaced_df:
         assert replaced_df[col].isnull().sum() == expected_null_count[col]
 
diff --git a/woodwork/tests/utils/test_utils.py b/woodwork/tests/utils/test_utils.py
index c6ebd29..f890a89 100644
--- a/woodwork/tests/utils/test_utils.py
+++ b/woodwork/tests/utils/test_utils.py
@@ -339,7 +339,6 @@ def test_is_nan():
 
     assert not _is_nan([None, 1, 3])
     assert not _is_nan([])
-    assert not _is_nan("none")
     assert not _is_nan(0)
     assert not _is_nan(False)
     assert not _is_nan({"key": "value"})