Repository: alteryx/woodwork
Pull request #1465 (2022-07-05 20:47:09)

Null invalid LatLong

codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1465?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) Report
> Merging [#1465](https://codecov.io/gh/alteryx/woodwork/pull/1465?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (de25ddd) into [main](https://codecov.io/gh/alteryx/woodwork/commit/34c0166857a0bd7d5e6f035ca518bf1c6ffddd1d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (34c0166) will **decrease** coverage by `0.06%`.
> The diff coverage is `76.66%`.

```diff
@@            Coverage Diff             @@
##             main    #1465      +/-   ##
==========================================
- Coverage   98.68%   98.62%   -0.07%     
==========================================
  Files          95       95              
  Lines       10587    10607      +20     
==========================================
+ Hits        10448    10461      +13     
- Misses        139      146       +7     
```


| [Impacted Files](https://codecov.io/gh/alteryx/woodwork/pull/1465?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | Coverage Δ | |
|---|---|---|
| [woodwork/tests/utils/test\_utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1465/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF91dGlscy5weQ==) | `100.00% <ø> (ø)` | |
| [woodwork/logical\_types.py](https://codecov.io/gh/alteryx/woodwork/pull/1465/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvbG9naWNhbF90eXBlcy5weQ==) | `98.06% <33.33%> (-1.94%)` | :arrow_down: |
| [woodwork/utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1465/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdXRpbHMucHk=) | `99.63% <95.23%> (-0.37%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1465?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1465?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Last update [34c0166...de25ddd](https://codecov.io/gh/alteryx/woodwork/pull/1465?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx).


Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 94e5996..a91b701 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -13,6 +13,7 @@ Future Release
         * Added ability to null invalid values for ``URL`` logical type (:pr:`1459`)
         * Added ability to null invalid values for ``PhoneNumber`` logical type (:pr:`1460`)
         * Added ability to null invalid values for ``AgeFractional`` and ``AgeNullable`` logical types (:pr:`1462`)
+        * Added ability to null invalid values for ``LatLong`` logical type (:pr:`1465`)
     * Fixes
     * Changes
         * Set the minimum acceptable version of pandas to ``1.4.0`` for woodwork and ``1.4.3`` for spark add-on (:pr:`1461`)
diff --git a/woodwork/logical_types.py b/woodwork/logical_types.py
index f86f13e..524b305 100644
--- a/woodwork/logical_types.py
+++ b/woodwork/logical_types.py
@@ -15,6 +15,7 @@ from woodwork.type_sys.utils import _get_specified_ltype_params
 from woodwork.utils import (
     _infer_datetime_format,
     _is_valid_latlong_series,
+    _is_valid_latlong_value,
     _reformat_to_latlong,
     camel_to_snake,
     import_or_none,
@@ -515,6 +516,9 @@ class LatLong(LogicalType):
 
     def transform(self, series, null_invalid_values=False):
         """Formats a series to be a tuple (or list for Spark) of two floats."""
+        if null_invalid_values:
+            series = _coerce_latlong(series)
+
         if _is_dask_series(series):
             name = series.name
             meta = (series, tuple([float, float]))
@@ -852,6 +856,10 @@ def _get_index_invalid_age(series):
     return series.lt(0)
 
 
+def _get_index_invalid_latlong(series):
+    return ~series.apply(_is_valid_latlong_value)
+
+
 def _coerce_string(series, regex=None):
     if pd.api.types.is_object_dtype(series) or not pd.api.types.is_string_dtype(series):
         series = series.astype("string")
@@ -893,3 +901,10 @@ def _coerce_age(series, fractional=False):
     if invalid.any():
         series[invalid] = None
     return series
+
+
+def _coerce_latlong(series):
+    invalid = _get_index_invalid_latlong(series)
+    if invalid.any():
+        series[invalid] = None
+    return series
diff --git a/woodwork/utils.py b/woodwork/utils.py
index 82e1a7d..ec22d0e 100644
--- a/woodwork/utils.py
+++ b/woodwork/utils.py
@@ -221,16 +221,7 @@ def _reformat_to_latlong(latlong, is_spark=False):
     NaN like values are replaced with np.nan.
     """
     if isinstance(latlong, str):
-
-        nan_values = [x for x in ww.config.get_option("nan_values") if len(x)]
-        nan_values = "|".join(nan_values)
-
-        latlong = re.sub(nan_values, "None", latlong)
-
-        try:
-            latlong = ast.literal_eval(latlong)
-        except ValueError:
-            pass
+        latlong = _parse_latlong(latlong) or latlong
 
     if isinstance(latlong, (list, tuple)):
         if len(latlong) != 2:
@@ -248,9 +239,10 @@ def _reformat_to_latlong(latlong, is_spark=False):
                 f"LatLong values must be in decimal degrees. {latlong} does not have latitude or longitude values that can be converted to a float."
             )
 
+        latlong = (latitude, longitude)
         if is_spark:
-            return [latitude, longitude]
-        return (latitude, longitude)
+            latlong = list(latlong)
+        return latlong
 
     if _is_nan(latlong):
         return np.nan
@@ -304,11 +296,21 @@ def _is_valid_latlong_value(val, is_spark=False):
             return False
 
         latitude, longitude = val
-        return isinstance(latitude, float) and isinstance(longitude, float)
+        lat_null, long_null = map(pd.isnull, val)
+        is_valid = isinstance(latitude, float) or lat_null
+        is_valid &= isinstance(longitude, float) or long_null
+        return is_valid
 
     if isinstance(val, float):
         return np.isnan(val)
 
+    if isinstance(val, str):
+        val = _parse_latlong(val)
+        if val is None:
+            return False
+        else:
+            return _is_valid_latlong_value(val)
+
     if is_spark and val is None:
         return True
 
@@ -620,3 +622,15 @@ def _replace_nan_strings(df: pd.DataFrame) -> pd.DataFrame:
         df[col] = replaced_series
 
     return df
+
+
+def _parse_latlong(latlong):
+    nan_values = [x for x in ww.config.get_option("nan_values") if len(x)]
+    nan_values = "|".join(nan_values)
+
+    latlong = re.sub(nan_values, "None", latlong)
+
+    try:
+        return ast.literal_eval(latlong)
+    except ValueError:
+        pass


Test patch:
diff --git a/woodwork/tests/logical_types/test_logical_types.py b/woodwork/tests/logical_types/test_logical_types.py
index 22ca414..b7e4190 100644
--- a/woodwork/tests/logical_types/test_logical_types.py
+++ b/woodwork/tests/logical_types/test_logical_types.py
@@ -587,3 +587,36 @@ def test_null_invalid_age_nullable():
     expected = pd.DataFrame({"data": data})
     df.ww.init(logical_types=types, null_invalid_values=True)
     pd.testing.assert_frame_equal(df, expected)
+
+
+def test_null_invalid_latlong():
+    valid = [
+        (33.670914, -117.841501),
+        "(40.423599, -86.921162)",
+        (-45.031705, None),
+        (None, None),
+    ]
+    types = {"data": "LatLong"}
+    invalid = ["text", -6.7, object, None]
+    df = pd.DataFrame({"data": valid + invalid})
+
+    with pytest.raises(
+        TypeValidationError,
+        match="LatLong value is not properly formatted.",
+    ):
+        df.ww.init(logical_types=types, null_invalid_values=False)
+
+    nan = float("nan")
+    nulls = [nan] * len(invalid)
+    data = pd.Series(
+        [
+            (33.670914, -117.841501),
+            (40.423599, -86.921162),
+            (-45.031705, nan),
+            (nan, nan),
+            *nulls,
+        ]
+    )
+    expected = pd.DataFrame({"data": data})
+    df.ww.init(logical_types=types, null_invalid_values=True)
+    pd.testing.assert_frame_equal(df, expected)
diff --git a/woodwork/tests/utils/test_utils.py b/woodwork/tests/utils/test_utils.py
index 8fbc510..57c42d3 100644
--- a/woodwork/tests/utils/test_utils.py
+++ b/woodwork/tests/utils/test_utils.py
@@ -358,7 +358,7 @@ def test_is_nan():
         ([None, None], False),
         ("None", False),
         ([1.0, 2.0], False),
-        ((pd.NA, pd.NA), False),
+        ((pd.NA, pd.NA), True),
         (("a", 2.0), False),
         ((1.0, 2.0, 3.0), False),
         (None, False),
@@ -380,7 +380,7 @@ def test_is_valid_latlong_value(test_input, expected):
         (pd.NA, False),
         (2.0, False),
         ([2.0], False),
-        ([None, None], False),
+        ([None, None], True),
         ("None", False),
         ((1.0, 2.0), False),
         ((pd.NA, pd.NA), False),