Repository: alteryx/woodwork
Pull request #1512 (2022-08-31 15:00:34)

Change pivot point of 2-digit year in datetime inference
Choose a different pivot point for 2-digit year rather than the default that pandas datetime uses.

Chose `current year + 10` as the pivot date, but can easily be changed to another

codecov[bot]: # [Codecov](https://codecov.io/gh/alteryx/woodwork/pull/1512?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) Report
> Merging [#1512](https://codecov.io/gh/alteryx/woodwork/pull/1512?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (ecfa056) into [main](https://codecov.io/gh/alteryx/woodwork/commit/6121dd9c7dbfceacf3cfc652e3a1fea80feb0539?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) (6121dd9) will **decrease** coverage by `80.48%`.
> The diff coverage is `1.68%`.

> :exclamation: Current head ecfa056 differs from pull request most recent head 850bcb7. Consider uploading reports for the commit 850bcb7 to get more accurate results

```diff
@@             Coverage Diff             @@
##             main    #1512       +/-   ##
===========================================
- Coverage   98.68%   18.20%   -80.49%     
===========================================
  Files          95       95               
  Lines       10743    10861      +118     
===========================================
- Hits        10602     1977     -8625     
- Misses        141     8884     +8743     
```


| [Impacted Files](https://codecov.io/gh/alteryx/woodwork/pull/1512?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | Coverage Δ | |
|---|---|---|
| [woodwork/logical\_types.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvbG9naWNhbF90eXBlcy5weQ==) | `46.24% <0.00%> (-53.76%)` | :arrow_down: |
| [woodwork/table\_accessor.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGFibGVfYWNjZXNzb3IucHk=) | `51.68% <ø> (-47.87%)` | :arrow_down: |
| [woodwork/tests/accessor/test\_statistics.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvYWNjZXNzb3IvdGVzdF9zdGF0aXN0aWNzLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/logical\_types/test\_logical\_types.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvbG9naWNhbF90eXBlcy90ZXN0X2xvZ2ljYWxfdHlwZXMucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/statistics\_utils/\_get\_dependence\_dict.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvc3RhdGlzdGljc191dGlscy9fZ2V0X2RlcGVuZGVuY2VfZGljdC5weQ==) | `12.84% <8.00%> (-87.16%)` | :arrow_down: |
| [woodwork/tests/test\_config.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdGVzdF9jb25maWcucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/test\_version.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdGVzdF92ZXJzaW9uLnB5) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_utils.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF91dGlscy5weQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/utils/test\_concat.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdXRpbHMvdGVzdF9jb25jYXQucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [woodwork/tests/type\_system/conftest.py](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx#diff-d29vZHdvcmsvdGVzdHMvdHlwZV9zeXN0ZW0vY29uZnRlc3QucHk=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [61 more](https://codecov.io/gh/alteryx/woodwork/pull/1512/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx) | |

Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=alteryx)


Patch:
diff --git a/docs/source/release_notes.rst b/docs/source/release_notes.rst
index 621f7ed..3d01077 100644
--- a/docs/source/release_notes.rst
+++ b/docs/source/release_notes.rst
@@ -7,6 +7,7 @@ Future Release
 ==============
     * Enhancements
     * Fixes
+        * Fix datetime pivot point to be set at current year + 10 rather than the default for two-digit years when ``datetime_format`` provided (:pr:`1512`)
     * Changes
         * Added ``ignore_columns`` as an argument when initializing a dataframe (:pr:`1504`)
     * Documentation Changes
@@ -14,7 +15,7 @@ Future Release
         * Add kickoff for create conda forge pull request from release (:pr:`1515`)
 
     Thanks to the following people for contributing to this release:
-    :user:`gsheni`, :user:`ParthivNaresh`
+    :user:`gsheni`, :user:`ParthivNaresh`, :user:`bchen1116`
 
 v0.18.0 August 31, 2022
 =======================
diff --git a/woodwork/logical_types.py b/woodwork/logical_types.py
index aed47c0..4c1442f 100644
--- a/woodwork/logical_types.py
+++ b/woodwork/logical_types.py
@@ -1,5 +1,6 @@
 import re
 import warnings
+from datetime import datetime
 from typing import Optional
 
 import numpy as np
@@ -306,6 +307,13 @@ class Datetime(LogicalType):
 
     def transform(self, series, null_invalid_values=False):
         """Converts the series data to a formatted datetime. Datetime format will be inferred if datetime_format is None."""
+
+        def _year_filter(date):
+            """Applies a filter to the years to ensure that the pivot point isn't too far forward."""
+            if date.year > datetime.today().year + 10:
+                date = date.replace(year=date.year - 100)
+            return date
+
         new_dtype = self._get_valid_dtype(type(series))
         series = self._remove_timezone(series)
         series_dtype = str(series.dtype)
@@ -356,6 +364,11 @@ class Datetime(LogicalType):
                     )
 
         series = self._remove_timezone(series)
+        if self.datetime_format is not None and "%y" in self.datetime_format:
+            if _is_spark_series(series):
+                series = series.transform(_year_filter)
+            else:
+                series = series.apply(_year_filter)
         return super().transform(series)
 
 


Test patch:
diff --git a/woodwork/tests/logical_types/test_logical_types.py b/woodwork/tests/logical_types/test_logical_types.py
index ed6dd42..cc418a5 100644
--- a/woodwork/tests/logical_types/test_logical_types.py
+++ b/woodwork/tests/logical_types/test_logical_types.py
@@ -1,4 +1,5 @@
 import re
+from datetime import datetime
 
 import numpy as np
 import pandas as pd
@@ -744,3 +745,175 @@ def test_replace_nans_same_types():
 
     assert new_series.dtype == "float"
     pd.testing.assert_series_equal(new_series, pd.Series([1.0, 3.0, 5.0, -6.0, np.nan]))
+
+
+def get_expected_dates(dates):
+    expected = []
+    for d in dates:
+        if d is not None:
+            year = int(re.findall(r"\d+", d)[0])  # gets the year
+            if year > datetime.today().year + 10:
+                year -= 100
+            if year <= datetime.today().year - 90:
+                year += 100
+            expected.append("{}-01-01".format(year))
+        else:
+            expected.append(d)
+    return expected
+
+
+@pytest.mark.parametrize("delim", ["/", "-", "."])
+@pytest.mark.parametrize("dtype", ["string", "object"])
+def test_datetime_pivot_point(dtype, delim):
+    if dtype == "string" and delim != "/":
+        pytest.skip("skipping because we don't want to overtest")
+    dates = [
+        "01/01/24",
+        "01/01/30",
+        "01/01/32",
+        "01/01/36",
+        "01/01/52",
+        "01/01/56",
+        "01/01/60",
+        "01/01/72",
+        "01/01/76",
+        "01/01/80",
+        None,
+        "01/01/88",
+    ]
+    datetime_str = "%m/%d/%y"
+    if delim != "/":
+        dates = [s.replace("/", delim) if s is not None else s for s in dates]
+        datetime_str = datetime_str.replace("/", delim)
+    expected_values = [
+        "2024-01-01",
+        "2030-01-01",
+        "2032-01-01",
+        "2036-01-01",
+        "2052-01-01",
+        "2056-01-01",
+        "1960-01-01",
+        "1972-01-01",
+        "1976-01-01",
+        "1980-01-01",
+        None,
+        "1988-01-01",
+    ]
+    expected_values = get_expected_dates(expected_values)
+    df = pd.DataFrame({"dates": dates}, dtype=dtype)
+    df_expected = pd.DataFrame({"dates": expected_values}, dtype="datetime64[ns]")
+    df.ww.init(logical_types={"dates": Datetime(datetime_format=datetime_str)})
+    pd.testing.assert_frame_equal(df, df_expected)
+
+
+@pytest.mark.parametrize("delim", ["/", "-", ".", ""])
+def test_datetime_pivot_point_should_not_apply(delim):
+    dates = [
+        "01/01/1924",
+        "01/01/1928",
+        "01/01/1960",
+        "01/01/1964",
+        "01/01/1968",
+        "01/01/1972",
+        "01/01/2076",
+        "01/01/2088",
+    ]
+    datetime_str = "%m/%d/%Y"
+    if delim == "-":
+        dates = [s.replace("/", delim) for s in dates]
+        datetime_str = datetime_str.replace("/", delim)
+    expected_values = [
+        "1924-01-01",
+        "1928-01-01",
+        "1960-01-01",
+        "1964-01-01",
+        "1968-01-01",
+        "1972-01-01",
+        "2076-01-01",
+        "2088-01-01",
+    ]
+    df = pd.DataFrame({"dates": dates})
+    df_expected = pd.DataFrame({"dates": expected_values}, dtype="datetime64[ns]")
+    df.ww.init(logical_types={"dates": Datetime(datetime_format=datetime_str)})
+    pd.testing.assert_frame_equal(df, df_expected)
+
+
+@pytest.mark.parametrize("type", ["pyspark", "dask"])
+def test_pyspark_dask_series(type):
+    dates = [
+        "01/01/24",
+        "01/01/28",
+        "01/01/30",
+        "01/01/32",
+        "01/01/36",
+        "01/01/40",
+        "01/01/72",
+        None,
+        "01/01/88",
+    ]
+    datetime_str = "%m/%d/%y"
+    expected_values = [
+        "2024-01-01",
+        "2028-01-01",
+        "2030-01-01",
+        "1932-01-01",
+        "1936-01-01",
+        "1940-01-01",
+        "1972-01-01",
+        None,
+        "1988-01-01",
+    ]
+    expected_values = get_expected_dates(expected_values)
+    df = pd.DataFrame({"dates": dates})
+    if type == "pyspark":
+        ps = pytest.importorskip(
+            "pyspark.pandas",
+            reason="Pyspark pandas not installed, skipping",
+        )
+        df = ps.from_pandas(df)
+    else:
+        dd = pytest.importorskip(
+            "dask.dataframe",
+            reason="Dask not installed, skipping",
+        )
+        df = dd.from_pandas(df, npartitions=2)
+    df.ww.init(logical_types={"dates": Datetime(datetime_format=datetime_str)})
+    df_expected = pd.DataFrame({"dates": expected_values}, dtype="datetime64[ns]")
+    df = to_pandas(df)
+    df.sort_index(inplace=True)
+    pd.testing.assert_frame_equal(df, df_expected)
+
+
+def test_datetime_pivot_point_no_format_provided():
+    dates = [
+        "01/01/24",
+        "01/01/30",
+        "01/01/32",
+        "01/01/36",
+        "01/01/52",
+        "01/01/56",
+        "01/01/60",
+        "01/01/72",
+        "01/01/76",
+        "01/01/80",
+        None,
+        "01/01/88",
+    ]
+    expected_values = [
+        "2024-01-01",
+        "2030-01-01",
+        "2032-01-01",
+        "2036-01-01",
+        "2052-01-01",
+        "2056-01-01",
+        "2060-01-01",
+        "1972-01-01",
+        "1976-01-01",
+        "1980-01-01",
+        None,
+        "1988-01-01",
+    ]
+    df = pd.DataFrame({"dates": dates})
+    df_expected = pd.DataFrame({"dates": expected_values}, dtype="datetime64[ns]")
+    df.ww.init(logical_types={"dates": Datetime})
+    pd.testing.assert_frame_equal(df, df_expected)