Repository: pandas-dev/pandas
Pull request: TST: added test for ea dtypes conversion to datetimetzdtype (GH37553)
- [x] closes #37553
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing.html#code-standards) for how to run them

Provided tests for type cast `ExtensionDtypes` -> `DatetimeTZDtype`.

Tests and linter pass locally.

Problem statement:
BUG: Improve error message when casting ExtensionDtype to datetime
We support casting `int` and `float` dtypes to `DatetimeTZDtype`:
``` python
In [11]: s = pd.Series(np.random.randint(10, size=3), dtype="int64") 
    ...: s.astype(pd.DatetimeTZDtype(tz="US/Pacific"))                                                  
Out[11]: 
0   1969-12-31 16:00:00.000000004-08:00
1   1969-12-31 16:00:00.000000007-08:00
2   1969-12-31 16:00:00.000000001-08:00
dtype: datetime64[ns, US/Pacific]
```
We should be able to do this with the corresponding extension dtypes (`Int`/`Float`). Currently this errors:
``` python
In [12]: s = pd.Series(np.random.randint(10, size=5), dtype="Int64") 
    ...: s.astype(pd.DatetimeTZDtype(tz="US/Pacific"))                                                  
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-12-469a4b536444> in <module>
      1 s = pd.Series(np.random.randint(10, size=5), dtype="Int64")
----> 2 s.astype(pd.DatetimeTZDtype(tz="US/Pacific"))

/workspaces/pandas-arw2019/pandas/core/generic.py in astype(self, dtype, copy, errors)
   5797         else:
   5798             # else, only a single dtype is given
-> 5799             new_data = self._mgr.astype(dtype=dtype, copy=copy, errors=errors)
   5800             return self._constructor(new_data).__finalize__(self, method="astype")
   5801 

/workspaces/pandas-arw2019/pandas/core/internals/managers.py in astype(self, dtype, copy, errors)
    620         self, dtype, copy: bool = False, errors: str = "raise"
    621     ) -> "BlockManager":
--> 622         return self.apply("astype", dtype=dtype, copy=copy, errors=errors)
    623 
    624     def convert(

/workspaces/pandas-arw2019/pandas/core/internals/managers.py in apply(self, f, align_keys, ignore_failures, **kwargs)
    422                     applied = b.apply(f, **kwargs)
    423                 else:
--> 424                     applied = getattr(b, f)(**kwargs)
    425             except (TypeError, NotImplementedError):
    426                 if not ignore_failures:

/workspaces/pandas-arw2019/pandas/core/internals/blocks.py in astype(self, dtype, copy, errors)
    608         if self.is_extension:
    609             try:
--> 610                 values = self.values.astype(dtype)
    611             except (ValueError, TypeError):
    612                 if errors == "ignore":

/workspaces/pandas-arw2019/pandas/core/arrays/integer.py in astype(self, dtype, copy)
    471             na_value = lib.no_default
    472 
--> 473         return self.to_numpy(dtype=dtype, na_value=na_value, copy=False)
    474 
    475     def _values_for_argsort(self) -> np.ndarray:

/workspaces/pandas-arw2019/pandas/core/arrays/masked.py in to_numpy(self, dtype, copy, na_value)
    222             data[self._mask] = na_value
    223         else:
--> 224             data = self._data.astype(dtype, copy=copy)
    225         return data
    226 

TypeError: data type not understood
```
We should get this to work for both `ExtensionArray` and the masked versions.

Hints from issue discussion:
Does the desirability of this change if #37179 changes how the non-EA case works?
> Does the desirability of this change if #37179 changes how the non-EA case works?

Right. As of #37179 this shouldn't be supported.

Minor but once #37179 is in I'd be keen to align the error message between EA/non-EA cases. Instead of 
```
TypeError: data type not understood    # current EA error message
```
I'd like to see 
```
TypeError: dtype Int64 cannot be converted to datetime64[ns]  # desired EA error message
```
This looks to work on master now (without the error which makes sense to me). Could use a test

```
In [1]: In [12]: s = pd.Series(np.random.randint(10, size=5), dtype="Int64")
   ...:     ...: s.astype(pd.DatetimeTZDtype(tz="US/Pacific"))
Out[1]:
0   1969-12-31 16:00:00.000000006-08:00
1   1969-12-31 16:00:00.000000006-08:00
2   1969-12-31 16:00:00.000000002-08:00
3   1969-12-31 16:00:00.000000009-08:00
4   1969-12-31 16:00:00.000000002-08:00
dtype: datetime64[ns, US/Pacific]
```

Commit message: TST: added test for ea dtypes conversion to datetimetzdtype

Patch to review:
diff --git a/pandas/tests/frame/methods/test_astype.py b/pandas/tests/frame/methods/test_astype.py
index 398c66584390b..c30d5aaf02352 100644
--- a/pandas/tests/frame/methods/test_astype.py
+++ b/pandas/tests/frame/methods/test_astype.py
@@ -691,6 +691,38 @@ def test_astype_noncontiguous(self, index_slice):
         expected = df.iloc[index_slice]
         tm.assert_frame_equal(result, expected, check_dtype=False)
 
+    @pytest.mark.parametrize(
+        "dtype",
+        tm.ALL_INT_EA_DTYPES + tm.FLOAT_EA_DTYPES,
+    )
+    def test_astype_ea_to_datetimetzdtype(self, dtype):
+        # GH37553
+        result = Series([4, 0, 9], dtype=dtype).astype(DatetimeTZDtype(tz="US/Pacific"))
+        expected = Series(
+            {
+                0: Timestamp("1969-12-31 16:00:00.000000004-08:00", tz="US/Pacific"),
+                1: Timestamp("1969-12-31 16:00:00.000000000-08:00", tz="US/Pacific"),
+                2: Timestamp("1969-12-31 16:00:00.000000009-08:00", tz="US/Pacific"),
+            }
+        )
+
+        if dtype in tm.FLOAT_EA_DTYPES:
+            expected = Series(
+                {
+                    0: Timestamp(
+                        "1970-01-01 00:00:00.000000004-08:00", tz="US/Pacific"
+                    ),
+                    1: Timestamp(
+                        "1970-01-01 00:00:00.000000000-08:00", tz="US/Pacific"
+                    ),
+                    2: Timestamp(
+                        "1970-01-01 00:00:00.000000009-08:00", tz="US/Pacific"
+                    ),
+                }
+            )
+
+        tm.assert_equal(result, expected)
+
 
 class TestAstypeCategorical:
     def test_astype_from_categorical3(self):


Code review comments:
diff_hunk: @@ -691,6 +691,38 @@ def test_astype_noncontiguous(self, index_slice):
         expected = df.iloc[index_slice]
         tm.assert_frame_equal(result, expected, check_dtype=False)
 
+    @pytest.mark.parametrize(
+        "dtype",
+        tm.ALL_INT_EA_DTYPES + tm.FLOAT_EA_DTYPES,
+    )
+    def test_astype_ea_to_datetimetzdtype(self, dtype):
+        # GH37553
+        result = Series([4, 0, 9], dtype=dtype).astype(DatetimeTZDtype(tz="US/Pacific"))
+        expected = Series(
+            {
+                0: Timestamp("1969-12-31 16:00:00.000000004-08:00", tz="US/Pacific"),
+                1: Timestamp("1969-12-31 16:00:00.000000000-08:00", tz="US/Pacific"),
+                2: Timestamp("1969-12-31 16:00:00.000000009-08:00", tz="US/Pacific"),
+            }
+        )
+
+        if dtype in tm.FLOAT_EA_DTYPES:
+            expected = Series(
+                {
+                    0: Timestamp(
+                        "1970-01-01 00:00:00.000000004-08:00", tz="US/Pacific"
+                    ),
+                    1: Timestamp(
+                        "1970-01-01 00:00:00.000000000-08:00", tz="US/Pacific"
+                    ),
+                    2: Timestamp(
+                        "1970-01-01 00:00:00.000000009-08:00", tz="US/Pacific"
+                    ),
+                }
+            )
+
+        tm.assert_equal(result, expected)
path: pandas/tests/frame/methods/test_astype.py
text: @user1:
If just comparing `Series`, could use `tm.assert_series_equal`. `assert_equal` is useful when doing something like parameterizing over `DataFrame` and `Series` (like with the `frame_or_series` fixture)