Repository: modin-project/modin

Issue:
call to_pydatetime get datetime64[ns] dtype
### System information
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: Win10 x64
- **Modin version** (`modin.__version__`):   modin- 0.11.3 or modin-0.14.0+27.g830e8dc
- **Python version**: 3.8.8
- **Code we can use to reproduce**:


<!--
You can obtain the Modin version with

python -c "import modin; print(modin.__version__)"
-->

### Describe the problem
<!-- Describe the problem clearly here. -->
call a datetime column dt attribute to_pydatetime should get python datetime but  return datetime64[ns] dtype data.
to_pydatetime only worked on DateTimeIndex

### Source code / logs
<!-- Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached. Try to provide a reproducible test case that is the bare minimum necessary to generate the problem. -->

```
import modin.pandas as pd

df = pd.DataFrame(pd.date_range('2022-01-01', '2022-01-10', freq='1h'), columns=['date'])

pydates = df.date.dt.to_pydatetime()

pydates.dtype
```


the result gives dtype('<M8[ns]') instead of dtype('O').

while DatetimeIndex worked as expected
```dateindx = pd.date_range('2022-01-01', '2022-01-10', freq='1h')

pydates2 = dateindx.to_pydatetime()

pydates2.dtype

```
The result is dtype('O')



Discussion:
@oraix thank you for reporting this bug! I can also reproduce it with your first snippet on Modin version 0.14.0+27.g830e8dce. If there's a quick fix, I'll make it. Otherwise, the Modin contributors will try to fix the bug soon.

Resolving patch:
diff --git a/modin/core/dataframe/algebra/default2pandas/default.py b/modin/core/dataframe/algebra/default2pandas/default.py
--- a/modin/core/dataframe/algebra/default2pandas/default.py
+++ b/modin/core/dataframe/algebra/default2pandas/default.py
@@ -85,10 +85,20 @@ def applyier(df, *args, **kwargs):
                 and func != "to_numpy"
                 and func != pandas.DataFrame.to_numpy
             ):
+                # When applying a DatetimeProperties function, if we don't
+                # specify the dtype for the DataFrame, the frame might get the
+                # wrong dtype, e.g. for to_pydatetime in
+                # https://github.com/modin-project/modin/issues/4436
+                astype_kwargs = {}
+                dtype = getattr(result, "dtype", None)
+                if dtype and isinstance(
+                    df, pandas.core.indexes.accessors.DatetimeProperties
+                ):
+                    astype_kwargs["dtype"] = dtype
                 result = (
-                    pandas.DataFrame(result)
+                    pandas.DataFrame(result, **astype_kwargs)
                     if is_list_like(result)
-                    else pandas.DataFrame([result])
+                    else pandas.DataFrame([result], **astype_kwargs)
                 )
             if isinstance(result, pandas.Series):
                 if result.name is None:
diff --git a/modin/core/storage_formats/pandas/query_compiler.py b/modin/core/storage_formats/pandas/query_compiler.py
--- a/modin/core/storage_formats/pandas/query_compiler.py
+++ b/modin/core/storage_formats/pandas/query_compiler.py
@@ -177,9 +177,10 @@ def _dt_func_map(func_name):
     def dt_op_builder(df, *args, **kwargs):
         """Apply specified function against ``dt`` accessor of the passed frame."""
         dt_s = df.squeeze(axis=1).dt
-        return pandas.DataFrame(
-            getattr(pandas.Series.dt, func_name)(dt_s, *args, **kwargs)
-        )
+        dt_func_result = getattr(pandas.Series.dt, func_name)(dt_s, *args, **kwargs)
+        # If we don't specify the dtype for the frame, the frame might get the
+        # wrong dtype, e.g. for to_pydatetime in https://github.com/modin-project/modin/issues/4436
+        return pandas.DataFrame(dt_func_result, dtype=dt_func_result.dtype)
 
     return dt_op_builder
 


Test patch:
diff --git a/modin/pandas/test/test_series.py b/modin/pandas/test/test_series.py
--- a/modin/pandas/test/test_series.py
+++ b/modin/pandas/test/test_series.py
@@ -1620,8 +1620,14 @@ def test_dtype(data):
     df_equals(modin_series.dtype, pandas_series.dtypes)
 
 
-def test_dt():
-    data = pd.date_range("2016-12-31", periods=128, freq="D", tz="Europe/Berlin")
+# Bug https://github.com/modin-project/modin/issues/4436 in
+# Series.dt.to_pydatetime is only reproducible when the date range out of which
+# the frame is created has timezone None, so that its dtype is datetime64[ns]
+# as opposed to, e.g. datetime64[ns, Europe/Berlin]. To reproduce that bug, we
+# use timezones None and Europe/Berlin.
+@pytest.mark.parametrize("timezone", [None, "Europe/Berlin"])
+def test_dt(timezone):
+    data = pd.date_range("2016-12-31", periods=128, freq="D", tz=timezone)
     modin_series = pd.Series(data)
     pandas_series = pandas.Series(data)
 
@@ -1661,10 +1667,11 @@ def test_dt():
         modin_series.dt.tz_localize(None),
         pandas_series.dt.tz_localize(None),
     )
-    df_equals(
-        modin_series.dt.tz_convert(tz="Europe/Berlin"),
-        pandas_series.dt.tz_convert(tz="Europe/Berlin"),
-    )
+    if timezone:
+        df_equals(
+            modin_series.dt.tz_convert(tz="Europe/Berlin"),
+            pandas_series.dt.tz_convert(tz="Europe/Berlin"),
+        )
 
     df_equals(modin_series.dt.normalize(), pandas_series.dt.normalize())
     df_equals(