Repository: dask/dask

Issue:
Dataframe datetime comparisons failing to work
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**: 
Comparing datetime columns with a defined date fails with this error: `TypeError('Invalid comparison between dtype=datetime64[ns] and Timestamp')`

**What you expected to happen**:
Similar to Pandas, I would expect to see the comparison actually go through and return with a proper boolean Series object. 

**Minimal Complete Verifiable Example**:

```python
import dask.dataframe as dd
import time
import pandas as pd
import operator
from datetime import datetime
from dask.distributed import Client

if __name__ == "__main__":
    start_time = time.time()
    client = Client()
    print(client)

    # Option 1
    # This works
    df = dd.read_csv("test.csv", sep="|").compute()
    time = pd.Timestamp("2012-07-29T00:15:20.336+00:00")
    df["date"] = dd.to_datetime(df["date"], unit="ns", utc=True)
    df = df[df["date"] > time]
    print(df.head())

    # Option 2
    # This also works
    df = dd.read_csv("test.csv", sep="|")
    time = pd.to_datetime("2012-07-29T00:15:20.336+00:00")
    df["date"] = dd.to_datetime(df["date"], unit="ns", utc=True)
    df = df[df["date"].map_partitions(operator.gt, time, meta=pd.Series([], dtype=object))]
    df.compute()
    print(df.head())

    # Option 3
    # This fails
    df = dd.read_csv("test.csv", sep="|")
    time = pd.to_datetime("2012-07-29T00:15:20.336+00:00")
    df["date"] = dd.to_datetime(df["date"], unit="ns", utc=True)
    df = df[df["date"] > time]
    df.compute()
    print(df.head())
```

Here is test.csv:
```
date|id
2012-07-29T00:15:11.336+00:00|1
2012-07-29T00:15:12.336+00:00|2
2012-07-29T00:15:13.336+00:00|3
2012-07-29T00:15:14.336+00:00|4
2012-07-29T00:15:15.336+00:00|5
2012-07-29T00:15:16.336+00:00|6
2012-07-29T00:15:17.336+00:00|7
2012-07-29T00:15:18.336+00:00|8
2012-07-29T00:15:19.336+00:00|9
2012-07-29T00:15:20.336+00:00|10
2012-07-29T00:15:21.336+00:00|11
2012-07-29T00:15:22.336+00:00|12
2012-07-29T00:15:23.336+00:00|13
2012-07-29T00:15:24.336+00:00|14
2012-07-29T00:15:25.336+00:00|15
2012-07-29T00:15:26.336+00:00|16
2012-07-29T00:15:27.336+00:00|17
2012-07-29T00:15:28.336+00:00|18
2012-07-29T00:15:29.336+00:00|19
```

**Anything else we need to know?**:
I am able to get around this problem right now with option 2 that I was able to find in another Github post: https://github.com/dask/dask/issues/5830. However, I would definitely prefer to have something like option 3 for the specific project I am working on. I can try to make a PR myself if the issue is solvable enough.

**Environment**:

- Dask version: 2021.07.0
- Python version: 3.9.6
- Operating System: Ubuntu 18.04 LTS
- Install method (conda, pip, source): conda



Discussion:
Thankd for raising this. The issue is with how dask tries to figure out what the output will look like. Dask does this by doing the computation on an empty object that has the same dtype and type as the original. This is what is failing in your example.

That being said I think this is actually a pandas bug. @jorisvandenbossche I can reproduce with this minimal example:

```python
import pandas as pd

s = pd.Series([], name="date", dtype="datetime64[ns]")
time = pd.Timestamp("2012-07-29T00:15:20.336+00:00")
s > time
```

```python-traceback
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py in _validate_comparison_value(self, other)
    539             try:
--> 540                 self._check_compatible_with(other)
    541             except (TypeError, IncompatibleFrequency) as err:

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py in _check_compatible_with(self, other, setitem)
    501             return
--> 502         self._assert_tzawareness_compat(other)
    503         if setitem:

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arrays/datetimes.py in _assert_tzawareness_compat(self, other)
    695             if other_tz is not None:
--> 696                 raise TypeError(
    697                     "Cannot compare tz-naive and tz-aware datetime-like objects."

TypeError: Cannot compare tz-naive and tz-aware datetime-like objects.

The above exception was the direct cause of the following exception:

InvalidComparison                         Traceback (most recent call last)
~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py in _cmp_method(self, other, op)
   1007         try:
-> 1008             other = self._validate_comparison_value(other)
   1009         except InvalidComparison:

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py in _validate_comparison_value(self, other)
    542                 # e.g. tzawareness mismatch
--> 543                 raise InvalidComparison(other) from err
    544 

InvalidComparison: 2012-07-29 00:15:20.336000+00:00

During handling of the above exception, another exception occurred:

TypeError                                 Traceback (most recent call last)
<ipython-input-6-0d850bbf5221> in <module>
----> 1 s > time

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/ops/common.py in new_method(self, other)
     67         other = item_from_zerodim(other)
     68 
---> 69         return method(self, other)
     70 
     71     return new_method

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arraylike.py in __gt__(self, other)
     46     @unpack_zerodim_and_defer("__gt__")
     47     def __gt__(self, other):
---> 48         return self._cmp_method(other, operator.gt)
     49 
     50     @unpack_zerodim_and_defer("__ge__")

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/series.py in _cmp_method(self, other, op)
   5496 
   5497         with np.errstate(all="ignore"):
-> 5498             res_values = ops.comparison_op(lvalues, rvalues, op)
   5499 
   5500         return self._construct_result(res_values, name=res_name)

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/ops/array_ops.py in comparison_op(left, right, op)
    268     ):
    269         # Call the method on lvalues
--> 270         res_values = op(lvalues, rvalues)
    271 
    272     elif is_scalar(rvalues) and isna(rvalues):

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/ops/common.py in new_method(self, other)
     67         other = item_from_zerodim(other)
     68 
---> 69         return method(self, other)
     70 
     71     return new_method

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arraylike.py in __gt__(self, other)
     46     @unpack_zerodim_and_defer("__gt__")
     47     def __gt__(self, other):
---> 48         return self._cmp_method(other, operator.gt)
     49 
     50     @unpack_zerodim_and_defer("__ge__")

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/arrays/datetimelike.py in _cmp_method(self, other, op)
   1008             other = self._validate_comparison_value(other)
   1009         except InvalidComparison:
-> 1010             return invalid_comparison(self, other, op)
   1011 
   1012         dtype = getattr(other, "dtype", None)

~/conda/envs/dask-upstream/lib/python3.8/site-packages/pandas/core/ops/invalid.py in invalid_comparison(left, right, op)
     32     else:
     33         typ = type(right).__name__
---> 34         raise TypeError(f"Invalid comparison between dtype={left.dtype} and {typ}")
     35     return res_values
     36 

TypeError: Invalid comparison between dtype=datetime64[ns] and Timestamp
```


This is intentional behaviour of pandas to disallow comparing timezone-aware and timezone-naive data (for the timezone-naive local data, there is no way to unambiguously know in what timezone it should be interpreted). 

But so the problem here is that dask doesn't know that the column is a timezone-aware column (the `meta` has the wrong dtype), and thus the dummy operation on the `meta` goes awry.

Using the example of above:

```
In [39]: df = dd.read_csv("test.csv", sep="|")

In [40]: df["date"] = dd.to_datetime(df["date"], unit="ns", utc=True)

In [41]: df
Out[41]: 
Dask DataFrame Structure:
                         date     id
npartitions=1                       
               datetime64[ns]  int64
                          ...    ...
Dask Name: assign, 4 tasks

In [42]: df._meta.dtypes
Out[42]: 
date    datetime64[ns]
id               int64
dtype: object

In [43]: df.compute().dtypes
Out[43]: 
date    datetime64[ns, UTC]
id                    int64
dtype: object
```

So you see that `meta` has a different dtype compared the materialized data (and on the actual data, the comparison operation works fine).

It might be possible to fix this in the `meta` handling of `to_datetime`:

https://github.com/dask/dask/blob/c59a509fad5bb5a7fbe536eb2d593803b8212a22/dask/dataframe/core.py#L6802-L6811

For this case, you could check the `utc` keyword in kwargs, and if `utc=True` properly make the empty meta a tz-aware dtype. 

This won't solve all cases though. As it could also be the data that you are parsing that has strings with tz info, which will also result in a tz-aware dtype. But to know that, you would need to actually parse part of the data.

Thank you so much for the explanation Joris! It seems like we should be able to fix this in dask as you suggest. 

Resolving patch:
diff --git a/dask/dataframe/core.py b/dask/dataframe/core.py
--- a/dask/dataframe/core.py
+++ b/dask/dataframe/core.py
@@ -6801,12 +6801,13 @@ def maybe_shift_divisions(df, periods, freq):
 
 @wraps(pd.to_datetime)
 def to_datetime(arg, meta=None, **kwargs):
+    tz_kwarg = {"tz": "utc"} if kwargs.get("utc") else {}
     if meta is None:
         if isinstance(arg, Index):
-            meta = pd.DatetimeIndex([])
+            meta = pd.DatetimeIndex([], **tz_kwarg)
             meta.name = arg.name
         else:
-            meta = pd.Series([pd.Timestamp("2000")])
+            meta = pd.Series([pd.Timestamp("2000", **tz_kwarg)])
             meta.index = meta.index.astype(arg.index.dtype)
             meta.index.name = arg.index.name
     return map_partitions(pd.to_datetime, arg, meta=meta, **kwargs)


Test patch:
diff --git a/dask/dataframe/tests/test_dataframe.py b/dask/dataframe/tests/test_dataframe.py
--- a/dask/dataframe/tests/test_dataframe.py
+++ b/dask/dataframe/tests/test_dataframe.py
@@ -3993,6 +3993,10 @@ def test_to_datetime():
         dd.to_datetime(ds.index, infer_datetime_format=True),
         check_divisions=False,
     )
+    assert_eq(
+        pd.to_datetime(s, utc=True),
+        dd.to_datetime(ds, utc=True),
+    )
 
 
 def test_to_timedelta():