Repository: dask/dask

Issue:
Cannot compare tz-naive and tz-aware timestamps on concat
**What happened**:

When concatenating two dask dataframes with indices dype=datetime64[ns, UTC], I get a `TypeError: Cannot compare tz-naive and tz-aware timestamps`. One of the the dask dataframe was created with `dd.from_pandas` and the other with `dd.read_parquet`

**What you expected to happen**:

An happy concatenation ;-)

**Minimal Complete Verifiable Example**:

```python
import dask.dataframe as dd
import pandas

filename = "outfile.parq"

# create a DF with tz=UTC and write to parquet file
df = pandas.DataFrame(
    columns=["A"], index=pandas.date_range("2018", "2019", freq="MS", tz="UTC"), data=1.0
)
ddf = dd.from_pandas(df, npartitions=1)
ddf.to_parquet(filename)

# read back this dask df into rddf
rddf = dd.read_parquet(filename)

# print indices
print(ddf.index)
print(rddf.index)


# attempt concatenation of dask df from parquet and new dask df
nddf = dd.concat([rddf, ddf])
# this raises the following traceback
# Traceback (most recent call last):
#   File ".../tst_parquet_tz.py", line 22, in <module>
#     nddf = dd.concat([rddf, ddf])
#   File "...\site-packages\dask\dataframe\multi.py", line 1108, in concat
#     if all(
#   File "...\site-packages\dask\dataframe\multi.py", line 1109, in <genexpr>
#     dfs[i].divisions[-1] < dfs[i + 1].divisions[0]
#   File "pandas\_libs\tslibs\timestamps.pyx", line 281, in pandas._libs.tslibs.timestamps._Timestamp.__richcmp__
#   File "pandas\_libs\tslibs\timestamps.pyx", line 295, in pandas._libs.tslibs.timestamps._Timestamp._assert_tzawareness_compat
# TypeError: Cannot compare tz-naive and tz-aware timestamps
```

**Anything else we need to know?**:

When printing both indices, we see that the dtype of both indices are `datetime64[ns, UTC]` yet the representation of the index coming from read-parquet shows tz-naive dates.
```
Dask Index Structure:
npartitions=1
2018-01-01 00:00:00+00:00    datetime64[ns, UTC]
2019-01-01 00:00:00+00:00                    ...
dtype: datetime64[ns, UTC]
Dask Name: from_pandas, 2 tasks
Dask Index Structure:
npartitions=1
2018-01-01    datetime64[ns, UTC]
2019-01-01                    ...
dtype: datetime64[ns, UTC]
Dask Name: read-parquet, 2 tasks
```

**Environment**:

Dask version: 2.30.0
Python version: 3.8
Operating System: win10
Install method (conda, pip, source): pip


Discussion:
Thanks for raising this. I was able to reproduce using fastparquet, but found that it works as expected when using pyarrow (`engine="pyarrow"`). 

It seems fastparquet isn't storing the timezone information properly in the metadata file, but it isn't clear to me whether or not this is a bug in the dask code base or if it lives in fastparquet. Either way there is an issue with `divisions` (which tries to know the edges of the partitions). As a workaround you can do `rddf = rddf.clear_divisions()` or you can use `pyarrow`. 

Ping @martindurant for fastparquet expertise.
Will look...

I'll mention that parquet does not store time zones, this information only goes into the pandas metadata (i.e., only for data that came from pandas originally), so it is quite likely that this hasn't been implemented in fastparquet at all.
More about fastparquet and timezones in https://github.com/dask/fastparquet/issues/532

If I can get some guidance on this topic, I can try a PR. I have no view on what is happening before/after neither on the divisions/partitions... So maybe a bit though for me.
Actually, it may be worth trying again versus fastparquet master, because previously failure to set the timezone was being ignored, and that particular failure should no longer happen. However, it *might* need more work to apply the same thing to the min/max values.
I have checked with master for dask and fastparquet and the issue is still there.


Resolving patch:
diff --git a/dask/dataframe/io/parquet/fastparquet.py b/dask/dataframe/io/parquet/fastparquet.py
--- a/dask/dataframe/io/parquet/fastparquet.py
+++ b/dask/dataframe/io/parquet/fastparquet.py
@@ -322,8 +322,9 @@ def read_metadata(
                             skip_cols.add(col)
                             continue
                         if isinstance(cs_min, np.datetime64):
-                            cs_min = pd.Timestamp(cs_min)
-                            cs_max = pd.Timestamp(cs_max)
+                            tz = getattr(dtypes[col], "tz", None)
+                            cs_min = pd.Timestamp(cs_min, tz=tz)
+                            cs_max = pd.Timestamp(cs_max, tz=tz)
                         d.update(
                             {
                                 "min": cs_min,


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -873,6 +873,9 @@ def test_read_parquet_custom_columns(tmpdir, engine):
         ),
         (pd.DataFrame({"x": [3, 2, 1]}).astype("M8[us]"), {}, {}),
         (pd.DataFrame({"x": [3, 2, 1]}).astype("M8[ms]"), {}, {}),
+        (pd.DataFrame({"x": [3000, 2000, 1000]}).astype("datetime64[ns]"), {}, {}),
+        (pd.DataFrame({"x": [3000, 2000, 1000]}).astype("datetime64[ns, UTC]"), {}, {}),
+        (pd.DataFrame({"x": [3000, 2000, 1000]}).astype("datetime64[ns, CET]"), {}, {}),
         (pd.DataFrame({"x": [3, 2, 1]}).astype("uint16"), {}, {}),
         (pd.DataFrame({"x": [3, 2, 1]}).astype("float32"), {}, {}),
         (pd.DataFrame({"x": [3, 1, 2]}, index=[3, 2, 1]), {}, {}),