Repository: dask/dask

Issue:
BUG: Failure in test_parquet.py as a result of pyarrow change from pandas.Timestamp to datetime.datetime
See results of nightly integration test:

https://circleci.com/gh/ursa-labs/crossbow/2846

It seems this unit test is reliable on the behavior of boxing `pyarrow.TimestampArray` scalar values to Python objects. We decided to return `datetime.datetime` from this API except for nanosecond resolution (see https://github.com/apache/arrow/commit/6f724576a98651ed32d915aa5367bd3b03bb4a86). It won't impact the conversions to pandas so it suggests this is some brittleness that is particular to this unit test

```
=================================== FAILURES ===================================
______________________ test_datasets_timeseries[pyarrow] _______________________

tmpdir = local('/tmp/pytest-of-root/pytest-0/test_datasets_timeseries_pyarr0')
engine = 'pyarrow'

    def test_datasets_timeseries(tmpdir, engine):
        tmp_path = str(tmpdir)
        df = dask.datasets.timeseries(
            start="2000-01-01", end="2000-01-10", freq="1d"
        ).persist()
        df.to_parquet(tmp_path, engine=engine)
    
        df2 = dd.read_parquet(tmp_path, engine=engine)
>       assert_eq(df, df2)

opt/conda/lib/python3.6/site-packages/dask/dataframe/io/tests/test_parquet.py:1809: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

a = Dask DataFrame Structure:
                  id    name        x        y
npartitions=9                                ...       ...     ...      ...      ...
2000-01-10       ...     ...      ...      ...
Dask Name: make-timeseries, 9 tasks
b = Dask DataFrame Structure:
                  id    name        x        y
npartitions=9                                ...-09       ...     ...      ...      ...
2000-01-09       ...     ...      ...      ...
Dask Name: read-parquet, 9 tasks
check_names = True, check_dtypes = True, check_divisions = True
check_index = True, kwargs = {}
at = <class 'pandas._libs.tslibs.timestamps.Timestamp'>
bt = <class 'datetime.datetime'>

    def assert_eq(
        a,
        b,
        check_names=True,
        check_dtypes=True,
        check_divisions=True,
        check_index=True,
        **kwargs
    ):
        if check_divisions:
            assert_divisions(a)
            assert_divisions(b)
            if hasattr(a, "divisions") and hasattr(b, "divisions"):
                at = type(np.asarray(a.divisions).tolist()[0])  # numpy to python
                bt = type(np.asarray(b.divisions).tolist()[0])  # scalar conversion
>               assert at == bt, (at, bt)
E               AssertionError: (<class 'pandas._libs.tslibs.timestamps.Timestamp'>, <class 'datetime.datetime'>)

opt/conda/lib/python3.6/site-packages/dask/dataframe/utils.py:773: AssertionError
= 1 failed, 104 passed, 138 skipped, 6 deselected, 3 xfailed in 22.36 seconds ==
Exited with code 1
```


Discussion:
It's nice to see this sort of thing coming from integration testing rather than from user bug reports.  Many thanks to Ursa folks for setting that up.

@rjzamora if you have a moment could you take a look at this?

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -217,10 +217,11 @@ def read_metadata(
                             if None in [cs_min, cs_max]:
                                 skip_cols.add(name)
                                 continue
+                            cs_vals = pd.Series([cs_min, cs_max])
                             d.update(
                                 {
-                                    "min": cs_min,
-                                    "max": cs_max,
+                                    "min": cs_vals[0],
+                                    "max": cs_vals[1],
                                     "null_count": column.statistics.null_count,
                                 }
                             )


Test patch: