Repository: dask/dask

Issue:
exception when reading an item with filters
**What happened**:

Using dask via pystore, an exception is raised when reading a parquet file with filters. The issue appears to be some statistics with None values.
```
Traceback (most recent call last):
  File ".../tst_bug_nonetype.py", line 47, in <module>
    collection.item("test", filters=[("year", "==", 2020)]).to_pandas()
  File "...\lib\site-packages\pystore\collection.py", line 77, in item
    return Item(item, self.datastore, self.collection,
  File "...\lib\site-packages\pystore\item.py", line 60, in __init__
    self.data = dd.read_parquet(
  File "...\lib\site-packages\dask\dataframe\io\parquet\core.py", line 243, in read_parquet
    parts, divisions, index, index_in_columns = process_statistics(
  File "...\lib\site-packages\dask\dataframe\io\parquet\core.py", line 712, in process_statistics
    out = sorted_columns(statistics)
  File "...\lib\site-packages\dask\dataframe\io\parquet\core.py", line 605, in sorted_columns
    assert divisions == sorted(divisions)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
```

**What you expected to happen**:

No exception

**Minimal Complete Verifiable Example**:

```python
#!/usr/bin/env python
# -*- coding: utf-8 -*-

import pandas
import pystore
from numpy import nan
from pandas import Timestamp

pystore.set_path("./pystore")
collection = pystore.store("my-store").collection("my-collection")

df = pandas.DataFrame.from_dict(
    {
        "time": {
            0: Timestamp("2019-12-31 21:00:00"),
            1: Timestamp("2019-12-31 22:00:00"),
            2: Timestamp("2019-12-31 23:00:00"),
            3: Timestamp("2020-11-25 19:00:00"),
            4: Timestamp("2020-11-26 02:00:00"),
            5: Timestamp("2020-11-26 03:00:00"),
            6: Timestamp("2020-11-26 04:00:00"),
        },
        "A": {
            0: 351.0,
            1: 355.0,
            2: 358.0,
            3: 266.0,
            4: 266.0,
            5: 268.0,
            6: nan,
        },
        "B": {
            0: 2063.0,
            1: 2051.0,
            2: 1749.0,
            3: 4281.0,
            4: 3526.0,
            5: 3462.0,
            6: nan,
        },
        "year": {0: 2019, 1: 2019, 2: 2020, 3: 2020, 4: 2020, 5: 2020, 6: 2020},
    }
)

collection.write("test", df, partition_on=["year"])

collection.item("test", filters=[("year", "==", 2020)]).to_pandas()
```

**Environment**:

- Dask version: 2.30.0
- Python version: 3.8
- Operating System: win10
- Install method (conda, pip, source): pip



Discussion:
Please try to formulate your example without pystore as an extra complication.
Note that I believe your data will have only one partition, so the filter (which is partition-based) will have no effect.
here is the pure dask version.
```python
import dask.dataframe
import pandas
from numpy import nan
from pandas import Timestamp

df = pandas.DataFrame.from_dict(
    {
        "time": {
            0: Timestamp("2019-12-31 21:00:00"),
            1: Timestamp("2019-12-31 22:00:00"),
            2: Timestamp("2019-12-31 23:00:00"),
            3: Timestamp("2020-11-25 19:00:00"),
            4: Timestamp("2020-11-26 02:00:00"),
            5: Timestamp("2020-11-26 03:00:00"),
            6: Timestamp("2020-11-26 04:00:00"),
        },
        "A": {0: 351.0, 1: 355.0, 2: 358.0, 3: 266.0, 4: 266.0, 5: 268.0, 6: nan},
        "B": {0: 2063.0, 1: 2051.0, 2: 1749.0, 3: 4281.0, 4: 3526.0, 5: 3462.0, 6: nan},
        "year": {0: 2019, 1: 2019, 2: 2020, 3: 2020, 4: 2020, 5: 2020, 6: 2020},
    }
)
ddf = dask.dataframe.from_pandas(df, npartitions=1)
dask.dataframe.to_parquet(ddf, "dask_parquet", partition_on="year")
# raise exception
dask.dataframe.read_parquet("dask_parquet", filters=[("year", "==", 2020)])
```

with the exception
```
Traceback (most recent call last):
  File ".../tst_bug_nonetype_pure_dask.py", line 45, in <module>
    ndf=dask.dataframe.read_parquet("dask_parquet",filters=[("year", "==", 2020)])
  File "...\site-packages\dask\dataframe\io\parquet\core.py", line 243, in read_parquet
    parts, divisions, index, index_in_columns = process_statistics(
  File "...\site-packages\dask\dataframe\io\parquet\core.py", line 711, in process_statistics
    out = sorted_columns(statistics)
  File "...\site-packages\dask\dataframe\io\parquet\core.py", line 605, in sorted_columns
    assert divisions == sorted(divisions)
TypeError: '<' not supported between instances of 'NoneType' and 'NoneType'
```

I can reproduce the problem; during computing stats for column A, which is not even of interest in this operation. I can report that things work if:
- there is more than one partition in `ddf`
- you use fastparquet without dask for reading
- you use pyarrow with or without dask for reading

due to
 /Users/mdurant/code/dask/dask/dataframe/io/parquet/core.py::sorted_columns
```
695         for stats in statistics[1:]:
```
you always get `success=True` when there is only one partition, even if it has no max/min due to the presence of NaNs.
Thanks for the reproducing example, @sdementen . With my diagnosis above, you should be able to make the fix and include your code as a test in dask/dataframe/io/tests/test_parquet.py

(the code line is actually number 843 is current master)
Thank you @martindurant for the insight.
However, you are maybe overestimating my skills re dask/parquet/partitions/etc.
I have fixed my issue locally by commenting the assert line that was raising the issue (i.e. I am not blocked by the issue)
If you want me to contribute a PR to fix this issue, I am still in the fog ;-)

Resolving patch:
diff --git a/dask/dataframe/io/parquet/core.py b/dask/dataframe/io/parquet/core.py
--- a/dask/dataframe/io/parquet/core.py
+++ b/dask/dataframe/io/parquet/core.py
@@ -839,7 +839,7 @@ def sorted_columns(statistics):
             continue
         divisions = [c["min"]]
         max = c["max"]
-        success = True
+        success = c["min"] is not None
         for stats in statistics[1:]:
             c = stats["columns"][i]
             if c["min"] is None:


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -3042,3 +3042,34 @@ def test_to_parquet_overwrite_raises(tmpdir, engine):
         dd.to_parquet(ddf, "./", engine=engine, overwrite=True)
     with pytest.raises(ValueError):
         dd.to_parquet(ddf, tmpdir, engine=engine, append=True, overwrite=True)
+
+
+def test_dir_filter(tmpdir, engine):
+    # github #6898
+    df = pd.DataFrame.from_dict(
+        {
+            "A": {
+                0: 351.0,
+                1: 355.0,
+                2: 358.0,
+                3: 266.0,
+                4: 266.0,
+                5: 268.0,
+                6: np.nan,
+            },
+            "B": {
+                0: 2063.0,
+                1: 2051.0,
+                2: 1749.0,
+                3: 4281.0,
+                4: 3526.0,
+                5: 3462.0,
+                6: np.nan,
+            },
+            "year": {0: 2019, 1: 2019, 2: 2020, 3: 2020, 4: 2020, 5: 2020, 6: 2020},
+        }
+    )
+    ddf = dask.dataframe.from_pandas(df, npartitions=1)
+    ddf.to_parquet(tmpdir, partition_on="year", engine=engine)
+    dd.read_parquet(tmpdir, filters=[("year", "==", 2020)], engine=engine)
+    assert all