Repository: dask/dask

Issue:
ParquetFile with filters should lead to df.known_divisions==True

### TLDR

This is related with https://github.com/dask/fastparquet/issues/370


Currently, `dask` together with `fastparquet`, do not lead to `df.known_divisions==True` in the example below and it should.

I'm soon uploading a PR that, together with https://github.com/dask/fastparquet/pull/371. Fixes this.

Please see below:

### Imports


```python
import numpy as np
import pandas as pd
import dask.dataframe as dd
import fastparquet
from fastparquet import ParquetFile
```

### Generate dummy data. Save to a partitioned parquet file


```python
def generate_df(size, id_val=None):
    df = pd.DataFrame({'unique' : np.arange(0, size),
                       'dates' : pd.date_range('2015-01-01',
                                             periods=size,
                                             freq='1min',
                                             name='dates')})
    if id_val is not None:
        df['id']=id_val
    return df
```


```python
df_list = [generate_df(4, id_val = id_val) for id_val in ['id1','id2'] ]
df_pd = pd.concat(df_list).sort_values('dates').set_index('dates')
df_pd
```




<div>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>unique</th>
      <th>id</th>
    </tr>
    <tr>
      <th>dates</th>
      <th></th>
      <th></th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2015-01-01 00:00:00</th>
      <td>0</td>
      <td>id1</td>
    </tr>
    <tr>
      <th>2015-01-01 00:00:00</th>
      <td>0</td>
      <td>id2</td>
    </tr>
    <tr>
      <th>2015-01-01 00:01:00</th>
      <td>1</td>
      <td>id1</td>
    </tr>
    <tr>
      <th>2015-01-01 00:01:00</th>
      <td>1</td>
      <td>id2</td>
    </tr>
    <tr>
      <th>2015-01-01 00:02:00</th>
      <td>2</td>
      <td>id1</td>
    </tr>
    <tr>
      <th>2015-01-01 00:02:00</th>
      <td>2</td>
      <td>id2</td>
    </tr>
    <tr>
      <th>2015-01-01 00:03:00</th>
      <td>3</td>
      <td>id1</td>
    </tr>
    <tr>
      <th>2015-01-01 00:03:00</th>
      <td>3</td>
      <td>id2</td>
    </tr>
  </tbody>
</table>
</div>




```python
df = dd.from_pandas(df_pd, npartitions=2)
```


```python
# The divisions
df.divisions
```




    (Timestamp('2015-01-01 00:00:00'),
     Timestamp('2015-01-01 00:02:00'),
     Timestamp('2015-01-01 00:03:00'))




```python
# 2 partitions as expected
print('partitions: ' + str(df.npartitions))
```

    partitions: 2



```python
df.to_parquet('./data.parquet', compression='snappy',
              engine='fastparquet', partition_on=['id'])
```

### Load ParquetFile


```python
pq_f = ParquetFile('./data.parquet')
```


```python
# the group files
pq_f.group_files
```




    {0: {'id=id1/part.0.parquet'},
     1: {'id=id2/part.0.parquet'},
     2: {'id=id1/part.1.parquet'},
     3: {'id=id2/part.1.parquet'}}



### Create dask.DataFrame with filters and check divisions

Assume I want to filter the parquet file only based on the following filters:


```python
filters = [('id', '==', 'id1')]
```

At the moment, this will lead to unknown divisions:


```python
df_1 = dd.read_parquet(pq_f, filters=filters)
print(df_1.divisions)
print(df_1.known_divisions)
```

    (None, None, None)
    False


I'm soon creating a PR that will fix this and lead to the expected behaviour:


```python
df_1 = dd.read_parquet(pq_f, filters=filters)
print(df_1.divisions)
print(df_1.known_divisions)
```

    (Timestamp('2015-01-01 00:00:00'), Timestamp('2015-01-01 00:02:00'), Timestamp('2015-01-01 00:03:00'))
    True




Discussion:

Resolving patch:
diff --git a/dask/dataframe/io/parquet.py b/dask/dataframe/io/parquet.py
--- a/dask/dataframe/io/parquet.py
+++ b/dask/dataframe/io/parquet.py
@@ -285,13 +285,15 @@ def _read_fastparquet(fs, fs_token, paths, columns=None, filters=None,
 
     if index_names and infer_divisions is not False:
         index_name = meta.index.name
-        minmax = fastparquet.api.sorted_partitioned_columns(pf)
+        try:
+            # is https://github.com/dask/fastparquet/pull/371 available in
+            # current fastparquet installation?
+            minmax = fastparquet.api.sorted_partitioned_columns(pf, filters)
+        except TypeError:
+            minmax = fastparquet.api.sorted_partitioned_columns(pf)
         if index_name in minmax:
             divisions = minmax[index_name]
-            idx_lst = [i for i, rg in enumerate(pf.row_groups)
-                       if rg in rgs]
-            divisions = [divisions['min'][i]
-                         for i in idx_lst] + [divisions['max'][idx_lst[-1]]]
+            divisions = divisions['min'] + [divisions['max'][-1]]
         else:
             if infer_divisions is True:
                 raise ValueError(


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -842,6 +842,29 @@ def test_divisions_read_with_filters(tmpdir):
     assert out.divisions == expected_divisions
 
 
+def test_divisions_are_known_read_with_filters(tmpdir):
+    check_fastparquet()
+    tmpdir = str(tmpdir)
+    #generate dataframe
+    df = pd.DataFrame({'unique': [0, 0, 1, 1, 2, 2, 3, 3],
+                       'id': ['id1', 'id2',
+                              'id1', 'id2',
+                              'id1', 'id2',
+                              'id1', 'id2']},
+                      index=[0, 0, 1, 1, 2, 2, 3, 3])
+    d = dd.from_pandas(df, npartitions=2)
+    #save it
+    d.to_parquet(tmpdir, partition_on=['id'], engine='fastparquet')
+    #read it
+    out = dd.read_parquet(tmpdir,
+                          engine='fastparquet',
+                          filters=[('id', '==', 'id1')])
+    #test it
+    assert out.known_divisions
+    expected_divisions = (0, 2, 3)
+    assert out.divisions == expected_divisions
+
+
 def test_read_from_fastparquet_parquetfile(tmpdir):
     check_fastparquet()
     fn = str(tmpdir)