Repository: dask/dask

Issue:
dask.dataframe.read_parquet producing empty columns for those partitioned on
When using pyarrow as the engine in read_parquet, I'm encountering an error working with the columns on which I've partitioned.

If I select the column that I partitioned on, I get an empty series back.

I am using dask version ('2.8.1'), pyarrow ('0.15.1'), nd fastparquet ('0.3.2').
```
$ conda create -c conda-forge -n dask.arrow.issue -y dask fastparquet pyarrow jupyterlab python-snappy
```

To reproduce

```
import dask, pyarrow, fastparquet
(dask.__version__, pyarrow.__version__, fastparquet.__version__) # ('2.9.1', '0.15.1', '0.3.2')

import pandas as pd
import dask.dataframe as dd
import pyarrow.parquet as pq

df = pd.DataFrame([{'str': str(i), 'int':i, 'group': 'ABC'[i%3]} for i in range(6)])

pq_dir = './parquet_partition_issue'
df.to_parquet(pq_dir, partition_cols=['group'], engine='fastparquet')

ddf_arrow = dd.read_parquet(pq_dir, engine='pyarrow')
ddf_fast = dd.read_parquet(pq_dir, engine='fastparquet')

print(len(ddf_arrow['group'].compute()))  # 0
print(len(ddf_fast['group'].compute()))  # 6
print(len(ddf_arrow.compute().group))  # 6
print(len(pq.read_table(pq_dir).to_pandas().group))  # 6
```



Discussion:
Thanks @caseyclements for the clear and reproducible example!   I am also seeing the error with master.  Unfortunately i don't know enough about the metadata handling differences between fastparquet and pyarrow.  @TomAugspurger or @martindurant do you have an idea of what's going on here ?
(also @rjzamora )
Note that this problem still occurs if parquet is written using pyarrow. I did it this way explicitly because of another (perhaps related) issue: #5774 
Thanks for raising @caseyclements - The problem was probably introduced in `optimize_read_parquet_getitem`.  The graph optimization will effectively use `columns=["group"]` for the `read_parquet` call, however a partition should *not* be specified as a column. 

I will confirm that this is the problem and see if I can find a reasonable fix.

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -342,6 +342,16 @@ def read_partition(
                 open_file_func=partial(fs.open, mode="rb"),
             )
 
+        # Ensure `columns` and `partitions` do not overlap
+        columns_and_parts = columns.copy()
+        if columns_and_parts and partitions:
+            for part_name in partitions.partition_names:
+                if part_name in columns:
+                    columns.remove(part_name)
+                else:
+                    columns_and_parts.append(part_name)
+            columns = columns or None
+
         df = piece.read(
             columns=columns,
             partitions=partitions,
@@ -349,7 +359,7 @@ def read_partition(
             use_threads=False,
             **kwargs.get("read", {}),
         ).to_pandas(categories=categories, use_threads=False, ignore_metadata=True)[
-            list(columns)
+            list(columns_and_parts)
         ]
 
         if index:


Test patch:
diff --git a/dask/dataframe/io/tests/test_parquet.py b/dask/dataframe/io/tests/test_parquet.py
--- a/dask/dataframe/io/tests/test_parquet.py
+++ b/dask/dataframe/io/tests/test_parquet.py
@@ -2243,3 +2243,17 @@ def test_roundtrip_pandas_chunksize(tmpdir, write_engine, read_engine):
     )
 
     assert_eq(pdf, ddf_read)
+
+
+def test_read_pandas_fastparquet_partitioned(tmpdir, engine):
+    check_fastparquet()
+
+    pdf = pd.DataFrame(
+        [{"str": str(i), "int": i, "group": "ABC"[i % 3]} for i in range(6)]
+    )
+    path = str(tmpdir)
+    pdf.to_parquet(path, partition_cols=["group"], engine="fastparquet")
+    ddf_read = dd.read_parquet(path, engine=engine)
+
+    assert len(ddf_read["group"].compute()) == 6
+    assert len(ddf_read.compute().group) == 6