Repository: dask/dask

Issue:
write_parquet using partition_on breaking on GCS
**What happened**:

Saving a partitioned dask dataframe on google cloud storage raises the following error

```shell
/usr/local/lib/python3.7/site-packages/dask/dataframe/io/parquet/arrow.py in _write_partitioned()
    246         )
    247         prefix = fs.sep.join([root_path, subdir])
--> 248         fs.mkdir(prefix, exists_ok=True)
    249         outfile = guid() + ".parquet"
    250         full_path = fs.sep.join([prefix, outfile])

TypeError: mkdir() got an unexpected keyword argument 'exists_ok'
```

**What you expected to happen**:

The file should be saved on GCS using a folder structure for the partitions.

**Minimal Complete Verifiable Example**:

```python
import dask.dataframe as dd
dd.from_pandas(pd.DataFrame({'foo': [1, 1, 2, 2]}), npartitions=1).to_parquet('gs://...', partition_on=['foo'])
```

**Anything else we need to know?**:

1. Locally it works fine
2. Maybe there is a typo on https://github.com/dask/dask/blob/05c17758aa9d44158538b7d5059524de18843ae3/dask/dataframe/io/parquet/arrow.py#L75

**Environment**:

- Dask version: 2.21.0
- GCSFS version: 0.6.2
- Python version: 3.7.6
- Operating System: Ubuntu 18.04
- Install method (conda, pip, source): pip



Discussion:
@martindurant

On Tue, Jul 28, 2020 at 8:19 AM Hermes Ribeiro Sant Anna <
notifications@github.com> wrote:

> *What happened*:
>
> Saving a partitioned dask dataframe on google cloud storage raises the
> following error
>
> /usr/local/lib/python3.7/site-packages/dask/dataframe/io/parquet/arrow.py in _write_partitioned()
>     246         )
>     247         prefix = fs.sep.join([root_path, subdir])
> --> 248         fs.mkdir(prefix, exists_ok=True)
>     249         outfile = guid() + ".parquet"
>     250         full_path = fs.sep.join([prefix, outfile])
>
> TypeError: mkdir() got an unexpected keyword argument 'exists_ok'
>
> *What you expected to happen*:
>
> The file should be saved on GCS using a folder structure for the
> partitions.
>
> *Minimal Complete Verifiable Example*:
>
> import dask.dataframe as dd
> dd.from_pandas(pd.DataFrame({'foo': [1, 1, 2, 2]}), npartitions=1).to_parquet('gs://...', partition_on=['foo'])
>
> *Anything else we need to know?*:
>
>    1. Locally it works fine
>    2. Maybe there is a typo on
>    https://github.com/dask/dask/blob/05c17758aa9d44158538b7d5059524de18843ae3/dask/dataframe/io/parquet/arrow.py#L75
>
> *Environment*:
>
>    - Dask version: 2.21.0
>    - GCSFS version: 0.6.2
>    - Python version: 3.7.6
>    - Operating System: Ubuntu 18.04
>    - Install method (conda, pip, source): pip
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/6467>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTDXUCMDPAGQUEYJ7DDR53T7NANCNFSM4PKUIGZQ>
> .
>

`exist_ok` has never been in the signature of `mkdir` (it is in `makedirs`). It may have been silently ignored as part of **kwargs; I could make that change. Also worth checking against master, since a lot of code has changed since 0.6.2 .
cc @rjzamora then in case we shouldn't be using that API

Resolving patch:
diff --git a/dask/dataframe/io/parquet/arrow.py b/dask/dataframe/io/parquet/arrow.py
--- a/dask/dataframe/io/parquet/arrow.py
+++ b/dask/dataframe/io/parquet/arrow.py
@@ -72,7 +72,7 @@ def _write_partitioned(
             subgroup, preserve_index=preserve_index, schema=subschema, safe=False
         )
         prefix = fs.sep.join([root_path, subdir])
-        fs.mkdir(prefix, exists_ok=True)
+        fs.mkdirs(prefix, exist_ok=True)
         full_path = fs.sep.join([prefix, filename])
         with fs.open(full_path, "wb") as f:
             pq.write_table(subtable, f, metadata_collector=md_list, **kwargs)


Test patch: