Support for partition_cols in to_parquet
State: closed

- [x] closes [#23283](https://github.com/pandas-dev/pandas/issues/23283)
- [x] tests passed
- [x] passes git diff upstream/master -u -- "*.py" | flake8 --diff
- [x] whatsnew entry

Labels:
name: IO Parquet
description: parquet, feather

Comments:
Hello @anjsudh! Thanks for updating the PR.

 - There are no PEP8 issues in the file [`pandas/core/frame.py`](https://github.com/anjsudh/pandas/blob/1f0978f57506ef71079080b3e57419229283694b/pandas/core/frame.py) !

 - There are no PEP8 issues in the file [`pandas/io/parquet.py`](https://github.com/anjsudh/pandas/blob/1f0978f57506ef71079080b3e57419229283694b/pandas/io/parquet.py) !

 - There are no PEP8 issues in the file [`pandas/tests/io/test_parquet.py`](https://github.com/anjsudh/pandas/blob/1f0978f57506ef71079080b3e57419229283694b/pandas/tests/io/test_parquet.py) !

 - There are no PEP8 issues in the file [`pandas/tests/util/test_testing.py`](https://github.com/anjsudh/pandas/blob/1f0978f57506ef71079080b3e57419229283694b/pandas/tests/util/test_testing.py) !

 - There are no PEP8 issues in the file [`pandas/util/testing.py`](https://github.com/anjsudh/pandas/blob/1f0978f57506ef71079080b3e57419229283694b/pandas/util/testing.py) !



##### Comment last updated on November 05, 2018 at 19:08 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23321?src=pr&el=h1) Report
> :exclamation: No coverage uploaded for pull request base (`master@11c0d28`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `90%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/23321/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/23321?src=pr&el=tree)

```diff
@@            Coverage Diff            @@
##             master   #23321   +/-   ##
=========================================
  Coverage          ?   92.25%           
=========================================
  Files             ?      161           
  Lines             ?    51277           
  Branches          ?        0           
=========================================
  Hits              ?    47305           
  Misses            ?     3972           
  Partials          ?        0
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.63% <90%> (?)` | |
| #single | `42.32% <15%> (?)` | |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/23321?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/23321/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.03% <ø> (ø)` | |
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/23321/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `84.76% <100%> (ø)` | |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/23321/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `86.71% <77.77%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23321?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/23321?src=pr&el=footer). Last update [11c0d28...8b45547](https://codecov.io/gh/pandas-dev/pandas/pull/23321?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

The error in https://travis-ci.org/pandas-dev/pandas/jobs/446310514#L2577 suggests that we'll need to put a minimum pyarrow version to support this behavior. Can you check what version that is, and raise a with an ImportError with a nice error message if the pyarrow is too old?
@WillAyd Hi, hope you can review the diff now? Have added the necessary tests
From a `pyarrow` perspective this is LGTM.
closes #23373 in a way
cc @martindurant for fastparquet things:

tldr: this PR is adding support for writing a dataframe partitioned by columns with the pyarrow engine.

1. fastparquet uses `partition_on`, pyarrow uses `partition_cols`. @xhochy @martindurant Any thoughts on standardizing on a name (stepping back, these do the same thing, correct?)
2. @martindurant thoughts on the change at 
https://github.com/pandas-dev/pandas/pull/23321#discussion_r228909418?
@WillAyd @jreback @TomAugspurger Have made the necessary changes. Hope you can have a look?
Merge conflict in `io/parquet.py`, if you could fix that up I think this will be good to go.
@jreback @WillAyd hope you can have a look ?
Nope. I don't really care about bugs in shutil :)

On Thu, Nov 8, 2018 at 10:31 AM William Ayd <notifications@github.com>
wrote:

> *@WillAyd* commented on this pull request.
> ------------------------------
>
> In doc/source/io.rst
> <https://github.com/pandas-dev/pandas/pull/23321#discussion_r231966117>:
>
> > +
> +    test
> +    ├── a=0
> +    │   ├── 0bac803e32dc42ae83fddfd029cbdebc.parquet
> +    │   └──  ...
> +    └── a=1
> +        ├── e6ab24a4f45147b49b54a662f0c412a3.parquet
> +        └── ...
> +
> +.. ipython:: python
> +   :suppress:
> +
> +   from shutil import rmtree
> +   try:
> +       rmtree('test')
> +   except Exception:
>
> Hmm OK. No concern around it catching exceptions that it shouldn't though?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/pandas-dev/pandas/pull/23321#discussion_r231966117>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ABQHIjUauolVWoYkrRq48bKY5stxCe4eks5utFxdgaJpZM4X4g_n>
> .
>

Looks good. Nice work @anjsudh!