BUG: Write compliant Parquet with `pyarrow`
State: closed

- [x] closes #43689
- [x] tests added / passed
- [x] Ensure all linting tests pass, see [here](https://pandas.pydata.org/pandas-docs/dev/development/contributing_codebase.html#pre-commit) for how to run them
- [x] whatsnew entry


Labels:
name: Bug
name: IO Parquet
description: parquet, feather
name: Stale

Comments:
pls try to solve this conditionally based on the pyarrow version
Doesn't it feel a bit odd to have a different output format depending on the version of PyArrow that just happens to be installed? 

(I believe `fastparquet` doesn't matter as they don't support any nested types) 
Hello @judahrand! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:







There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2021-09-30 21:10:49 UTC
Do you think this is something which can / should be back ported to `1.3.4` or should it only appear in `1.4.0`?
> Do you think this is something which can / should be back ported to `1.3.4` or should it only appear in `1.4.0`?

no
Test timed out rather than failed.
@jreback I've found your feedback on this change quite unhelpful so far. Are you happy with this change in general? I'm not going to put more time into this if the change isn't likely to be merged anyway. I've still not really had any acknowledgement that you understand the problem that this change addresses. Do you? 

If I adjust the release note to a single entry and get the tests passing are you happy with this change?
> @jreback I've found your feedback on this change quite unhelpful so far. Are you happy with this change in general? I'm not going to put more time into this if the change isn't likely to be merged anyway. I've still not really had any acknowledgement that you understand the problem that this change addresses. Do you?
> 
> If I adjust the release note to a single entry and get the tests passing are you happy with this change?

@judahrand yes the change is acceptable for code & tests. a 1-line whatsnew note is all that is needed here.

ps. a helpful attitude is a good thing. We have many PRs.
> ps. a helpful attitude is a good thing. We have many PRs.
Completely agree.

> @judahrand yes the change is acceptable for code & tests. a 1-line whatsnew note is all that is needed here.

Great, I'll deal with this when I get a chance.

@jorisvandenbossche if good here
Thanks for the ping.

My first reaction is: if we think this is the better default, we should bring that up to change it in pyarrow, and not in pandas (otherwise you get inconsistencies between when using pandas or pyarrow, or eg when using dask which can use pyarrow directly instead of going through the pandas layer). 
In https://issues.apache.org/jira/browse/ARROW-11497 it was actually also mentioned that long term Arrow would like to switch to conforming the spec by default, but on the short term it only exposed this keyword.

Now, it's certainly harder / higher barrier to change this in Arrow (and you put already a lot of effort in this PR, thanks for that!), but I would personally still prefer to at least first bring it up, and see what the reaction is of other Arrow devs.

I opened https://issues.apache.org/jira/browse/ARROW-14196 for this
> I opened https://issues.apache.org/jira/browse/ARROW-14196 for this

Haha! I didn't clock that had only just been opened when I commented on it!
This pull request is stale because it has been open for thirty days with no activity. Please update or respond to this comment if you're still interested in working on this.
closing as this going to be addressed as above in pyarrow