ENH: Draft metadata specification doc for Apache Parquet
State: closed

cc @martindurant @mrocklin @jreback @cpcloud

closes #16010 

Labels:
name: Docs
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
This does not provide for non-string column names. I'm open to ideas about how to deal with that
Thank you, @wesm , this looks perfect from a quick scan.
One question: do we want to make provision for special object columns, such as bool-with-null, int-ith-null and dict/list (the latter being natural for json encoding)? 
For boolean with null, I expect we would have

```
{'type': 'bool',
 'numpy_type': 'object'}
```

We'll have to run type inference on the object columns at some point anyhow to know what Parquet type to write them to. 

For JSON, we could use `'type': 'json', 'numpy_type': 'object'`?
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=h1) Report
> Merging [#16315](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/81aa70c2a598f9c12d8cb56a4d758b7f213770fe?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16315/graphs/tree.svg?width=650&height=150&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16315      +/-   ##
==========================================
- Coverage   90.39%   90.37%   -0.02%     
==========================================
  Files         161      161              
  Lines       50863    50863              
==========================================
- Hits        45978    45968      -10     
- Misses       4885     4895      +10
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.16% <ø> (-0.01%)` | :arrow_down: |
| #single | `40.33% <ø> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `90.68% <0%> (-0.35%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.59% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=footer). Last update [81aa70c...2014a68](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=h1) Report
> Merging [#16315](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/d92f06a06fe45f99cbc8a813f4128447f9466aa5?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/16315/graphs/tree.svg?width=650&height=150&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #16315      +/-   ##
==========================================
- Coverage   90.38%   90.37%   -0.02%     
==========================================
  Files         161      161              
  Lines       50916    50949      +33     
==========================================
+ Hits        46021    46043      +22     
- Misses       4895     4906      +11
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.14% <ø> (ø)` | :arrow_up: |
| #single | `40.2% <ø> (-0.13%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.69% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/series.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvc2VyaWVzLnB5) | `94.71% <0%> (ø)` | :arrow_up: |
| [pandas/core/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY2F0ZWdvcmljYWwucHk=) | `95.87% <0%> (+0.01%)` | :arrow_up: |
| [pandas/core/generic.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ2VuZXJpYy5weQ==) | `91.96% <0%> (+0.01%)` | :arrow_up: |
| [pandas/core/common.py](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tbW9uLnB5) | `91.05% <0%> (+0.02%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=footer). Last update [d92f06a...2d00f55](https://codecov.io/gh/pandas-dev/pandas/pull/16315?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Any more comments? 
Looks good from my (Parquet) perspective. Interesting thing would be on how to deal with object columns.
A small comment on putting this in the docs: this will now create a top level entry in the toc of our docs called "Storing pandas Objects in Various File Formats". I think most users will think of something else when they see that section in the toc. 
Maybe we can put it somewhere under the 'internals' or 'contributing' page? Or give it a slightly more developer-oriented title?
I'm about 0+ on @jorisvandenbossche comment. I can see the point of not having a top-level, but this is almost a new concept, downstream library documentation, and not really internals. So would be +1 on have a 'Downstream Documention' (maybe a better name)?
@wesm if you'd move to ``developer.rst`` (added here: https://github.com/pandas-dev/pandas/commit/46dc536f554aa0dc2b6a925356c7f6b8dc88f34d).

Note the dev docs are built, but not being uploaded ATM.
OK, no problem
Updated
lgtm. (tiny typo). merge when ready.
If you want to add a note in whatsnew pointing to the new section would be ok (up 2 you)
Perhaps a little late but no chance of having `Interval` types in this?
If you'd like to add, submit a PR?
@wesm, cool. curious about the format definition over here: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#interval from this I gather that `INTERVAL` ≈ Timedelta? maybe this observation is not significant at all — still trying to wrap my head around the path forward for parquet/feather et al
@buyology pls create a new issue to add the ``Interval`` type. This is not actually support by ``pyarrow`` ATM. and confusingly, it is NOT the INTERVAL type at all (which is simply a ``Timedelta``). Rather this is a new type.
There is no "INTERVAL" in pandas, we have timedelta. That parquet has such a thing doesn't really concern us, if we can represent everything in pandas without it (TIME_MILLIS will do), and in fact I'd say that type is pretty useless (does anyone use it?).
@martindurant starting in 0.20.1 we do: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#intervalindex (the scalar type is ``Interval``)
@jreback : stop innovating, life will be easier :)
So there are something completely different, now I see what you mean above.