DOC: Add expanded index descriptors for specifying for RangeIndex-as-metadata in Parquet file schema 
State: closed

Closes #25672 

- [x] closes #xxxx
- [x] tests added / passed (N/A)
- [ ] passes `git diff upstream/master -u -- "*.py" | flake8 --diff` (N/A)
- [ ] whatsnew entry (N/A)

Labels:
name: IO Parquet
description: parquet, feather

Comments:
cc @cpcloud @TomAugspurger @martindurant @xhochy @jreback for review. This is pending for Apache Arrow 0.13.0 so if we want to make any changes it would be good to do it soon =) 
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=h1) Report
> Merging [#25709](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/86879ac9af524adc684bec77c621d27b19a36ca0?src=pr&el=desc) will **not change** coverage.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25709/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=tree)

```diff
@@           Coverage Diff           @@
##           master   #25709   +/-   ##
=======================================
  Coverage   91.25%   91.25%           
=======================================
  Files         172      172           
  Lines       52963    52963           
=======================================
  Hits        48330    48330           
  Misses       4633     4633
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.82% <ø> (ø)` | :arrow_up: |
| #single | `41.73% <ø> (ø)` | :arrow_up: |


------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=footer). Last update [86879ac...766aa50](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=h1) Report
> Merging [#25709](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fae84ec6af8bd94d7de7f21f0398abd72e5b1787?src=pr&el=desc) will **decrease** coverage by `1.21%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/25709/graphs/tree.svg?width=650&token=eZ4WkYLtcO&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #25709      +/-   ##
==========================================
- Coverage   93.02%   91.81%   -1.22%     
==========================================
  Files         187      175      -12     
  Lines       50109    52581    +2472     
==========================================
+ Hits        46613    48276    +1663     
- Misses       3496     4305     +809
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `90.36% <ø> (-1.33%)` | :arrow_down: |
| #single | `41.89% <ø> (-0.65%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/plotting/\_misc.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19taXNjLnB5) | `38.46% <0%> (-26.41%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `75% <0%> (-25%)` | :arrow_down: |
| [pandas/compat/\_\_init\_\_.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2NvbXBhdC9fX2luaXRfXy5weQ==) | `70.7% <0%> (-21.61%)` | :arrow_down: |
| [pandas/io/gcs.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2djcy5weQ==) | `80% <0%> (-20%)` | :arrow_down: |
| [pandas/io/s3.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL3MzLnB5) | `89.47% <0%> (-10.53%)` | :arrow_down: |
| [pandas/core/computation/expr.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvY29tcHV0YXRpb24vZXhwci5weQ==) | `88.52% <0%> (-9.26%)` | :arrow_down: |
| [pandas/core/groupby/base.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9iYXNlLnB5) | `91.83% <0%> (-8.17%)` | :arrow_down: |
| [pandas/io/excel/\_xlrd.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2V4Y2VsL194bHJkLnB5) | `93.93% <0%> (-6.07%)` | :arrow_down: |
| [pandas/core/groupby/categorical.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZ3JvdXBieS9jYXRlZ29yaWNhbC5weQ==) | `95.45% <0%> (-4.55%)` | :arrow_down: |
| [pandas/core/indexing.py](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhpbmcucHk=) | `90.88% <0%> (-4.17%)` | :arrow_down: |
| ... and [183 more](https://codecov.io/gh/pandas-dev/pandas/pull/25709/diff?src=pr&el=tree-more) | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=footer). Last update [fae84ec...10d1e86](https://codecov.io/gh/pandas-dev/pandas/pull/25709?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

Fixed the range thing
To enhance backwards compatibility I'm going to change the `{'kind': 'serialized', 'field_name': $COLUMN}` to simply `$COLUMN`. So only "special" indexes will have the dict representation. Stay tuned
I think this is ready to be merged? (at least, I think it reflects the situation in the last pyarrow release).

@martindurant is this OK from the fastparquet side? I think reading such metadata is not yet supported? (https://github.com/dask/fastparquet/issues/414) 
I don't know if fastparquet has plans to also support it when writing parquet files in the long term. If not, we could maybe make this "optional" in the description (as pyarrow can read serialized range indexes fine).
No, this is not yet handled in fastparquet :(
I have found that users of fastparquet are much less likely to ask for a non-useful index to be written, and if there is none when reading, indeed a standard rangeindex is made.

> I have found that users of fastparquet are much less likely to ask for a non-useful index to be written

The default is also to not write such indexes in fastparquet, which was probably a good choice. But basically this is a somewhat different way of pyarrow to deal with it, by only storing it in the metadata.
@jorisvandenbossche is this ready to merge?
Yes, this can be merged now. Did a small clean-up (`_start` -> `start` etc)