DOC: Update index parameter in pandas to_parquet
State: closed

- [ ] closes #xxxx (reference: https://github.com/python-sprints/pandas-mentoring/issues/156)
- [ ] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [ ] whatsnew entry

The documentation ([https://dev.pandas.io/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet](https://dev.pandas.io/reference/api/pandas.DataFrame.to_parquet.html#pandas.DataFrame.to_parquet)) says that for `DataFrame.to_parquet` when the value for the index parameter is `None`, the behavior will depend on the engine. However if we try this both engines, `pyarrow` and `fastparquet`, will keep the index. This PR proposes to set the default value of the index parameter as `index=True` instead of `index=None` since both engines keep the index anyway. Would appreciate any discussion/feedback :)

Sample code to demonstrate how both engines treat the index:
```
import pandas as pd 
  
data = {'name':['a', 'b', 'c', 'd']} 
df = pd.DataFrame(data, index =['rank_1', 'rank_2', 'rank_3', 'rank_4'])

df.info()
```
Output:
```
<class 'pandas.core.frame.DataFrame'>
Index: 4 entries, rank_1 to rank_4
Data columns (total 1 columns):
name    4 non-null object
dtypes: object(1)
memory usage: 64.0+ bytes
```
```
# pyarrow
df.to_parquet("test_pyarrow.parquet", engine="pyarrow")
df_read_pyarrow = pd.read_parquet("test_pyarrow.parquet", engine="pyarrow")
df_read_pyarrow.info()
```
Output:
```
<class 'pandas.core.frame.DataFrame'>
Index: 4 entries, rank_1 to rank_4
Data columns (total 1 columns):
name    4 non-null object
dtypes: object(1)
memory usage: 64.0+ bytes
```

```
# fastparquet
df.to_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet = pd.read_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet.info()
```
Output:
```
<class 'pandas.core.frame.DataFrame'>
Index: 4 entries, rank_1 to rank_4
Data columns (total 1 columns):
name    4 non-null object
dtypes: object(1)
memory usage: 64.0+ bytes
```

Labels:
name: Clean
name: IO Parquet
description: parquet, feather

Comments:
Thanks for working on this @galuhsahid, looks great. There are couple of tests that need to be updated (see the CI). Let me know if you need help with it.

@wesm does this change make sense? I think by default we're saving the index with parquet with both engines. But we've got this `index=None` that the documentation doesn't clarify what it does, and personally I think it doesn't add value, but make things confusing (may be I'm missing something).
@datapythonista Okay so because of the failing tests I tried `to_parquet` without defining an index, and I observed a different behavior when I used the `fastparquet` engine for `index=True` and `index=None`. If I use `index=True` it will return `Int64Index`, and if I use `index=None` it will return `RangeIndex`.

I'm not really sure if this difference has any impact to our proposal, should we take this into consideration? Otherwise I can update the test to expect `Int64Index` instead of `RangeIndex`.

```
import pandas as pd

data = {'name':['a', 'b', 'c', 'd']}
df = pd.DataFrame(data)
df.to_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet = pd.read_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet.index
```

Output:
```
RangeIndex(start=0, stop=4, step=1)
```


```
import pandas as pd

data = {'name':['a', 'b', 'c', 'd']}
df = pd.DataFrame(data)
df.to_parquet("test_fastparquet.parquet", engine="fastparquet")
df_read_fastparquet = pd.read_parquet("test_fastparquet.parquet", engine="fastparquet", index=True)
df_read_fastparquet.index
```

Output:
```
Int64Index([0, 1, 2, 3], dtype='int64', name='index')
```

Thanks @galuhsahid for doing the research. I see that there is a difference between `index=None` and `index=True` with `fastparquet`, but only with the default  `RangeIndex`.

With a default index:
```python
>>> import pandas
>>> pandas.DataFrame({'foo': [1, 2]}).to_parquet('foo.parquet', engine='fastparquet', index=None)
>>> print(pandas.read_parquet('foo.parquet').index)
RangeIndex(start=0, stop=2, step=1)
>>> pandas.DataFrame({'foo': [1, 2]}).to_parquet('foo.parquet', engine='fastparquet', index=True)
>>> print(pandas.read_parquet('foo.parquet').index)
Int64Index([0, 1], dtype='int64', name='index')
```

With a set index:
```python
>>> import pandas
>>> pandas.DataFrame({'foo': [1, 2]}, index=['a', 'b']).to_parquet('foo.parquet', engine='fastparquet', index=None)
>>> print(pandas.read_parquet('foo.parquet').index)
Index(['a', 'b'], dtype='object', name='index')
>>> pandas.DataFrame({'foo': [1, 2]}, index=['a', 'b']).to_parquet('foo.parquet', engine='fastparquet', index=True)
>>> print(pandas.read_parquet('foo.parquet').index)
Index(['a', 'b'], dtype='object', name='index')
```

This looks like a bug to me, I don't see why we would like that behavior. @martindurant any thoughts on this. Do you think it makes sense to just have `index=True` and `index=False` and avoid this difference when `index=None`?
Can @jorisvandenbossche have a closer look at this before merging? I'm concerned this may cause a performance regression. 

We made changes in pyarrow to use `preserve_index=True` force serialization of the index column as data in the Parquet file, even for RangeIndex. If `preserve_index=None` then metadata-only RangeIndex is stored. 
Yes, @wesm is right. In the latest pyarrow release, `preserve_index=None` (the default) or `preserve_index=True` mean different things (the main difference is for RangeIndex being written as a column or just as metadata). So if pandas would change the default from None to True, that means it would override the default of pyarrow (leading again to writing the RangeIndex as a full column). 
Thanks for the info @jorisvandenbossche, should we just clarify in the documentation what `None` means then?
Yes, that is a good idea. From their docs, it seems that fastparquet uses the same logic of None vs True/False.
@galuhsahid do you want to update this PR with what discussed above? Restore the default parameter, and explain in the docstring what `None` means.
@datapythonista Sure thanks for notifying! Will push an update later today.
Hello @galuhsahid! Thanks for updating this PR. We checked the lines you've touched for [PEP 8](https://www.python.org/dev/peps/pep-0008) issues, and found:





There are currently no PEP 8 issues detected in this Pull Request. Cheers! :beers: 

##### Comment last updated at 2019-09-15 15:33:29 UTC
Thanks a lot @galuhsahid !