DOC/TST: Update the parquet (pyarrow >= 0.15) docs and tests regarding Categorical support
State: closed

- [x] closes #27955
- [x] tests added / passed
- [x] passes `black pandas`
- [x] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: Docs
name: IO Parquet
description: parquet, feather

Comments:
Several questions/thoughts:
1. Should I also add a test in this part: https://github.com/pandas-dev/pandas/blob/802f67046bbae0a815b2fe9d20d2217485bbc942/pandas/tests/io/test_parquet.py#L409 as well? I thought adding a test there would be a duplicate of what I'm doing in https://github.com/pandas-dev/pandas/blob/802f67046bbae0a815b2fe9d20d2217485bbc942/pandas/tests/io/test_parquet.py#L451, but maybe I'm missing something
2. In https://github.com/apache/arrow/pull/5110 the test in pyarrow demonstrates how category values will be preserved regardless whether they occur in the data or not; should the test in pandas demonstrate this as well?
> Should I also add a test in this part: 

I think your analysis is correct that it would be duplicative. 
I think once we don't have the version check anymore (in the future if we would only support pyarrow >= 0.15), then we can move the ``test_categorical`` into the `test_basic`. But for now I would keep it as is.

> In apache/arrow#5110 the test in pyarrow demonstrates how category values will be preserved regardless whether they occur in the data or not; should the test in pandas demonstrate this as well?

Yes, I think that is nice to test. In addition, you could also test if the `ordered` flag is preserved as well. You can add additional columns to the `df` with different cases in `test_categorical`.
Thanks for the review! I've added another test for null, out-of-order values & unobserved category and updated the doc as well.

However when I tried to add another test to check if order is preserved, it seems like it's not preserved. I'm not really sure if this is expected or if I'm missing something:

```
import pandas as pd
from pandas.io.parquet import read_parquet, to_parquet

df = pd.DataFrame()
df["a"] = pd.Categorical(["a", "b", "c", "a"], categories=["b", "c", "d"], ordered=True)

df.to_parquet(<path>)
actual = read_parquet(<path>)

df["a"]    
0    NaN
1      b
2      c
3    NaN
Name: a, dtype: category
Categories (3, object): [b < c < d]

actual["a"]
0    NaN
1      b
2      c
3    NaN
Name: a, dtype: category
Categories (3, object): [b, c, d]

```

If it's expected then maybe I can add another test that expects it to still return a categorical data type without order, if needed.

> However when I tried to add another test to check if order is preserved, it seems like it's not preserved. I'm not really sure if this is expected or if I'm missing something:

Indeed, I see the same trying your code. That is certainly not expected I think. Can you open an issue for this? (Arrow uses a JIRA tracker: https://issues.apache.org/jira/projects/ARROW/issues; I can also open an issue if you want)
 
Hmm, the failure on Azure is actually related:

```=================================== FAILURES ===================================
_____________________ TestParquetPyArrow.test_categorical ______________________
[gw0] darwin -- Python 3.5.6 /Users/vsts/miniconda3/envs/pandas-dev/bin/python

self = <pandas.tests.io.test_parquet.TestParquetPyArrow object at 0x146354a90>
pa = 'pyarrow'

    def test_categorical(self, pa):
    
        # supported in >= 0.7.0
        df = pd.DataFrame()
        df["a"] = pd.Categorical(list("abcdef"))
    
        # test for null, out-of-order values, and unobserved category
        dtype = pd.CategoricalDtype(["foo", "bar", "baz"])
        df["b"] = pd.Categorical.from_codes(codes=[1, 0, 0, 1, -1, 1], dtype=dtype)
    
        if LooseVersion(pyarrow.__version__) >= LooseVersion("0.15.0"):
            check_round_trip(df, pa)
        else:
            # de-serialized as object for pyarrow < 0.15
            expected = df.assign(a=df.a.astype(object))
>           check_round_trip(df, pa, expected=expected)

pandas/tests/io/test_parquet.py:468: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pandas/tests/io/test_parquet.py:175: in check_round_trip
    compare(repeat)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

repeat = 2

    def compare(repeat):
        for _ in range(repeat):
            df.to_parquet(path, **write_kwargs)
            with catch_warnings(record=True):
                actual = read_parquet(path, **read_kwargs)
    
>           tm.assert_frame_equal(expected, actual, check_names=check_names)
E           AssertionError: Attributes are different
E           
E           Attribute "dtype" are different
E           [left]:  CategoricalDtype(categories=['foo', 'bar', 'baz'], ordered=False)
E           [right]: object

pandas/tests/io/test_parquet.py:171: AssertionError
```
It seems that the `expected` still has the categorical dtype (so the `assign` failed to overwrite the original column), but not sure why this only happens on the mac/windows builds ..
Ah, no, I already see ;). In the `expected` for the branch for older pyarrow versions, you still need to convert column `'b'` to object dtype as well
> Indeed, I see the same trying your code. That is certainly not expected I think. Can you open an issue for this? (Arrow uses a JIRA tracker: https://issues.apache.org/jira/projects/ARROW/issues; I can also open an issue if you want)

I've opened the issue here: https://issues.apache.org/jira/browse/ARROW-6302
Thanks!
@jorisvandenbossche I've added the test for the ordered flag. Is there any other test that you think we should have for now?
> I've just tried categorical dtypes using integer and in pyarrow it gets de-serialized into float. Not sure if it's expected, if someone can confirm it'd be great.

For Arrow, that is expected, see https://issues.apache.org/jira/browse/ARROW-6140 (the fact that it is float and not int is because there are missing values)

> With fastparquet, both string and non-string types get de-serialized back as category, however it does not preserve order for both, while in pyarrow (for string) we do preserve order. Should we make this clear in the docs?

That could be mentioned yes.

I've updated the doc to explain more about the differences regarding how the categorical support works between pyarrow and fastparquet
@galuhsahid Thanks a lot for this PR! (and for the fixes on the pyarrow side as well! :-))