COMPAT: pyarrow >= 0.7.0 compat
State: closed

closes #17581


Labels:
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO Parquet
description: parquet, feather

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=h1) Report
> Merging [#17588](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0e85ca7f5afb26880e81ba6ff3965c8c0f27bbaf?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17588/graphs/tree.svg?token=eZ4WkYLtcO&width=650&height=150&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17588      +/-   ##
==========================================
- Coverage   91.22%    91.2%   -0.02%     
==========================================
  Files         163      163              
  Lines       49625    49625              
==========================================
- Hits        45270    45261       -9     
- Misses       4355     4364       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.99% <ø> (ø)` | :arrow_up: |
| #single | `40.19% <ø> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.77% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=footer). Last update [0e85ca7...5bc3def](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=h1) Report
> Merging [#17588](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/0e85ca7f5afb26880e81ba6ff3965c8c0f27bbaf?src=pr&el=desc) will **decrease** coverage by `0.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17588/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&height=150&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17588      +/-   ##
==========================================
- Coverage   91.22%    91.2%   -0.02%     
==========================================
  Files         163      163              
  Lines       49625    49625              
==========================================
- Hits        45270    45261       -9     
- Misses       4355     4364       +9
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.99% <ø> (ø)` | :arrow_up: |
| #single | `40.19% <ø> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.77% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=footer). Last update [0e85ca7...0c75389](https://codecov.io/gh/pandas-dev/pandas/pull/17588?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

```
In [13]: import pyarrow

In [14]: pyarrow.__version__
Out[14]: '0.7.0'

In [15]: df = pd.DataFrame({'a': pd.Categorical(list('abc'))})

In [16]: df.dtypes
Out[16]: 
a    category
dtype: object

In [17]: df.to_parquet('foo.pq', engine='pyarrow')

In [18]: pd.read_parquet('foo.pq', engine='pyarrow')
Out[18]: 
   a
0  a
1  b
2  c

In [19]: pd.read_parquet('foo.pq', engine='pyarrow').dtypes
Out[19]: 
a    object
dtype: object
```

@wesm 
de-serializing seems not to preserve the cat type ?
Correct, it comes back as non-categorical. _Parquet does not have a categorical type_. We can try to emulate it as best we can (not foolproof; when dictionaries grow too big, encoding is turned off) but it will take some more work in parquet-cpp.
no need to emulate. will make a note in the docs.
We're going to support direct column reads as categorical soon hopefully, which will use the dictionary page if there is one, but that has some edge cases:

* Dictionaries generally will be different from file to file
* Dictionaries may be different within each row group
* A column chunk may switch to plain encoding mid-stream (if the dictionary got too big)