TST/DOC: test pyarrow tz data + doc / enable cross compat tests for pyarrow/fastparquet
State: closed

Commits:

TST: add parquet test with tz datetime data for pyarrow
    
+ clean-up basic data types tests: make common dataframe with types
supported by both pyarrow and fastparquet

DOC: document differences between pyarrow and fastparquet in supported data types

TST: enable pyarrow/fastparquet cross compatibility tests on smaller subset of dataframe

Closes https://github.com/pandas-dev/pandas/issues/17448 
Also adds a test for https://github.com/pandas-dev/pandas/issues/18628


Labels:
name: Testing
description: pandas testing functions or related to the test suite
name: IO Parquet
description: parquet, feather

Comments:
Hello @jorisvandenbossche! Thanks for updating the PR.



Cheers ! There are no PEP8 issues in this Pull Request. :beers: 

##### Comment last updated on December 08, 2017 at 08:47 Hours UTC
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=h1) Report
> Merging [#18662](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fdba133336c2ec8f1269eb15e73d48797840c48b?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18662/graphs/tree.svg?token=eZ4WkYLtcO&src=pr&width=650&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18662      +/-   ##
==========================================
+ Coverage   91.57%   91.57%   +<.01%     
==========================================
  Files         153      153              
  Lines       51210    51210              
==========================================
+ Hits        46894    46895       +1     
+ Misses       4316     4315       -1
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.43% <ø> (+0.01%)` | :arrow_up: |
| #single | `40.67% <ø> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/util/testing.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL3V0aWwvdGVzdGluZy5weQ==) | `81.82% <0%> (-0.2%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.81% <0%> (-0.1%)` | :arrow_down: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.25% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=footer). Last update [fdba133...b05ae5d](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=h1) Report
> Merging [#18662](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/fdba133336c2ec8f1269eb15e73d48797840c48b?src=pr&el=desc) will **increase** coverage by `<.01%`.
> The diff coverage is `n/a`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/18662/graphs/tree.svg?token=eZ4WkYLtcO&width=650&src=pr&height=150)](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #18662      +/-   ##
==========================================
+ Coverage   91.57%   91.57%   +<.01%     
==========================================
  Files         153      153              
  Lines       51210    51212       +2     
==========================================
+ Hits        46894    46899       +5     
+ Misses       4316     4313       -3
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `89.43% <ø> (+0.02%)` | :arrow_up: |
| #single | `40.67% <ø> (-0.11%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.81% <0%> (-0.1%)` | :arrow_down: |
| [pandas/core/indexes/datetimes.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvaW5kZXhlcy9kYXRldGltZXMucHk=) | `95.68% <0%> (ø)` | :arrow_up: |
| [pandas/plotting/\_converter.py](https://codecov.io/gh/pandas-dev/pandas/pull/18662/diff?src=pr&el=tree#diff-cGFuZGFzL3Bsb3R0aW5nL19jb252ZXJ0ZXIucHk=) | `65.25% <0%> (+1.81%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=footer). Last update [fdba133...6161f65](https://codecov.io/gh/pandas-dev/pandas/pull/18662?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

The failure is the unreliable parallel_coordinates, so unrelated
did this pass CI?

on pyarrow 0.7.1, fp 0.1.3, on macosx
```
(pandas) bash-3.2$ pytest pandas/tests/io/test_parquet.py --tb=short
=========================================================================================== test session starts ===========================================================================================
platform darwin -- Python 3.6.1, pytest-3.2.1, py-1.4.34, pluggy-0.4.0
rootdir: /Users/jreback/pandas, inifile: setup.cfg
plugins: xdist-1.16.0, cov-2.3.1
collected 38 items                                                                                                                                                                                         

pandas/tests/io/test_parquet.py .....F............s...s...x...s.s.....

================================================================================================ FAILURES =================================================================================================
_________________________________________________________________________________________ test_cross_engine_pa_fp _________________________________________________________________________________________
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/api.py:96: in __init__
    with open_with(fn2, 'rb') as f:
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/util.py:44: in default_open
    return open(f, mode)
E   NotADirectoryError: [Errno 20] Not a directory: '/var/folders/h3/mr_r3bkj5yg0pbx9fr3tk1r00000gp/T/tmpii71wdx8/_metadata'

During handling of the above exception, another exception occurred:
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/api.py:119: in _parse_header
    fmd = read_thrift(f, parquet_thrift.FileMetaData)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/thrift_structures.py:22: in read_thrift
    obj.read(pin)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py:1899: in read
    _elem53.read(iprot)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py:1742: in read
    _elem33.read(iprot)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py:1656: in read
    self.meta_data.read(iprot)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py:1487: in read
    self.statistics.read(iprot)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/parquet_thrift/parquet/ttypes.py:298: in read
    iprot.skip(ftype)
../miniconda3/envs/pandas/lib/python3.6/site-packages/thrift/protocol/TProtocol.py:208: in skip
    self.readString()
../miniconda3/envs/pandas/lib/python3.6/site-packages/thrift/protocol/TProtocol.py:184: in readString
    return binary_to_str(self.readBinary())
../miniconda3/envs/pandas/lib/python3.6/site-packages/thrift/compat.py:37: in binary_to_str
    return bin_val.decode('utf8')
E   UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 2: invalid start byte

During handling of the above exception, another exception occurred:
pandas/tests/io/test_parquet.py:186: in test_cross_engine_pa_fp
    result = read_parquet(path, engine=fp)
pandas/io/parquet.py:211: in read_parquet
    return impl.read(path, columns=columns, **kwargs)
pandas/io/parquet.py:123: in read
    return self.api.ParquetFile(path).to_pandas(columns=columns, **kwargs)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/api.py:102: in __init__
    self._parse_header(f, verify)
../miniconda3/envs/pandas/lib/python3.6/site-packages/fastparquet/api.py:122: in _parse_header
    self.fn)
E   fastparquet.util.ParquetException: Metadata parse failed: /var/folders/h3/mr_r3bkj5yg0pbx9fr3tk1r00000gp/T/tmpii71wdx8
======================================================================== 1 failed, 32 passed, 4 skipped, 1 xfailed in 2.93 seconds ========================================================================
```
It did yes, but might be we don't have to good combination of versions on the CI to catch that
So it seems we don't have pyarrow on Mac, so this will be skipping those tests. 
Do you know if that was on purpose? To have a build with only fastparquet?
@jorisvandenbossche no wasn't on purpose, should add that.
Can you also open an issue on the fastparquet tracker?
yep