COMPAT: handle pyarrow deprecation of timestamps_to_ms in .from_pandas with pyarrow < 0.6.0
State: closed

closes #17438

Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: Compat
description: pandas objects compatability with Numpy or Python functions
name: IO Parquet
description: parquet, feather

Comments:
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=h1) Report
> Merging [#17447](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/5bca6ce860f66ca6f92327086a954b9e0326a85f?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `75%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17447/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17447      +/-   ##
==========================================
- Coverage   91.16%   91.14%   -0.03%     
==========================================
  Files         163      163              
  Lines       49581    49586       +5     
==========================================
- Hits        45199    45193       -6     
- Misses       4382     4393      +11
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.92% <75%> (-0.01%)` | :arrow_down: |
| #single | `40.25% <12.5%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `65.38% <75%> (-0.37%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.72% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=footer). Last update [5bca6ce...ea2ff0c](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=h1) Report
> Merging [#17447](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/5bca6ce860f66ca6f92327086a954b9e0326a85f?src=pr&el=desc) will **decrease** coverage by `0.02%`.
> The diff coverage is `75%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/17447/graphs/tree.svg?height=150&src=pr&token=eZ4WkYLtcO&width=650)](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #17447      +/-   ##
==========================================
- Coverage   91.16%   91.14%   -0.03%     
==========================================
  Files         163      163              
  Lines       49581    49586       +5     
==========================================
- Hits        45199    45193       -6     
- Misses       4382     4393      +11
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.92% <75%> (-0.01%)` | :arrow_down: |
| #single | `40.25% <12.5%> (-0.07%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/parquet.py](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree#diff-cGFuZGFzL2lvL3BhcnF1ZXQucHk=) | `65.38% <75%> (-0.37%)` | :arrow_down: |
| [pandas/io/gbq.py](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree#diff-cGFuZGFzL2lvL2dicS5weQ==) | `25% <0%> (-58.34%)` | :arrow_down: |
| [pandas/core/frame.py](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=tree#diff-cGFuZGFzL2NvcmUvZnJhbWUucHk=) | `97.72% <0%> (-0.1%)` | :arrow_down: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=footer). Last update [5bca6ce...ea2ff0c](https://codecov.io/gh/pandas-dev/pandas/pull/17447?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

:+1:, thanks @jreback!
Possibly related to this (didn't look into detail, but just saw this error in the doc builds):

```
>>>-------------------------------------------------------------------------
Exception in /tmp/doc/source/io.rst at block ending on line 4530
Specify :okexcept: as an option in the ipython:: block to suppress this message
---------------------------------------------------------------------------
ArrowInvalid                              Traceback (most recent call last)
<ipython-input-501-033b23d0b657> in <module>()
----> 1 df.to_parquet('example_pa.parquet', engine='pyarrow')
~/build/pandas-dev/pandas/pandas/core/frame.py in to_parquet(self, fname, engine, compression, **kwargs)
   1620         from pandas.io.parquet import to_parquet
   1621         to_parquet(self, fname, engine,
-> 1622                    compression=compression, **kwargs)
   1623 
   1624     @Substitution(header='Write out the column names. If a list of strings '
~/build/pandas-dev/pandas/pandas/io/parquet.py in to_parquet(df, path, engine, compression, **kwargs)
    176         raise ValueError("parquet must have string column names")
    177 
--> 178     return impl.write(df, path, compression=compression)
    179 
    180 
~/build/pandas-dev/pandas/pandas/io/parquet.py in write(self, df, path, compression, coerce_timestamps, **kwargs)
     75             self.api.parquet.write_table(
     76                 table, path, compression=compression,
---> 77                 coerce_timestamps=coerce_timestamps, **kwargs)
     78 
     79     def read(self, path):
~/miniconda3/envs/pandas/lib/python3.6/site-packages/pyarrow/parquet.py in write_table(table, where, row_group_size, version, use_dictionary, compression, use_deprecated_int96_timestamps, coerce_timestamps, **kwargs)
    794     try:
    795         writer = ParquetWriter(where, table.schema, **options)
--> 796         writer.write_table(table, row_group_size=row_group_size)
    797     except:
    798         if writer is not None:
_parquet.pyx in pyarrow._parquet.ParquetWriter.write_table()
error.pxi in pyarrow.lib.check_status()
ArrowInvalid: Casting from timestamp[ns] to timestamp[ms] would lose data: 1356998400000000001

<<<-------------------------------------------------------------------------
```

Failing example in the docs: https://pandas-docs.github.io/pandas-docs-travis/io.html#parquet
I pushed a fix for https://github.com/pandas-dev/pandas/pull/17447#issuecomment-328494887 @jorisvandenbossche