WIP: add pd.read_ipc and DataFrame.to_ipc to provide efficient serialization to/from memory
State: closed


Labels:
name: Enhancement
name: IO Data
description: IO issues that don't fit into a more specific label

Comments:
cc @mrocklin
@wesm @cpcloud 
Given that this is a pretty 'advanced' feature (and presumably an unstable format?), maybe would make sense to expose these functions somewhere like `pandas.api.lib` rather than in the main namespace?
@chris-b1 yeah probably. This is just trying to get working (today) locally. :> (its not actually anywhere atm, but in the local module).
# [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=h1) Report
> Merging [#15907](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=desc) into [master](https://codecov.io/gh/pandas-dev/pandas/commit/f35209e2279154ed7060dd0e17d41da96f9c0186?src=pr&el=desc) will **decrease** coverage by `0.1%`.
> The diff coverage is `0%`.

[![Impacted file tree graph](https://codecov.io/gh/pandas-dev/pandas/pull/15907/graphs/tree.svg?height=150&width=650&token=eZ4WkYLtcO&src=pr)](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master   #15907      +/-   ##
==========================================
- Coverage      91%    90.9%   -0.11%     
==========================================
  Files         145      146       +1     
  Lines       49576    49631      +55     
==========================================
  Hits        45118    45118              
- Misses       4458     4513      +55
```

| Flag | Coverage Δ | |
|---|---|---|
| #multiple | `88.67% <0%> (-0.1%)` | :arrow_down: |
| #single | `40.53% <0%> (-0.05%)` | :arrow_down: |

| [Impacted Files](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [pandas/io/ipc.py](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=tree#diff-cGFuZGFzL2lvL2lwYy5weQ==) | `0% <0%> (ø)` | |

------

[Continue to review full report at Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=footer). Last update [f35209e...f727477](https://codecov.io/gh/pandas-dev/pandas/pull/15907?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

I agree putting such things into an experimental namespace would be a good idea
Genuine question: since we are typically trying to limit pandas' scope, why should we include this in the core package? Instead of having this live in the package implementing this / a separate package ('pandas-ipc') providing this API?
> Genuine question: since we are typically trying to limit pandas' scope, why should we include this in the core package? Instead of having this live in the package implementing this / a separate package ('pandas-ipc') providing this API?

True, we are trying to limit scope. This is essentialy an ipc version of ``arrow`` (which also back ``feather``). So this ATM expands scope, though eventually would limit it somewhat (as the back-end would have less tooling).

Could consider this as ``pandas-ipc`` though this is pretty 'simple' and not meant as a full-fledged package, mostly a passthru / convenient interface for others (e.g. dask).
I think it would be useful for pandas to have more robust support for various transient serialization formats, with a choice between pickle (most compatible) vs msgpack vs arrow vs other things. Whether the implementation of this goes into core pandas, or into a "leaf" library that gets imported, I don't have a strong opinion
closing for now. This will be pretty transparent with ``pyarrow``, so this interface would just be a simple wrapper. let's see if its even needed.
so this is now available in a released version in arrow: https://arrow.apache.org/docs/python/ipc.html (IIRC 0.5.0 has full support). appetite for this in main pandas as ``read_ipc/to_ipc`` ?

@wesm @cpcloud @jorisvandenbossche 


Depending on what memory format these functions create it may affect the name. If it's vanilla arrow stream (schema + sequence of record batches), then it might be better to call it `read_arrow_stream` / `to_arrow_stream` (in the latter case, you could select a batch size to use for chunking the DataFrame once someone implements this)