ENH: support for msgpack serialization/deserialization
State: closed

warning: prototype!

msgpack serialization/deseriiization
- support all pandas objects: Timestamp,Period,all index types,Series,DataFrame,Panel,Sparse suite
- docs included (in io.rst)
- iterator support
- top-level api support

Here are 2 features which I think msgpack supports, but have to look further
- no support for compression directly, but can compress the file (e.g. gzip)
- access is sequential
- versioning, is not that hard because its pretty easy to deal with a 
   change in the schema (which is not directly stored), and this is MUCH more
   transparent than pickle

usage is exactly like pickle (aside from that its in a different namespace), allowing
arbitrary combinations of storage, e.g. this supports the added storage of pandas objects,
but obviously can store `{ 'frame1' : df1, 'frame2' : df2 }` etc

storage example `DataFrame(np.random.rand(1000,10))` on my machine stores in 128k file size.
and scales pretty well, e.g. 10k rows is 1.26mb

Not completely happy with the `packers` name, any suggestions?

closes #686

```
In [1]: df = DataFrame(randn(10,2),
   ...:                      columns=list('AB'),
   ...:                      index=date_range('20130101',periods=10))

In [2]: pd.to_msgpack('foo.msg',df)

In [3]: pd.read_msgpack('foo.msg')
Out[3]: 
                   A         B
2013-01-01  0.676700 -1.702599
2013-01-02 -0.070164 -1.368716
2013-01-03 -0.877145 -1.427964
2013-01-04 -0.295715 -0.176954
2013-01-05  0.566986  0.588918
2013-01-06 -0.307070  1.541773
2013-01-07  1.302388  0.689701
2013-01-08  0.165292  0.273496
2013-01-09 -3.492113 -1.178075
2013-01-10 -1.069521  0.848614
```


Labels:

Comments:
@y-p can we add msgpack to the travis build (the full one?)

wait a minute, why aren't you using `scripts/use_build_cache.py` + tox/detox?
It'll make you so much more productive. You may in fact prove that there's a 
schwarzschild radius for coding and turn into a black hole.

hahah...you do keep pushing on tox....breaking down and installing tox.....

I just did `tox` in my main pandas repo and it looks likes its intalling stuff.....

This would be so great. I keep getting unreproducible pickle failures (SystemError) telling me no exception is set. I would love to be able to serialize without using pytables/pickle/npy.

I started on avro, but seemed harder to do...this is pretty straightforward

avro looks kind of insane

what about names space?

```
df.to_msgpack(path)
com.to_msgpack(path,dict/list/obj of stuff)
read_msgpack(path)
```

to keep consistent?

next question, I 'borrowed' the numpy impl (though I modified it) from this: https://github.com/lebedov/msgpack_numpy, license looks ok.....do we include a reference (in our code)? should I send him an e-mail?

That looks fine to me (the namespace). Looks like the impl calls `ndarray.tolist` which will be horribly slow for large arrays.

Here's why its a prototype
slow and bigger than pickle.....hmm...

```
In [11]: df
Out[11]: 
<class 'pandas.core.frame.DataFrame'>
DatetimeIndex: 10000 entries, 2000-01-01 00:00:00 to 2027-05-18 00:00:00
Freq: D
Data columns (total 10 columns):
0    10000  non-null values
1    10000  non-null values
2    10000  non-null values
3    10000  non-null values
4    10000  non-null values
5    10000  non-null values
6    10000  non-null values
7    10000  non-null values
8    10000  non-null values
9    10000  non-null values
dtypes: float64(10)

In [12]: %timeit packers.save('foo.msg',df)
10 loops, best of 3: 59.1 ms per loop

In [13]: %timeit df.save('foo.pickle')
100 loops, best of 3: 11 ms per loop

-rw-rw-r--  1 jreback users  880622 May  3 16:21 foo.pickle
-rw-rw-r--  1 jreback users 1300971 May  3 16:21 foo.msg
```

Hm. I love coding in Cython and it looks like there's a C interface, so I could hack an interface for you if u want. Not sure if it's worth it though. Maybe for 1e6 ord of mag # of rows could dispatch to Cython...

well, `save`/`load` suggest a privileged status, but we'd like to nudge users
towards the new format when this becomes fully operational, so not sure about the naming.

Also, need to tread _very_ lightly here, data serialization bugs are a big deal, we can't just
play cowboy with people's data. Maybe add an option to embed a version of the pickle data 
in the msgpack file for a major version or two?  that way people can recover from errors
at the cost of increased storage space.

Or maybe add a "safe=True" option that reads back the data and performs a comparison,
to warn the user if something goes wrong?

so out-of-thebox msgpack IS faster..its my datetimes that are screwing me, pickle is probably
a lot smarter...I was just doing isoformat().... what's fastes way of serializing datestime?

tuple of values, plus tz string?
some ordinal trick?

string index

```
In [1]: index = [rands(10) for _ in xrange(25000)]

In [2]: df = DataFrame({'float1' : randn(25000),
   ...:                 'float2' : randn(25000)},
   ...:                index=index)

In [3]: %timeit df.to_msgpack('foo.msg')
100 loops, best of 3: 14.2 ms per loop

In [4]: %timeit df.save('foo.pickle')
10 loops, best of 3: 20.1 ms per loop
```

datetimeindex

```
In [9]: %timeit df.to_msgpack('foo.msg')
10 loops, best of 3: 120 ms per loop

In [10]: %timeit df.save('foo.pickle')
100 loops, best of 3: 8.2 ms per loop
```

re licensing: IANL, but the license is 2 clause BSD so that's compatible with pandas, 
if you're using practically the entire thing, perhaps just make it a dependency, if only parts, 
try to put them in a separate file, with the licensing banner at the top.

In any case, a copy of LICENSE goes in LICENSES/, and as a courtesy drop the author an 
email to let him know his code is useful and that it's being integrated into a fairly 
high-profile library, as a form of thanks.

Better....had to avoid all of the boxing/unboxing in the index......

```
In [13]: index=date_range('20000101',periods=25000,freq='H')

In [14]: df = DataFrame({'float1' : randn(25000),
                'float2' : randn(25000)},
               index=index)

In [15]: %timeit df.save('foo.pickle')
100 loops, best of 3: 9.38 ms per loop

In [16]: %timeit pd.to_msgpack('foo.msg',df)
100 loops, best of 3: 13 ms per loop
```

hmmm....ok...I already integrated the code (I had to change it a bit)....so I'll put the license file and the banner at the top....thxs

@y-p how do I get an e-mail address for someone on github?

clone a repo they've committed to and use `git log`

This is really awesome by the way. I'm +1000 on having a pickle-independent serialization format. Was waiting for the superhero (you) to do it. I'm gonna try to find some time in the next few days to play with this a bit to look at some of the low level details. It might be worth bringing in the python-msgpack cython code so we have full control over things

@wesm definitley worth investigation. I pushed some docs updates; will finish up the rest of the missing types soon.

strongly urge 0.12. big change + more time for tire kicking in master.

@y-p changed to 0.12

@wesm now supports all pandas types (incl Sparse), docs included, added iterator support and added
ability to append to msgpack (you can actually do this with pickle to FYI)...you need to open the file with `a+b`

still open on whether there can implement some sort of random accessability......

go for perf improvements!

This is so cool. Can you save arbitrary objects, e.g., a dict of dicts of DataFrames?

With 1e7 rows there is sadness. Also, not sure if this is expected but

``` python
df = DataFrame(rand(1e7, 16))
pd.to_msgpack('huge.msg', df) # takes a long time :(
!du huge.msg # 1795796000
df.values.nbytes == 1280000000
# difference of ~500 MB?
```

yes

I think we need to use the msgpack array type to store the ndarrays
but I have found an example of its use 

Are the extra bytes from converting to a `list`?

not sure (ndarrays r stored as 1-dim lists), so could be

I'm not sure that's strictly true. Conceptually yes, but lists are bulkier because they can be heterogeneously typed, whereas ndarrays are almost always homogeneously typed. Anyway, I'm not trying to be pedantic here, just trying to get to the bottom of efficiently storing arrays.

going to try with an array type, its clear lists are done element by element are prob inefficient in terms of storage size (as they need markers bettween elements), and speed, e.g. 1-by one....