BUG: GH10581 where read_msgpack does not respect encoding
State: closed

Close #10581 by upgrading to `msgpack` 0.4.6


Labels:

Comments:
Since there is no forward compat, should I remove `items` from `encode` #10623?

do we care about using 0.17.0 and writing a pack that it back be read by < 0.17.0 (say when non-encoded). I think you could actually. (you want to do that experiment?)

Yes, having a new key-value pair for the new `ExtType` in the output of `encode` would do it, but that'd come with a nontrivial increase in size, but if the original one is deprecated later then I think it might be worth it.  Is that what you have in mind?  If so what should the new key be named, since the current one is the most obvious choice?

not sure what u mean by your last 

Is adding the `ExtType` part into the dict by a key like `b_data` and keeping the original `data` to be deprecated later what you had in mind? 

Ah. You meant using `ExtType` only when some other encoding is specified?  

no what I meant was I believe we currently have forward compat (if we leave in items and don't worry about encoding)

so just wanted to test if that is true

I see.  This change is not forward compat because of I have put things that are binary into `ExtType`, which is new.  And old `msgpack` would not be able to deal with this.

updated

@jreback any change needed?

looks reasonable. let me give a test on windows 

can you rebase, then ping me and I will give a run on windows

@jreback rebased

something is off with the pathing of the libraries. 
works fine when built in-place, but  building via

`python setup.py build` (which is how most build systems do it - travis builds in-place though)

cannot find the libraries (they are built correctly)

```
[jreback-~/pandas] nosetests  build/lib.macosx-10.5-x86_64-2.7/pandas/io/tests/test_packers.py 
E
======================================================================
ERROR: Failure: ImportError (No module named msgpack)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/jreback/anaconda/lib/python2.7/site-packages/nose/loader.py", line 414, in loadTestsFromName
    addr.filename, addr.module)
  File "/Users/jreback/anaconda/lib/python2.7/site-packages/nose/importer.py", line 47, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/Users/jreback/anaconda/lib/python2.7/site-packages/nose/importer.py", line 94, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/Users/jreback/pandas/build/lib.macosx-10.5-x86_64-2.7/pandas/__init__.py", line 48, in <module>
    from pandas.io.api import *
  File "/Users/jreback/pandas/build/lib.macosx-10.5-x86_64-2.7/pandas/io/api.py", line 15, in <module>
    from pandas.io.packers import read_msgpack, to_msgpack
  File "/Users/jreback/pandas/build/lib.macosx-10.5-x86_64-2.7/pandas/io/packers.py", line 63, in <module>
    from pandas.msgpack import Unpacker as _Unpacker, Packer as _Packer, ExtType
ImportError: No module named msgpack

----------------------------------------------------------------------
Ran 1 test in 0.001s
```

Ah yes I will need to append `pandas.msgpack` to the `packages` arg in `setup()`.  Will do that tonight.

k, thanks, lmk

updated. Tested building without `--inplace`, so it should work now. 

thanks @kawochen 

awesome effort!