Fix zlib and blosc imports in to_msgpack
State: closed

6717aa06dcaa1950ffb46fef454f5df9404209bd removed zlib and blosc from the global namespace.

```
from pandas import read_csv
table = read_csv('aadhaar_data.csv')
table.to_msgpack('d.msg')

# NameError: global name 'blosc' is not defined
table.to_msgpack('d-blosc.msg', compress='blosc')

# NameError: global name 'zlib' is not defined
table.to_msgpack('d-zlib.msg', compress='zlib')
```

This pull request restores zlib and blosc compression in to_msgpack via local imports.


Labels:
name: Bug
name: Compat
description: pandas objects compatability with Numpy or Python functions

Comments:
Could you please add tests for these? Looks like this functionality was not tested before.

this needs some tests
and a perf check 

they were not in the local imports originally because of perf

What's a perf check?

since this is importing each time in a lower level function I would like to see if that impacts things

show to/from msgpack with a good sized frame with mixed types with compression and without 

look at prun and see if the imports cause an issue

What is prun?

In my experience, module initialization happens only on the first import.
http://stackoverflow.com/questions/296036/does-python-optimize-modules-when-they-are-imported-multiple-times

You can test this by creating `a.py`

```
print 'hey'
```

and creating `b.py`

```
import a
import a
```

and then run `python b.py` to see that "hey" only prints once.

The following code also only prints "hey" once.

```
import a
import a

def f():
    import a

f()
```

Lastly, this variation also only prints "hey" once.

```
def f():
    import a

def g():
    import a

f()
g()
```

These examples hold true for python2.7 and python3.4.

use ipython

of course they are only imported once 

but the function call and import checks themselves are non trivial 

https://github.com/pydata/pandas/wiki/Performance-Testing

Here are some performance results using the following code saved as `compress.py`.

```
from cStringIO import StringIO
from pandas import read_csv, read_msgpack
from urllib2 import urlopen

url = 'http://earthquake.usgs.gov/earthquakes/feed/v1.0/summary/all_month.csv'
table = read_csv(StringIO(urlopen(url).read()))
print 'prun z = read_msgpack(table.to_msgpack())'
```

## When zlib and blosc are imported globally

$ ipython -i compress.py 
In [1]: prun z = read_msgpack(table.to_msgpack())

```
500 function calls (492 primitive calls) in 0.030 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.022    0.022    0.025    0.025 packers.py:134(read)
     1    0.002    0.002    0.003    0.003 {method 'pack' of 'pandas.msgpack.Packer' objects}
    15    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
    15    0.001    0.000    0.001    0.000 {numpy.core.multiarray.array}
     2    0.000    0.000    0.000    0.000 {method 'encode' of 'unicode' objects}
```

$ ipython -i compress.py 
In [1]: prun z = read_msgpack(table.to_msgpack(compress='zlib'))

```
508 function calls (500 primitive calls) in 0.059 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.027    0.013    0.027    0.013 {zlib.compress}
     1    0.024    0.024    0.027    0.027 packers.py:134(read)
     1    0.003    0.003    0.031    0.031 {method 'pack' of 'pandas.msgpack.Packer' objects}
     2    0.002    0.001    0.002    0.001 {zlib.decompress}
    15    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
```

$ ipython -i compress.py 
In [1]: prun z = read_msgpack(table.to_msgpack(compress='blosc'))

```
532 function calls (524 primitive calls) in 0.053 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.042    0.042    0.046    0.046 packers.py:134(read)
     1    0.003    0.003    0.005    0.005 {method 'pack' of 'pandas.msgpack.Packer' objects}
     2    0.001    0.001    0.001    0.001 {blosc.blosc_extension.compress}
    15    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
    15    0.001    0.000    0.001    0.000 {numpy.core.multiarray.array}
```

```
$ ipython -i compress.py 
In [1]: timeit read_msgpack(table.to_msgpack())
10 loops, best of 3: 27.7 ms per loop

$ ipython -i compress.py 
In [1]: timeit read_msgpack(table.to_msgpack(compress='zlib'))
10 loops, best of 3: 49.4 ms per loop

$ ipython -i compress.py 
In [1]: timeit read_msgpack(table.to_msgpack(compress='blosc'))
10 loops, best of 3: 28.7 ms per loop
```

## When zlib and blosc are imported locally

prun z = read_msgpack(table.to_msgpack())

```
500 function calls (492 primitive calls) in 0.029 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.022    0.022    0.025    0.025 packers.py:133(read)
     1    0.002    0.002    0.003    0.003 {method 'pack' of 'pandas.msgpack.Packer' objects}
    15    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
    15    0.001    0.000    0.001    0.000 {numpy.core.multiarray.array}
     2    0.001    0.000    0.001    0.000 index.py:1508(get_indexer)
```

$ ipython -i compress.py 
In [1]: prun z = read_msgpack(table.to_msgpack(compress='zlib'))
In [2]: prun z = read_msgpack(table.to_msgpack(compress='zlib'))
In [3]: prun z = read_msgpack(table.to_msgpack(compress='zlib'))

```
509 function calls (501 primitive calls) in 0.081 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.043    0.043    0.048    0.048 packers.py:133(read)
     2    0.027    0.013    0.027    0.013 {zlib.compress}
     1    0.004    0.004    0.031    0.031 {method 'pack' of 'pandas.msgpack.Packer' objects}
     2    0.002    0.001    0.002    0.001 {zlib.decompress}
    15    0.001    0.000    0.001    0.000 {numpy.core.multiarray.array}

479 function calls (471 primitive calls) in 0.053 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.022    0.011    0.022    0.011 {zlib.compress}
     1    0.022    0.022    0.025    0.025 packers.py:133(read)
     1    0.002    0.002    0.025    0.025 {method 'pack' of 'pandas.msgpack.Packer' objects}
     2    0.002    0.001    0.002    0.001 {zlib.decompress}
     1    0.001    0.001    0.053    0.053 <string>:1(<module>)

479 function calls (471 primitive calls) in 0.053 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     2    0.022    0.011    0.022    0.011 {zlib.compress}
     1    0.022    0.022    0.025    0.025 packers.py:133(read)
     1    0.002    0.002    0.025    0.025 {method 'pack' of 'pandas.msgpack.Packer' objects}
     2    0.002    0.001    0.002    0.001 {zlib.decompress}
     1    0.002    0.002    0.053    0.053 <string>:1(<module>)
```

$ ipython -i compress.py 
In [1]: prun z = read_msgpack(table.to_msgpack(compress='blosc'))
In [2]: prun z = read_msgpack(table.to_msgpack(compress='blosc'))
In [3]: prun z = read_msgpack(table.to_msgpack(compress='blosc'))

```
596 function calls (588 primitive calls) in 0.056 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.043    0.043    0.047    0.047 packers.py:133(read)
     1    0.003    0.003    0.007    0.007 {method 'pack' of 'pandas.msgpack.Packer' objects}
     2    0.002    0.001    0.002    0.001 {blosc.blosc_extension.compress}
    15    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
    15    0.001    0.000    0.001    0.000 {numpy.core.multiarray.array}

503 function calls (495 primitive calls) in 0.055 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.043    0.043    0.046    0.046 packers.py:133(read)
     1    0.003    0.003    0.006    0.006 {method 'pack' of 'pandas.msgpack.Packer' objects}
     1    0.002    0.002    0.055    0.055 <string>:1(<module>)
     2    0.002    0.001    0.002    0.001 {blosc.blosc_extension.compress}
    13    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}

503 function calls (495 primitive calls) in 0.054 seconds
Ordered by: internal time
ncalls  tottime  percall  cumtime  percall filename:lineno(function)
     1    0.042    0.042    0.045    0.045 packers.py:133(read)
     1    0.003    0.003    0.006    0.006 {method 'pack' of 'pandas.msgpack.Packer' objects}
     1    0.002    0.002    0.054    0.054 <string>:1(<module>)
     2    0.002    0.001    0.002    0.001 {blosc.blosc_extension.compress}
    13    0.001    0.000    0.001    0.000 {method 'get' of 'dict' objects}
```

```
$ ipython -i compress.py 
In [1]: timeit read_msgpack(table.to_msgpack())
10 loops, best of 3: 28.2 ms per loop

$ ipython -i compress.py 
In [1]: timeit read_msgpack(table.to_msgpack(compress='zlib'))
10 loops, best of 3: 49.1 ms per loop

$ ipython -i compress.py 
In [1]: timeit read_msgpack(table.to_msgpack(compress='blosc'))
10 loops, best of 3: 29.5 ms per loop
```

I didn't expect such a quick response to the pull request.  Thanks for maintaining the pandas package.  

ok, seems reasonable to use the local imports.
- remove some of the blosc imports (e.g. from 2.6, 2.7_LOCALE). Then in the tests test for blosc imports, if it fails, skip the test.
- add a release note in v0.16.1, you can put it in bug fixes, use this issue number as the reference number

This seems to be a documented issue between blosc and conda.
https://groups.google.com/a/continuum.io/forum/#!topic/conda/2a9Fyvzedug
Specifically, conda only installs c-blosc and not python-blosc.  Hmm.

hmm ok let me see if I can have this fixed

@invisibleroads this looks good!

This needs to be tested slightly differently. `encode_decode` passes the kwargs thru, so what you are doing is fine. 

BUT, you are adding the compression type to the objects themselves (again which is ok), I don't think this is actually used on decompression (its not tested that way now). AND I am not sure that you CAN use it, as you would have to decompress first, look at the compression variable, but then you are done already.

So I am not sure how to get around this problem, other than having a top-level section which has some uncompressed attributes.

So either we need to take off the `compress : ....` on the objects themselves as it doesn't serve any purpose or test it (and/or figure out how to encode the compression).

I added the `compress` attributes because `read_msgpack` was throwing an exception.  Specifically, `to_msgpack` compressed the table indices without setting the `compress` attribute and `read_msgpack > unconvert > np.fromstring` was throwing an exception because it did not know that the data had been compressed.  You can recreate this exception by reverting the extra `compress` attributes.

After realizing this, I added the `compress` attribute for every object type where `encode` applies `convert` (which does the actual compression) to `obj.values`.  I also checked that `decode` applies the `unconvert` function to these same object types.  You can see this if you search for all the uses of `convert` and `unconvert` in `packers.py`.

```
def encode(obj):
    if isinstance(obj, Index):
        return {
            'data': convert(obj.values),
            'compress': compressor,
        }

def decode(obj):
    elif typ == 'index':
        data = unconvert(
            obj['data'],
            np.typeDict[obj['dtype']],
            obj.get('compress'))
```

I think the `compress` attributes are necessary whenever `convert/unconvert` are used, but I'm not sure how or whether these attributes need to be tested explicitly.

The other alternative is to remove `convert/unconvert` from those object types that you do not want compressed.

Are you at PyCon?  Maybe we can negotiate this during sprints on Monday.

sure at pycon starting tomorrow (thru Tuesday)

@jreback 
You suggested creating a compressed file with an earlier version of pandas to make sure that the newer version of pandas can decompress that file without error.  However, it doesn't seem like the user would have been able to compress the file in the first place because of the local import error, so luckily we won't need to include a try/except to cover that case.

@jreback 
As suggested, I tried compressing a dataframe using an earlier version of pandas to_msgpack (f2882b85c2f639bd8e296c4cc5febcb3289a0892) but was unable to decompress the same file using read_msgpack because _to_msgpack compresses indices (without indicating that they are compressed) and read_msgpack tried to load the indices (without knowing that they are compressed)_.

```
In [10]: read_msgpack('timestamps-zlib.msg')
ValueError: string size must be a multiple of element size

In [11]: read_msgpack('timestamps-blosc.msg')
ValueError: Shape of passed values is (2, 13), indices imply (2, 15)
```

Since the user would not have been able to compress and decompress with to_msgpack/read_msgpack in any version of pandas, we decided that a try/except is not necessary and the code is good as is and is ready to be merged.