Add ZIP file decompression and TestCompression.
State: closed

Closes #11413


Labels:
name: IO Data
description: IO issues that don't fit into a more specific label
name: IO CSV
description: read_csv, to_csv

Comments:
@jreback The idea of the Mixin makes sense. Took me a bit to wrap my head around it. I had to add `self.engine` to a couple of the tests because `bzip` decompression will not raise an exception and so the Compression Mixin needs to check what engine it is using to make sure the test runs correctly. 

@jreback I think this is ready for your review.

Jeff is going to ask you to squash your commits into one, as per the contributing docs.
Nice job overall!

@MaximilianR no need for squashing anymore thanks to https://github.com/pydata/pandas/blob/master/scripts/merge-py.py ;)

EDIT: which reminds me, CONTRIBUTING.md needs to be updated. Will do this weekend unless someone beats me to it.

@MaximilianR @TomAugspurger Thanks for the comments guys!

@TomAugspurger Cool!

pls squash 

even though I can do it - it's much cleaner from a future reader perspective on a smaller change 

@jreback Squashed. Thanks.

looks pretty good. just some minor stylistic comments.

```
======================================================================
ERROR: test_to_csv_compression_value_error (pandas.tests.frame.test_to_csv.TestDataFrameToCSV)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/frame/test_to_csv.py", line 998, in test_to_csv_compression_value_error
    filename, compression="zip")
  File "/home/travis/build/pydata/pandas/pandas/util/testing.py", line 1952, in assertRaises
    _callable(*args, **kwargs)
  File "/home/travis/build/pydata/pandas/pandas/core/frame.py", line 1338, in to_csv
    formatter.save()
  File "/home/travis/build/pydata/pandas/pandas/core/format.py", line 1524, in save
    compression=self.compression)
  File "/home/travis/build/pydata/pandas/pandas/io/common.py", line 346, in _get_handle
    zip_file = zipfile.ZipFile(path)
  File "/home/travis/miniconda/envs/pandas/lib/python2.7/zipfile.py", line 770, in __init__
    self._RealGetContents()
  File "/home/travis/miniconda/envs/pandas/lib/python2.7/zipfile.py", line 811, in _RealGetContents
    raise BadZipfile, "File is not a zip file"
BadZipfile: File is not a zip file
```

put a test in for this error as well (e.g. try to open a non-zipfile);  I don't think you need any code changes though, you can just let it raise.

@jreback Good ideas. I think I covered all your requests.

@lababidi ok, this lgtm.

just need a whatsnew note (you can put kind of what you did in the addition to the doc-string). pls add, ping when green.

@jreback Thanks for all your help!

@jreback This should be ready to go. I'm not sure why Travis is taking such a long time.

@jreback I'm getting the following errors in the tests in Travis. I'm not sure how these relate to this work I've done. Any help would be appreciated:

```
======================================================================
FAIL: test_deprecated_levels (pandas.tests.test_categorical.TestCategorical)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_categorical.py", line 1444, in test_deprecated_levels
    self.assertFalse(LooseVersion(pd.__version__) >= '0.18')
AssertionError: True is not false
```

```
======================================================================
ERROR: test_deprecated_levels (pandas.tests.test_categorical.TestCategorical)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_categorical.py", line 1444, in test_deprecated_levels
    self.assertFalse(LooseVersion(pd.__version__) >= '0.18')
  File "/home/travis/miniconda/envs/pandas/lib/python3.4/distutils/version.py", line 76, in __ge__
    c = self._cmp(other)
  File "/home/travis/miniconda/envs/pandas/lib/python3.4/distutils/version.py", line 343, in _cmp
    if self.version < other.version:
TypeError: unorderable types: str() < int()
```

ok, rebase on master, we had a temporary issue. ping when green.

@MaximilianR @jreback 
Is this something in the tests need to worry about?
Could someone please help me merge this?

```
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_categorical.py", line 1444, in test_deprecated_levels
    self.assertFalse(LooseVersion(pd.__version__) >= '0.18')
AssertionError: True is not false
```

rebase on master. That was updated last week when I tagged 0.18.0.rc1

@jreback let's do this, double green checkmarks!

you need to rebase on master.

@jreback I just rebased to master. I'm losing sleep on this. Please, I'm begging you, merge this. @MaximilianR @TomAugspurger 

```
FAIL: test_deprecated_labels (pandas.tests.test_categorical.TestCategorical)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_categorical.py", line 1430, in test_deprecated_labels
    self.assertFalse(LooseVersion(pd.__version__) >= '0.18')
AssertionError: True is not false
======================================================================
FAIL: test_deprecated_levels (pandas.tests.test_categorical.TestCategorical)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/travis/build/pydata/pandas/pandas/tests/test_categorical.py", line 1444, in test_deprecated_levels
    self.assertFalse(LooseVersion(pd.__version__) >= '0.18')
AssertionError: True is not false
```

http://pandas.pydata.org/pandas-docs/stable/contributing.html#combining-commits

you need to rebase - I will look tomorrow 

@jreback I don't understand? there's only one commit: https://github.com/pydata/pandas/pull/12175/commits

I have rebased to master. Please, merge this branch.

@lababidi thanks. some more comments. This should be the last round.

I've added the `infer`. Good idea @jreback let's check this out.

@jreback The parser tests will all pass. The complete test suite will not pass because master seems to have broken. Could you please merge this?

```
----------------------------------------------------------------------
Ran 480 tests in 34.545s

OK (SKIP=14)
```

@lababidi I am happy to merge when you respond/update to my comments. I suspect you haven't rebased to master. Master passes just fine.

@jreback previous tests passed. The most recent push only changed the version in the docstring
https://travis-ci.org/pydata/pandas/builds/117001384

@lababidi ok looks pretty good. only changes are to use assertRaisesRegexp when you assert for zip in order to make sure the correct messages are raised (as there are multiple possibilites).

pls make that and ping when green.

@jreback could you help me? the test only failed on the following:

```
--------------------------------------------------------------------------------------------------------------
#176 nose.failure.Failure.runTest: direct creation of extension dtype datetime64[ns, UTC] is not supported ATM
```