Repository: pandas-dev/pandas
Pull request: BUG: Fix the error when reading the compressed UTF-16 file
- [X] closes #18071
- [X] tests added / passed
- [X] passes `git diff upstream/master -u -- "*.py" | flake8 --diff`
- [x] whatsnew entry


Problem statement:
Reading zipped utf-16 file: AttributeError: 'UTF8Recoder' object has no attribute 'seek' 
#### Code Sample, a copy-pastable example if possible

```python
df = pd.read_table('myfile.zip', 
                   compression='zip',
                   encoding='utf-16')
```
#### Problem description
      Traceback (most recent call last):

     File "<ipython-input-4-6daea174e70a>", line 3, in <module>
      encoding='utf-16')

    File "/Users/Nick/anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 705, in parser_f
      return _read(filepath_or_buffer, kwds)

    File "/Users/Nick/anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 445, in _read
      parser = TextFileReader(filepath_or_buffer, **kwds)

    File "/Users/Nick/anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 814, in __init__
      self._make_engine(self.engine)

    File "/Users/Nick/anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 1045, in _make_engine
      self._engine = CParserWrapper(self.f, **self.options)

    File "/Users/Nick/anaconda/lib/python3.5/site-packages/pandas/io/parsers.py", line 1684, in __init__
      self._reader = parsers.TextReader(src, **kwds)

    File "pandas/_libs/parsers.pyx", line 391, in pandas._libs.parsers.TextReader.__cinit__

    File "pandas/_libs/parsers.pyx", line 664, in pandas._libs.parsers.TextReader._setup_parser_source

    File "/Users/Nick/anaconda/lib/python3.5/zipfile.py", line 1026, in __init__
      self._RealGetContents()

    File "/Users/Nick/anaconda/lib/python3.5/zipfile.py", line 1090, in _RealGetContents
      endrec = _EndRecData(fp)

    File "/Users/Nick/anaconda/lib/python3.5/zipfile.py", line 241, in _EndRecData
      fpin.seek(0, 2)

     AttributeError: 'UTF8Recoder' object has no attribute 'seek'

#### Expected Output

A table!

#### Output of ``pd.show_versions()``

<details>

  pd.show_versions()

  INSTALLED VERSIONS
  ------------------
  commit: None
  python: 3.5.3.final.0
  python-bits: 64
  OS: Darwin
  OS-release: 16.7.0
  machine: x86_64
  processor: i386
  byteorder: little
  LC_ALL: None
  LANG: en_US.UTF-8
  LOCALE: en_US.UTF-8

  pandas: 0.21.0
  pytest: 3.0.7
  pip: 9.0.1
  setuptools: 33.1.1.post20170320
  Cython: 0.25.2
  numpy: 1.11.3
  scipy: 0.19.0
  pyarrow: None
  xarray: None
  IPython: 5.3.0
  sphinx: 1.5.4
  patsy: 0.4.1
  dateutil: 2.6.1
  pytz: 2016.10
  blosc: None
  bottleneck: 1.2.0
  tables: 3.3.0
  numexpr: 2.6.2
  feather: None
  matplotlib: 1.5.1
  openpyxl: 2.4.5
  xlrd: 1.0.0
  xlwt: 1.2.0
  xlsxwriter: 0.9.6
  lxml: 3.7.3
  bs4: 4.5.3
  html5lib: 0.999
  sqlalchemy: 1.1.5
  pymysql: None
  psycopg2: 2.6.2 (dt dec pq3 ext lo64)
  jinja2: 2.9.5
  s3fs: 0.0.9
  fastparquet: None
  pandas_gbq: None
  pandas_datareader: None
</details>

Hints from issue discussion:
Hmmm...that looks kind of odd.  Mind sharing the ZIP file for us to reproduce?
@gfyoung sorry for delay -- couldn't upload original file, but here's what seems to be a minimal working example

[sample.zip](https://github.com/pandas-dev/pandas/files/1439381/sample.zip)

Yep, can reproduce this error.  Investigation and PR patch is welcome!

Commit message: TST: Add test for reading the zipped UTF-16 file

Patch to review:
diff --git a/pandas/_libs/parsers.pyx b/pandas/_libs/parsers.pyx
index a5ce6c560d844..e2d9cdd0540a4 100644
--- a/pandas/_libs/parsers.pyx
+++ b/pandas/_libs/parsers.pyx
@@ -374,6 +374,17 @@ cdef class TextReader:
                   float_precision=None,
                   skip_blank_lines=True):
 
+        # encoding
+        if encoding is not None:
+            if not isinstance(encoding, bytes):
+                encoding = encoding.encode('utf-8')
+            encoding = encoding.lower()
+            self.c_encoding = <char*> encoding
+        else:
+            self.c_encoding = NULL
+
+        self.encoding = encoding
+
         self.parser = parser_new()
         self.parser.chunksize = tokenize_chunksize
 
@@ -495,17 +506,6 @@ cdef class TextReader:
             self.parser.double_converter_nogil = NULL
             self.parser.double_converter_withgil = round_trip
 
-        # encoding
-        if encoding is not None:
-            if not isinstance(encoding, bytes):
-                encoding = encoding.encode('utf-8')
-            encoding = encoding.lower()
-            self.c_encoding = <char*> encoding
-        else:
-            self.c_encoding = NULL
-
-        self.encoding = encoding
-
         if isinstance(dtype, dict):
             dtype = {k: pandas_dtype(dtype[k])
                      for k in dtype}
@@ -684,6 +684,12 @@ cdef class TextReader:
             else:
                 raise ValueError('Unrecognized compression type: %s' %
                                  self.compression)
+
+            if b'utf-16' in (self.encoding or b''):
+                source = com.UTF8Recoder(source, self.encoding.decode('utf-8'))
+                self.encoding = b'utf-8'
+                self.c_encoding = <char*> self.encoding
+
             self.handle = source
 
         if isinstance(source, basestring):
diff --git a/pandas/io/parsers.py b/pandas/io/parsers.py
index 1b6414ea974fa..3108390af62d2 100755
--- a/pandas/io/parsers.py
+++ b/pandas/io/parsers.py
@@ -1671,7 +1671,8 @@ def __init__(self, src, **kwds):
 
         ParserBase.__init__(self, kwds)
 
-        if 'utf-16' in (kwds.get('encoding') or ''):
+        if kwds.get('compression') is None \
+           and 'utf-16' in (kwds.get('encoding') or ''):
             if isinstance(src, compat.string_types):
                 src = open(src, 'rb')
                 self.handles.append(src)
diff --git a/pandas/tests/io/parser/common.py b/pandas/tests/io/parser/common.py
index 6a996213b28bb..3c29ca08320b4 100644
--- a/pandas/tests/io/parser/common.py
+++ b/pandas/tests/io/parser/common.py
@@ -750,6 +750,16 @@ def test_utf16_example(self):
             result = self.read_table(buf, encoding='utf-16')
             assert len(result) == 50
 
+    def test_compressed_utf16_example(self):
+        # GH18071
+        path = tm.get_data_path('utf16_ex.zip')
+        expected_path = tm.get_data_path('utf16_ex.txt')
+
+        result = self.read_table(path, encoding='utf-16', compression='zip')
+        expected = self.read_table(expected_path, encoding='utf-16')
+
+        tm.assert_frame_equal(result, expected)
+
     def test_unicode_encoding(self):
         pth = tm.get_data_path('unicode_series.csv')
 
diff --git a/pandas/tests/io/parser/data/utf16_ex.zip b/pandas/tests/io/parser/data/utf16_ex.zip
new file mode 100644
index 0000000000000..f5fb659f4e433
Binary files /dev/null and b/pandas/tests/io/parser/data/utf16_ex.zip differ


Code review comments:
diff_hunk: @@ -76,7 +76,7 @@ I/O
 ^^^
 
 - Bug in class:`~pandas.io.stata.StataReader` not converting date/time columns with display formatting addressed (:issue:`17990`). Previously columns with display formatting were normally left as ordinal numbers and not converted to datetime objects.
-
+- Bug in :func:`read_table` when reading the compressed UTF-16 file (:issue:`18071`)
path: doc/source/whatsnew/v0.21.1.txt
text: @user1:
can you change to read_csv which is the common spelling here