Repository: pandas-dev/pandas

Issue:
TST: test_encode in test_html
@cpcloud not sure if this is something I did (or didn't do)

I was testing the index sub-class on 3.4 (may have appeared on travis too)

`/mnt/home/jreback/venv/py3.4/index/pandas/io/tests/data/html_encoding/chinese_utf-16.html'`

```
======================================================================
ERROR: test_encode (pandas.io.tests.test_html.TestReadHtmlEncoding)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/mnt/home/jreback/venv/py3.4/index/pandas/io/tests/test_html.py", line 627, in test_encode
    from_string = self.read_string(f, encoding).pop()
  File "/mnt/home/jreback/venv/py3.4/index/pandas/io/tests/test_html.py", line 622, in read_string
    return self.read_html(fobj.read(), encoding=encoding, index_col=0)
  File "/mnt/home/jreback/venv/py3.4/index/pandas/io/tests/test_html.py", line 610, in read_html
    return read_html(*args, **kwargs)
  File "/mnt/home/jreback/venv/py3.4/index/pandas/io/html.py", line 843, in read_html
    parse_dates, tupleize_cols, thousands, attrs, encoding)
  File "/mnt/home/jreback/venv/py3.4/index/pandas/io/html.py", line 709, in _parse
    raise_with_traceback(retained)
  File "/mnt/home/jreback/venv/py3.4/index/pandas/compat/__init__.py", line 705, in raise_with_traceback
    raise exc.with_traceback(traceback)
TypeError: Cannot read object of type 'bytes'

----------------------------------------------------------------------
Ran 64 tests in 61.014s

FAILED (errors=1)
(py3.4)jreback@sheep:~/venv/py3.4/index$ nosetests pandas//io/tests/test_html.py  --pdb --pdb-failure^C
(py3.4)jreback@sheep:~/venv/py3.4/index$ python ci/print_versions.py 

INSTALLED VERSIONS
------------------
commit: d1c4fbb0d170cfaf920a27907c014e8cc45752d1
python: 3.4.0.beta.3
python-bits: 64
OS: Linux
OS-release: 2.6.32-5-amd64
machine: x86_64
processor: 
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: None
nose: 1.3.3
Cython: 0.20.2
numpy: 1.8.0
scipy: 0.13.3
statsmodels: None
IPython: None
sphinx: None
patsy: 0.3.0
scikits.timeseries: None
dateutil: 2.2
pytz: 2014.4
bottleneck: 0.8.0
tables: 3.1.0
numexpr: 2.4
matplotlib: 1.3.1
openpyxl: 2.0.4
xlrd: 0.9.3
xlwt: None
xlsxwriter: 0.5.6
lxml: 3.3.5
bs4: 4.3.2
html5lib: 0.999
httplib2: None
apiclient: None
rpy2: None
sqlalchemy: 0.9.6
pymysql: 0.6.1.None
psycopg2: 2.5.2 (dt dec pq3 ext)
```



Discussion:
looks like those farm animals are `byte`-ing.

sorry i couldn't resist

jokes aside not sure what's going on here let me take a look

@jreback any reason why this doesn't currently fail?

i can repro locally, but this failure isn't showing up on travis

I think this is using a pretty recent bs4 (4.3.2), not testing on travis with that

but that's not actually where the bug is. it happens because i don't check for bytes and str types in `_read` i only check for str types ... this is way before any parser libraries get called

this line in the tests

``` python
        with open(f, 'rb') as fobj:
            return self.read_html(fobj.read(), encoding=encoding, index_col=0)
```

should pass in a `bytes` type in py3 and this line

``` python
    elif isinstance(obj, string_types):
```

should test `False` (because `string_types == (str,)` in py3 and `obj` is `bytes`) and thus the type error

@jreback don't worry about it i'll figure it out

actually those tests aren't even being run on travis

https://travis-ci.org/cpcloud/pandas/jobs/32559013

really, they don't appear to be skipped?

i know it's strange ... i think there's an installation issue tho not sure

i changed to `python setup.py develop` in the conda builds

i don't see why that should matter tho

ok it turns out that _does_ matter. when i install using the sdist method everything passes and when i run it using `make develop` it doesn't. mind boggling

the test _is_ run tho which is why this is strange

@jreback any reason to use `sdist` vs `make develop` on travis?

no idea

oh duh this is totally a data path issue,

`self.files` is `[]` so nothing is actually iterated over


Resolving patch:
diff --git a/pandas/io/html.py b/pandas/io/html.py
--- a/pandas/io/html.py
+++ b/pandas/io/html.py
@@ -15,8 +15,8 @@
 
 from pandas.io.common import _is_url, urlopen, parse_url
 from pandas.io.parsers import TextParser
-from pandas.compat import (lrange, lmap, u, string_types, iteritems, text_type,
-                           raise_with_traceback)
+from pandas.compat import (lrange, lmap, u, string_types, iteritems,
+                           raise_with_traceback, binary_type)
 from pandas.core import common as com
 from pandas import Series
 
@@ -51,6 +51,9 @@
 _RE_WHITESPACE = re.compile(r'[\r\n]+|\s{2,}')
 
 
+char_types = string_types + (binary_type,)
+
+
 def _remove_whitespace(s, regex=_RE_WHITESPACE):
     """Replace extra whitespace inside of a string with a single space.
 
@@ -114,13 +117,13 @@ def _read(obj):
             text = url.read()
     elif hasattr(obj, 'read'):
         text = obj.read()
-    elif isinstance(obj, string_types):
+    elif isinstance(obj, char_types):
         text = obj
         try:
             if os.path.isfile(text):
                 with open(text, 'rb') as f:
                     return f.read()
-        except TypeError:
+        except (TypeError, ValueError):
             pass
     else:
         raise TypeError("Cannot read object of type %r" % type(obj).__name__)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -583,6 +583,7 @@ def pxd(name):
                                   'tests/data/*.xlsm',
                                   'tests/data/*.table',
                                   'tests/data/*.html',
+                                  'tests/data/html_encoding/*.html',
                                   'tests/test_json/data/*.json'],
                     'pandas.tools': ['tests/*.csv'],
                     'pandas.tests': ['data/*.pickle',


Test patch:
diff --git a/pandas/io/tests/test_html.py b/pandas/io/tests/test_html.py
--- a/pandas/io/tests/test_html.py
+++ b/pandas/io/tests/test_html.py
@@ -622,6 +622,7 @@ def read_string(self, f, encoding):
             return self.read_html(fobj.read(), encoding=encoding, index_col=0)
 
     def test_encode(self):
+        assert self.files, 'no files read from the data folder'
         for f in self.files:
             _, encoding = _lang_enc(f)
             from_string = self.read_string(f, encoding).pop()